Kyle Roth

I'm a third-year computer science PhD student at DIRO at the Université de Montréal, studying NLP for procedural knowledge. I am advised by Bang Liu. Here's my resume (and the academic version).

Previously I was a speech scientist at Cobalt Speech & Language, a company that designs custom speech recognition, text-to-speech, and dialogue models. While at Cobalt I worked on some neat projects, including language modeling for recognizing air traffic control speech and creating an online training system for ASR models.

I graduated from BYU with a BS in Applied and Computational Mathematics (ACME) with an emphasis in linguistics and a minor in computer science. ACME's rigorous curriculum includes graduate-level courses in algorithms, analysis, optimization, statistics, data science, optimal control, and machine learning.

During my undergrad I interned with Cobalt Speech, as well as Emergent Trading, an automated trading firm that made the news for reporting a problem in a Eurodollar exchange rule that unfairly favored larger competitors. (I developed the analysis tools that were used to track the issue down and determine how our opponent was taking advantage of the rule.)

We are developing a custom retrieval-augmented generation (RAG) system to leverage LLMs to work with arbitrary procedural knowledge. We test this system on three datasets: cooking recipes, coding tutorials, and step-by-step math solutions. A preliminary version is on arXiv, and an expanded version representing procedural knowledge as graphs will be out soon.

In this work we developed a pipeline that first searches over unique functions called in the codebase, then leverages a standard static analysis tool to find all snippets that call the top functions, and finally checks those snippets using several dense models of varying sizes (and runtimes) to selectively filter these snippets by each model's confidence. The thresholds of this pipeline are optimized for search speed using simulated annealing over a small dataset of query-snippet pairs, with a constraint on false positives and false negatives. This system has been deployed in production on the Moderne platform to perform high-quality code search on industrial-scale codebases.

I am currently leading a project to design an open-source, privacy-oriented, extensible chat UI and LLM application framework. The application framework defines a set of "skills" as Python code that calls upon resources such as LLMs, retrievers, local files, conversational memory, multi-modal models, or web services. These sandboxed skill scripts can be developed and shared by humans, or automatically generated and fine-tuned using LLMs. The project is designed to logically separate the skill system from the local UI application so the former can be generally applicable in new environments. This will build on the formalisms we have developed for AAG.