Kyle Roth

I'm a computer science PhD candidate at DIRO at the Université de Montréal, advised by Bang Liu. I apply a critical sociotechnical perspective to study the use of LLMs within larger pipelines and agent systems, especially in the context of code generation. Here's my resume (and the academic version).

During the course of my PhD program, my understanding of the political nature of "AI" (and the implications for my own research) has shifted substantially. My latest post "AI research is dead, long live AI" gets into this a little, but a lot of the older posts here I no longer see the same way. More changes to this site are on the way.

Previously I was a speech scientist at Cobalt Speech & Language, a company that designs custom speech recognition, text-to-speech, and dialogue models. While at Cobalt I worked on some neat projects, including language modeling for recognizing air traffic control speech and creating an online training system for ASR models.

I graduated from BYU with a BS in Applied and Computational Mathematics (ACME) with an emphasis in linguistics and a minor in computer science. ACME's curriculum includes graduate-level courses in algorithms, analysis, optimization, statistics, data science, optimal control, and machine learning.

During my undergrad I interned both with Cobalt Speech and Emergent Trading.

We developed a simple formalism for procedural knowledge, built a custom retrieval system using this formal structure, and designed a custom LLM pipeline to retrieve and adapt known procedures to new tasks by analogy. We test this system on three datasets: cooking recipes, coding tutorials, and step-by-step math solutions. This version is on arXiv, and we're now building a version that uses a more complex, graph-based formalism.

I am currently leading a project to design an open-source, privacy-oriented, extensible chat UI and LLM application framework. The application framework defines a set of "skills" as Python code that calls upon resources such as LLMs, retrievers, local files, conversational memory, multi-modal models, or web services. These sandboxed skill scripts can be developed and shared by humans, or automatically generated and fine-tuned using LLMs. The project is designed to logically separate the skill system from the local UI application so the former can be generally applicable in new environments. This will build on the formal representations we have developed for AAG.

In this work we developed a pipeline that first searches over unique functions called in the codebase, then leverages a standard static analysis tool to find all snippets that call the top functions, and finally checks those snippets using several dense models of varying sizes (and runtimes) to selectively filter these snippets by each model's confidence. The thresholds of this pipeline are optimized for search speed using simulated annealing over a small dataset of query-snippet pairs, with a constraint on false positives and false negatives. This system has been deployed in production on the Moderne platform to perform high-quality code search on industrial-scale codebases.