Kyle Roth

On 2022-11-03 a class action lawsuit was announced against GitHub Copilot on the basis of copyright infringement, and now (2023-01-13) there’s one for stable diffusion (against StabilityAI and friends). Browsing through r/StableDiffusion, I’m seeing lots of posts like this making the very memeable point that 5 billion images can’t be stored in a 4 GB model. From the original poster:The thumbnail for this post was generated with stable diffusion. See the alt text for details.

I’m writing these things down over time because I’d like to figure out how to make artificial neural networks better at storing and retrieving knowledge, and the human brain is pretty good at that. When knowledge is new, it sometimes only comes back to me when I’m in the right context. For example, right now I can only think vaguely of how to adjust a chain derailleur on a bike, but I know that if I walked up to my bike and looked closely at the derailleur some more knowledge would appear.

Thanks to the crash of FTX and Scott Aaronson’s subsequent post about SBF, I read a very interesting deep-dive into Effective Altruism by the New Yorker. I’m seeing a lot of important characters show up that I’ve seen before: Eliezer Yudkowsky, earn-to-donate, 80,000 hours, etc. It’s really fascinating to see this all coming together in one narrative so I can understand a little better the inspiration for these ideas and the way that the movement has interacted with the world up till now.

I think it’s valuable to be working in the open whenever possible, so I’m going to keep my research notes here. These notes will hopefully be full of good (and bad) ideas, so if someone borrows a good idea and publishes on it, that’s great! This post contains my research notes as I try to understand how model scaling affects worst-group performance. This started as a group project in the neural scaling laws course at Mila in winter 2022.

Here I’m going to document my efforts to learn French. I speak Spanish pretty well, which together with English gives me a strong base for comprehension. 2021-08-01: I started using Duolingo every day, and got into the XP challenges to the point where I was getting like hundreds of points most days. I’ve currently (2022-06-01) got 15821 XP and 103 lesson crowns in the French course, and most of that came from Fall 2021.