Kyle Roth

These are my notes from a conversation between Yoshua Bengio and Kate Crawford held at Mila on 2023-03-20, announcing the release of a new book created as a joint report between Mila and UNESCO called Missing links in AI governance (link). There were news articles in French (Le Devoir), but not as many in English unfortunately (Datanami). Bengio: What motivates you to do what you do? This topic has turned from an academic move to a societal one really quickly, and it’s scary.

On 2022-11-03 a class action lawsuit was announced against GitHub Copilot on the basis of copyright infringement, and now (2023-01-13) there’s one for stable diffusion (against StabilityAI and friends). Browsing through r/StableDiffusion, I’m seeing lots of posts like this making the very memeable point that 5 billion images can’t be stored in a 4 GB model. From the original poster:The thumbnail for this post was generated with stable diffusion, positive prompt “artist stands at a conveyor belt in an art factory, quickly painting a piece before moving on to the next blank canvas on the belt” and negative prompt “blurry, distorted”.

I’m writing these things down over time because I’d like to figure out how to make artificial neural networks better at storing and retrieving knowledge, and the human brain is pretty good at that. When knowledge is new, it sometimes only comes back to me when I’m in the right context. For example, right now I can only think vaguely of how to adjust a chain derailleur on a bike, but I know that if I walked up to my bike and looked closely at the derailleur some more knowledge would appear.

Thanks to the crash of FTX and Scott Aaronson’s subsequent post about SBF, I read a very interesting deep-dive into Effective Altruism by the New Yorker. I’m seeing a lot of important characters show up that I’ve seen before: Eliezer Yudkowsky, earn-to-donate, 80,000 hours, etc. It’s really fascinating to see this all coming together in one narrative so I can understand a little better the inspiration for these ideas and the way that the movement has interacted with the world up till now.

I think it’s valuable to be working in the open whenever possible, so I’m going to keep my research notes here. These notes will hopefully be full of good (and bad) ideas, so if someone borrows a good idea and publishes on it, that’s great! This post contains my research notes as I try to understand how model scaling affects worst-group performance. This started as a group project in the neural scaling laws course at Mila in winter 2022.