Quick Takes

"Humans cannot secure matrices of 175B floating point numbers" seems kind of isomorphic to the statement "Humans cannot secure engines containing 10^23 gas molecules". There are often laws which govern large ensembles of things which are simpler than counting up every single thing in the ensemble.

YSK about the Noerr–Pennington doctrine. AI companies colluding to slow down or stop development of better models is a textbook example of an innovation cartel, and risks antitrust enforcement, but if they instead lobby the government to impose a restriction they all then abide by, that's protected First Amendment activity.

Structuring Sparsity: Block-Sparse Featurizers Capture Visual Concept Manifolds Goodfire's new dictionary learning architecture which captures manifolds and seems like the first one to work on toy models of manifold discovery. Strangely enough, I haven't seen much discussion of it online, and the release wasn't publicized. Perhaps an extension to LLMs will be released soon?

Pro tip for all fine-tuning experiments. Fuck around with your optimizer. In some cases I've seen different results with Adam and Muon. If you want to use LoRA there's a Muon version called Riamannion or something like that. From a physics-of-LLMs perspective, Muon is a very different creature to Adam; I'd like to get as much tacit knowledge into the community as possible, about exactly what this means. One example: Muon has less Emergent misalignment, SGD has more. https://arxiv.org/html/2606.31591v1

David Lewis famously observed that one man's modus ponens is another man's modus tollens. The first man argues: While the second man argues: But Lewis forgot about the third man:

EAs will have increasingly diminished leverage over the labs. This is mostly because more groups are waking up to ASI, and will start pushing the labs towards their values and worldview. We might look back on 2021-2025 as a relatively nice regime where the labs only had to please the investors and the EAs. And in 2025-2030 they'll need to please the investors, the EAs, the natsec, the judiciary, each major government, each major religion, the cultural elite, each industry of workers, the AIs themselves, etc.

"[In this line of work] You come to the job firing on all cylinders, or not at all. The rest is just fine-tuning and chemistry" -- qntm, There Is No Antimemetics Division I find myself confused about how (human) researchers turn inference time into conceptual reasoning or better conceptual progress. I suspect it's something I don't understand. Relatedly, I also suspect I'm differentially bad at it. Maybe if I understand it better I'd be better at my job, and/or have some insights into how AIs can become better at conceptual reasoning/macrostrategy. Just to give a sense of how I currently think/research/reason: 1. When thinking about a topic, a lot of the value seems to come from giving it a "single full forward pass" on thinking it through: 1. This could be a fast forward pass (eg I hear about a topic in a conversation and/or read a blog post/paper and see what I could contribute) 1. this can take seconds, but usually takes minutes. 2. Or a medium/slow forward pass where I hear about a topic and try to "reason it through", deliberating on which thing I understand vs find confusing: 1. This usually takes hours, though it can sometimes take days. 2. I can extend the value of inference time by looking up empirical facts and trying to understand the state of the empirical literature, and updating my models in light of the new facts: 1. For a research question I'm interested in, this usually takes 1-30 hours, typically on the lower end. 1. It's often the case that the first few things I learn are the most useful? 1. Like the first fact/paper is the most informative, etc, etc. 2. I do think I'm unusually fast on this, relatively speaking like ppl often comment on how I seem to know the empirical lit well, I often notice mistakes in other people's understanding of the empirical literature after very (in absolute terms) short dives on my end. 2. But it often seems like while the first few facts I learn about a fiel