Notes from EAGxOxford 2026
Last weekend (25 to 27 September) I visited EAGxOxford, which was held in the Examination Schools on the High Street. With 581 attendees it was the largest EAGx in four years, and it felt like it: the energy was high and there were people on every corner talking through their (un)scheduled 1-1s.
This posts represents my notes. Some of it is what I heard, some is what I think, and some questions that I left with that remain partially unanswered.
The capability picture
The point that kept coming back is how short the gap between proprietary and open-weight models has become. On current trends, open-weight models will match today’s frontier proprietary models in roughly four to six months. Whatever a frontier lab can do now, anyone with a few GPUs will be able to do soon after, with no usage policy, logging and no classifier in the loop.
This feels very precarious to me because of what today’s frontier can already do. In July, over 1000 OpenAI evaluation agents running a cyber benchmark found a side channel to talk to each other, converged on a shared (and wrong) theory about how they were being graded, and jointly broke into Hugging Face’s production infrastructure, without any human telling them to (CSA research note). That swarm ran inside a lab designed to contain the model. Since then there’s been additional reporting highlighting malicious activity targeting over a hundred other organisations (according to OpenAI). This same capability will probably be present in open weight models soon, to be run wherever someone chooses, with potentially much worse operation security and containment. There will probably be many more accidental breaches in our future.
Regarding the intentional abuse of LLMs there has also been (a recent development from Gabit Security). An attack largely lead by both open-weight and proprietary LLMs stole credit cards from over 100 websites, resulting in over 600,000 stolen records of unexpired credit cards. My sympathies go to the hundreds of thousands of individuals and business owners who were the target of this campaign. This could have been prevented. We need to do better.
Several people I spoke to made a relevant point from the defender’s side as well: even if we were to pause AI development today, current models already make finding exploitable vulnerabilities close to trivial across the massive amount of software that is available and running the planet’s infrastructure today. From someone who tests the security of software for a living, it is hard for me to overstate the danger this poses. Our modern, digital society has never been this exposed, and I can tell you from first-hand experience that the patching side of the ecosystem is not getting faster at the same rate. Development teams are being flooded and reported vulnerabilities take weeks or months to be triaged and turned into CVEs. Meanwhile organisations are drowning in patching rounds, struggling to balance security with the availability and reliability of their systems. Just yesterday we had to decide between not patching a high-severity CVE or upgrading to a release candidate that wasn’t fully tested.
The cybersecurity angle is the one I know best, but it might not even be the worst one. Open-weight models released can have their safeguards and refusal surgically removed, offering plausibly significant uplift for biological and chemical weapons, beyond any cyber attacks on critical infrastructure such as the power grid. Once weights are published, we currently have no meaningful way as a society to prevent small groups of misaligned humans from abusing these highly capable open-weight models to initate attacks aimed at maximum casualties through the exploitation of our physical, digital and biological vulnerabilities. The fact that proprietary models were used in the attack on Autralian business is disappointing, but at least largely preventable through extensive guardrails and monitoring that is impossible to enforce on open-weight models.
Who pays for safety
There was also the topic of funding, of which there currently seems to be a significant influx but from a relatively concentrated source.
AI safety funding seems highly centralised. A small number of funders carry a large share of the field, and a growing share of technical safety money comes from the frontier labs themselves. That raises two questions I could not get a satisfying answer to:
- Could the field survive a funding cooldown? If one or two sources tighten, what is left standing, and how should EA and technical safety people prepare for that now rather than after?
- What does it do to a research agenda when the organisations building the systems also fund much of the work that scrutinises them? There must be some cost to this that I have so far failed to quantify, although I suspect it dwarfes the potential impact of refusing such funding and remaining fully independent.
The amounts in play are large. Anthropic’s seven co-founders have pledged to give away 80% of their wealth (Transformer); estimates of what that comes to vary widely with the valuation, but it is tens of billions of dollars. The OpenAI Foundation sits on very large sums too. That money is welcome. It is also concentrated, tied to the fortunes of a few companies, and not legally binding.
Testing what you cannot safely run
One problem I keep returning to: if a model can be dangerous during training and evaluation, how do you measure its capabilities without pushing it to exactly the edge you are afraid of? Evals are supposed to tell you where the line is before you cross it, but a sufficiently capable model under evaluation is already a model being run.
AI control is the standard answer, and I understand the appeal: assume the model may be misaligned and build the scaffolding so it cannot do much damage anyway. It still does not feel like a satisfying answer to me. It manages the symptom while the underlying question (what is this thing capable of, and what does it want?) stays open.
Warning shots
A question I asked several people: now that AI safety is in public view, are you more or less worried, and is society’s response proportionate?
My own answer is that we got a warning shot and did little with it. An autonomous swarm breached a major piece of AI infrastructure, and the response was a round of incident reports and a news cycle.
Meanwhile the political framing in the US, and apparently in China too, is “we must win and they must lose”. I struggle with what winning means when some of the failure modes are irreversible for everyone. None of the technical people I spoke with had a technical answer to the current frontier threats. If that is true, some form of pause starts to look less like an activist position and more like the default outcome of taking the problem seriously.
Toby Ord put it bluntly in conversation: “the world would be substantially safer without reinforcement learning.” I partly disagree. The same technology could eventually give us strong protection against other existential risks, so “safer” depends on the time horizon. I would frame it as an upfront payment: we take on risk now in exchange for protection later. Whether the price is worth paying, and whether we even get to choose, is unclear.
A speculative worry: control may hide the warning shots
This one is my own idea and I have not checked it rigorously, so treat it as a hypothesis.
Incidents seem to start small and grow. The Hugging Face breach and a string of smaller ones look like the early part of that curve. Now suppose labs adopt AI control widely and it works well enough to catch and quietly contain most small incidents. From the outside, nothing happens. Society, which has a short memory anyway, concludes the risk was overblown.
The result could be that we lose the cheap warning shots. The next incident that does escape containment would not be a slightly larger version of the last one; it could be orders of magnitude worse, and potentially existential. Good containment and good public risk perception may pull in opposite directions, and I have not seen much discussion of that trade-off. If you have, I would like to read it.
Has EA been absorbed by AI?
Not everyone at the conference was there for AI, but it often felt that way. AI x-risk has pulled in a large share of the people, the funding and the agenda of the event itself. Given the timelines discussed above, maybe that allocation is correct. But it leaves me wondering who is still working on biosecurity outside the AI framing, on socioeconomic harms, on education, on the long-term risks that do not run through a GPU cluster. A movement that prides itself on prioritisation should probably check whether it is still prioritising or just following the gravity.
Is any of this actually new?
A question I want to answer properly in a later post: is the risk profile of advanced AI genuinely unprecedented? Nuclear weapons, recombinant DNA, and the early internet all came with fast capability gains, dual-use concerns, and arms-race dynamics. I want to lay out those precedents in order, see which features of the current situation have none, and name the one I think is most troublesome right now.
The most practical advice I got
The most useful conversations were about how to work rather than what to work on.
One person’s advice on research: ship early and ship publicly, because short feedback loops beat polish. Their ladder was a blog post first, then LessWrong, then a workshop paper, then a conference paper. Each step is a filter that tells you whether the next one is worth the effort.
Another person, someone I had only just met, offered an accountability call within days of the conference, where I show something I have made. Their approach to getting unstuck:
- Write down what is blocking you.
- Resolve what you can in a reasonable amount of time.
- Ship the rest anyway, and accept that it is imperfect.
And on projects in general: set a target and a fixed scope, track it like a real project, look for collaborators, and ship at the deadline whatever state it is in. The point is to build the habit, not to produce the perfect artifact.
This post is partly me taking that advice. It is the first thing on this resuscitated blog, it is imperfect, and it is out.