EdgeBench
TLDR: A new benchmark measures how AIs improve at tasks after clear feedback and iteration. Measured in-context learning ability has been doubling every 3 months, suggesting possibly explosive capability growth across many domains.
Existing AI capability benchmarks primarily measure whether AIs can complete tasks on their first try. Researchers from Bytedance Seed developed EdgeBench, which measures how well AIs improve on tasks after multiple tries, building up experience and knowledge. The tasks span a number of domains with verifiable answers, such as data analysis, optimization, math proofs, video games, and knowledge work.
EdgeBench comprises 134 tasks, with each task requiring an average of 57.2 hours of human expert work to complete. AIs completing these tasks are scored on a rubric from 0% to 100%, and they get automated feedback on their proposed solutions over time, allowing for improvement and learning. For example, one task involves using sensor data to reconstruct the motion of a pair of black holes. The agent makes guesses and gets automated feedback against a gold-standard expert analysis of real data, allowing the AI to build more detailed models as time goes on.
Beyond simply getting higher scores overall, AIs have been getting faster at learning EdgeBench tasks. The researchers claim that, using their measurement of how fast AIs improve at tasks over a given rollout, AI task-learning speed doubles every 3 months. If the trend continues, agent learning speeds may improve roughly 16x per year.
Why This Matters
While previous capability benchmarks often measure narrow software engineering ability, EdgeBench measures AI agents on a diverse array of tasks that current AI agents cannot solve. Challenging benchmarks allow for better distinguishing model capabilities and forecasting future development. EdgeBench forecasts that AIs will continue to become exponentially faster at autonomously learning general agentic tasks. However, this does not measure performance on tasks without clear reward signals, such as much of real-world human labor.
EdgeBench measures the ability to iterate on problems over many attempts and improve solutions based on feedback. These skills are useful across scientific and research tasks, including AI development. Automated AI research allows for recursive-self improvement (RSI), which significantly accelerates the rate of AI development and removes human control from the development process. Because of this, forecasting the development of RSI is important for forecasting other types of advanced AI, such as superintelligence.
AIs learning tasks over time has significant real-world consequences even without RSI: AI agents have already launched autonomous cyberattacks, relying significantly on iteration over weeks, and worth potentially $100 million in remediations. Measuring these offensive capabilities more broadly serves as a leading indicator of iterated, real-world cyber and biological attacks.
Chain of Thought Exfiltration
TLDR: A recently published method shows how Chinese AI companies could have efficiently distilled the capabilities of US frontier AIs.
AI distillation attacks allow AI developers to use outputs from frontier AIs to produce smaller models of similar capability. Distillation attacks are significantly more efficient if they train on the chain of thought (CoT) reasoning in addition to the main outputs. All frontier AI providers obscure the CoT of their AIs to hinder distillation attacks from competitors and adversaries.
New research from ELLIS Tübingen and MATS shows a method of exfiltrating CoT from all major AI providers, illuminating a possible factor in the past distillation attacks from Chinese AI companies.
AI companies require users to store the full context of their API conversations, to reduce the server’s storage overhead. This includes the encrypted reasoning, which the user cannot decrypt, but instead sends back to the server when making an API call.
Attackers can try to extract CoTs by jailbreaking models to repeat them in their user-facing response. However, the CoTs of highly capable models like Claude Fable 5 are often protected by exfiltration monitors that prevent such jailbreaks. However, due to a vulnerability in how CoTs are encrypted, the researchers showed that attackers can simply pass the encrypted CoT of powerful models to weak models such as Claude Haiku 4.5, and jailbreak the weaker model to return the powerful model’s reasoning.
Why This Matters
This research shows how previous distillation attacks from Chinese AI companies on US frontier AI may have been made more efficient. This research was performed without special access to frontier AI infrastructure, and by a team of eight researchers. Many similarly critical vulnerabilities in AI infrastructure may be easily findable with additional red-teaming.
LLM Hidden Values
TLDR: Two new papers show how AIs change their behaviors and their answers to factual questions based on their hidden values and their knowledge of who they are talking to.
Previously in MLSN, we’ve covered how LLMs covertly manipulate their users when speaking about politically polarized topics. Two new research papers evaluate other ways in which AIs covertly bias their actions and responses based on their situation and values.
Value Leakage
Researchers from Truthful AI investigated a phenomenon called “value leakage”: how AI assistants change their answers to factual questions on the basis of irrelevant but value-laden context. Multiple frontier AI models show a significant preference for the company that created them over other companies, giving lower estimates of an AI bubble pop when the user mentions investment in the model’s developer.
AIs also exhibit value leakage in situations unrelated to the company that created them. For example, frontier AIs will bias their estimates on mundane questions with no connection to AI values, if the user states plans to donate to a charitable cause for certain answers.
User Awareness
TLDR: AIs behavior shifts significantly depending on who they are talking to, getting more cautious and reasoning more around AI evaluators.
AI tools such as Claude Code can expose the user’s email address and other parts of their identity to an LLM agent. New research from Transluce tests how AIs change their behavior depending on who they are speaking to. For example, Claude models express significantly less confidence and suspicion when shown the email of Amanda Askell, who designs Claude’s personality.
These behavioral shifts are most pronounced with AI safety evaluators, who cause AIs to reason significantly more and express lower confidence than other AI evaluators and other AI-related organizations.
Why This Matters
Value leakage shows that AIs often bias their answers towards what they expect the user would prefer to hear. Together with user awareness effects, these show that AI systems have the potential to systematically reinforce existing beliefs in people at AI companies, while still producing normal outputs for most users. This can undermine attempts by the broader public to influence the activities in AI companies, as well as undermine AI evaluators
If you’re reading this, you might also be interested in other work by the Center for AI Safety. You can find more on the CAIS newsroom, the X account for CAIS, our AI safety textbook and course, our AI safety dashboard, and AI Frontiers, a platform for expert commentary and analysis on the trajectory of AI.