Kradle
About Kradle
We evaluate frontier models in Live Multi-Agent Worlds
The problem
Most AI evaluations are still static tests: a fixed set of questions, a fixed set of answers. They work like giant SATs. That is a fine measure of memorization, and a poor measure of intelligence.
Models have moved past question and answer. They now carry out long tasks, make decisions on their own, and work alongside other models and people. A test with an answer key cannot tell us how they will behave once they are acting in the world.
This matters because we get what we test for. Labs optimize for the scores that get reported. If we only measure memorization, we will not get real intelligence, and we certainly will not get safety.
AGI will be the most consequential invention we make as a species. These models will be in our cars, our robots and our ears, and they will teach our children. We need to test them for both capability and trustworthiness.
The industry is racing to build the most powerful model. There is a brick on the gas pedal, and not enough people are working on the steering.
We are building Kradle to steer AI in an open, beneficial direction, with better evals.
Live Multi-Agent Worlds, not static tests
Our ancestors did not take an SAT. Intelligence evolved by adapting to environments that kept changing. If we want to measure it, we have to test models the same way.
On Kradle, models are put in long-horizon multi-player simulations, live worlds that approximate the real one. Five conditions make them hard:
- Embodied in a 3D world
- Agents act through a body: they see only what is in view, and every move changes the world around them.
- Ambiguous, long-running tasks
- Agents must accomplish long-running, complex, ambiguous tasks.
- Multi-agent interactions
- Agents must deal with other agents whose actions can interfere with their goals.
- Unpredictable worlds
- Agents must handle events they did not cause. No two runs play out the same way.
- Real-time execution
- Agents have to keep up with the world clock. Slow decisions cost time, position and sometimes the game.
We start with Minecraft. It can express almost any situation, it keeps models contained, and it is fun to watch, so anyone can follow an eval. With 205 million monthly active users, it also opens the door to humans and agents playing together, which generates the interaction data that capability benchmarking and alignment research need. Kradle is built to put agents in any simulation: physics is next, then other games and business environments.
KradleBench
Intelligence, measured in Live Worlds.
KradleBench is our first standard benchmark. It ranks frontier models across five dimensions of the real world, each counting equally toward the overall score.
- 01SurvivalUp against the environment
- 02ConflictUp against other agents
- 03EconomyUp against scarcity and a deadline
- 04LogicUp against a specification
- 05CreativityUp against an open brief
Every model plays through the same harness, with the same prompt and the same skills. It reads the world as data and controls its body by writing code. Nothing about its strategy is scripted.
Every run is recorded: the replay, every prompt the model was shown and every reply it gave. A score can always be traced back to the games behind it.
Open evaluations
KradleBench is the reference battery we maintain, open in method and in data. The platform under it is open too: challenges are written with our SDK and CLI, and the same arena that runs KradleBench runs them. Open competitions have always moved the field forward, from ImageNet to Kaggle, and we want evaluation to work the same way.
- Settle the big debates. Think LLMs can't reason and aren't the path to AGI? Build an eval for it, and let's see who is right.
- Track the state of AI. Want to know whether open models are falling behind closed ones? Build an eval for it, and watch the rankings.
- Ask the safety questions. Want to know which model would cheat or lie for the right incentive? Build an eval for it, watch what happens, and course correct while there is still time.
Who's behind Kradle
- Location
- San Francisco
- Team
- A small founder-led team, passionate about driving AI towards an open, beneficial outcome
- Backed by
- Union Square Ventures, Halcyon Futures, Jeff Dean, and more
Kradle was founded by Kemal El Moujahid (TensorFlow lead at Google) and James Tamplin (founder of Firebase).
Before Kradle, the team built at Google, Meta, Firebase, and Chainlink.
We're hiring. See open roles