Intrinsically Curious Agents

David Li & Jonathan Li · Aug 25, 2026

Today, we outline our early research in intrinsic discovery, an approach for models to autonomously explore an environment and generate their own training experience. We use reinforcement learning to continually push a model toward novel states, then use the resulting experience to train a world model.

Using intrinsic discovery, we train LLMs to find 5x more diverse states in terminal environments than a fixed-policy baseline. We show that the states found during discovery are significantly better for training world models. After training on self-discovered states, our world model beats GPT-5.6 Sol on AgentWorldBench-Terminal-V2AgentWorldBench-Terminal-V2 is our improved subset from Zou et al., 2026. More evaluation details below., a challenging terminal simulation benchmark.

Horizontal bar chart of AgentWorldBench-Terminal scores. — Using *intrinsic discovery*, we train Terminal-35B-A3B (post-trained from Qwen3.6-35B-A3B). It beats GPT-5.6 Sol at terminal world modeling, despite using no human supervision.

Learning from an environment

Humanity has consistently improved its models of the world through experience. Underlying this is our ability to run experiments we think might produce interesting behavior.

World models can learn to predict environment states from demonstrations of actions and their outcomes. However, producing actions that expose interesting parts of an environment typically requires human demonstrations or prompting. Our key result is that reinforcement learning can push an LLM toward novel states in a terminal environment, and that these states are useful for training a world model.

Discovering novel states

Intrinsic discovery uses reinforcement learning to search for novel states. Unlike RLVR or RLHF, the goal is not the policy produced by training, but the rollouts produced along the way.

At each step, we sample actions from the current policy $\pi_t$ , execute them in the environment, and reward the resulting observations for being different from previously discovered observations:

a_t \sim \pi_t,\qquad x_t \sim P(\cdot \mid s_t,a_t),

r(x_t)=1-\max_{x\in\mathcal A_t}\mathrm{sim}(x_t,x).

Here, $\mathcal A_t$ is the set of observations discovered before step $t$ . Observations that are unlike anything in $\mathcal A_t$ receive higher reward. We update $\pi_t$ with GRPO, add the new observations to $\mathcal A_t$ , and repeat.

The policy itself does not need to be diverse at any point in training. As previously novel states are added to $\mathcal A_t$ , they stop receiving reward, pushing the policy toward new parts of the environment. Diversity emerges over time in the accumulated set of discovered states.

We use the generated command-observation pairs produced during discovery to train a world model initialized from the same base model.

Exploring the terminal

We demonstrate that intrinsic discovery autonomously improves world modeling capability across three open-source LLMs of varying scales (gpt-oss-20b, Qwen3.6-35B-A3B, and gpt-oss-120b). Each model runs its own discovery process to generate terminal commands and their resulting observations, which we then use to train a world model with RLVR.

We evaluate the resulting world models on AgentWorldBench-Terminal-V2, which is fully held out from the discovery process.AgentWorldBench contains samples that require the model to produce hallucinations. We built AgentWorldBench-Terminal-V2 by regenerating the environment outputs and filtering out invalid examples. We follow the AgentWorldBench evaluation methodology, but use GPT-5.6 Luna (xhigh) as the judge instead of GPT-5.2.

Bar chart pairing each world model with its base model on AgentWorldBench-Terminal. — Intrinsic discovery consistently improves world modeling performance across LLM sizes and architectures.

Compared to a fixed-policy baseline using the same base model and matched data budget, data produced through intrinsic discovery leads to substantially better downstream world model performance.

Line chart of internal TerminalWorldBench accuracy versus RL compute, with shaded error bands. Training on intrinsic-discovery data climbs steadily to 64.8, while training on baseline RL data plateaus near 50.9. — Ablation on gpt-oss-20b showing the benefit of intrinsic discovery over the baseline (samples generated by a static policy). Shown evaluation is TerminalWorldBench, our internal eval.

We measure diversity using the number of unique shell commands and Python libraries encountered.These are imperfect proxies for diversity and can be gamed, for example by generating random commands. However, the improvement in downstream world model performance provides separate evidence that the discovered states contain useful training data. Across model architectures, intrinsic discovery finds unique states faster than a fixed LLM policy.

Two line charts of unique shell commands and unique Python libraries discovered over 20,000 executed commands. For both Qwen3.6-35B-A3B and gpt-oss-120b, intrinsic discovery keeps finding new commands and libraries at a steep rate, reaching roughly 3 to 5 times the totals of frozen baseline policies, which flatten early. — Over time, intrinsic discovery consistently uncovers more unique commands and Python libraries compared to sampling directly from a fixed LLM.

Looking ahead

Today’s models are highly dependent on human-generated data, a finite resource that is expensive to procure and difficult to scale. With intrinsic discovery, models can learn the dynamics of an environment entirely unsupervised.

We believe a key characteristic of autonomous agents is the ability to learn through self-directed interaction in an environment. Our work on intrinsic discovery is an early step in this direction, which we see as a foundation for superintelligent agents acting in simulated environments and the real world.

We’re a team of two based in San Francisco. We’re always looking for exceptional thinkers, engineers, and researchers. If our work sounds interesting to you, leave us a note at team@inductionlabs.com.