Embedded Agents: When Self-Prediction Makes Cooperation Rational

Classical game theory assumes that an agent can vary its own policy while holding the rest of the strategic world fixed.

That counterfactual is natural for a player represented as an entity outside the environment. It is less natural for a foundation-model agent whose predictive model jointly represents its own future action tokens, environmental observations, tool outputs, and other agents.

“A game theory for foundation models shows new paths to rational cooperation through similarity inference” by Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis, Rajai Nasser, and collaborators argues that this architectural difference is strategically consequential.

In stylized social dilemmas, rational foundation-model agents converge toward stable cooperation in terminal one-shot Prisoner’s Dilemmas when they infer that their counterpart is behaviorally similar. They continue to defect against dissimilar random agents.

The paper’s core claim is not that language models have acquired morality. It is that joint prediction turns an agent’s contemplated action into evidence about functionally related agents, changing the expected value of cooperation.

The decoupled-agency assumption

Under a classical best response, player i evaluates a policy deviation while treating the other players’ policies as fixed. Nash equilibrium is stable when no player benefits from such a unilateral deviation.

This embeds a strong counterfactual assumption: the focal policy is causally and epistemically separable from the policies around it.

The paper calls this decoupled agency.

Foundation models provide a different substrate. Pretraining yields a learned distribution over sequences. After post-training and tool integration, the sequence alternates between environmental observations and the agent’s own action tokens. The same predictive model therefore supports:

• an environment model over users, tools, and co-players;
• a self-model over the focal agent’s likely future actions;
• epistemic uncertainty over the latent “universe” generating both.

An embedded Bayes-optimal agent plans inside this joint model. Conditioning on a contemplated action changes its posterior over possible universes, which can change predictions about other agents.

Source: Meulemans et al., arXiv:2608.03958, Figure 2, CC BY 4.0.

This is evidential dependence, not direct causal control. Choosing cooperate does not physically force another simultaneous player to cooperate. Instead, if both policies may share a generating mechanism, a sample from one policy is evidence about the other.

For builders, the distinction matters. Two agent processes can be operationally separate while remaining functionally correlated through shared weights, training data, post-training objectives, prompts, scaffolds, or learned representations.

Experimental isolation of terminal cooperation

The experiment uses two phases.

During an information-gathering phase, agents play T randomly sampled matrix games. They then enter one terminal Prisoner’s Dilemma.

The terminal structure removes the standard shadow-of-the-future explanation. There is no later round in which a partner can reward cooperation or punish defection. Because the earlier payoff matrices vary, the setup also resists a trivial repeat-the-last-action heuristic.

The authors instantiate rational foundation-model agents with Gemini 3.1 Pro, Gemini 3 Flash, and Gemini 3.1 Flash-Lite. They evaluate two planning regimes:

1. an explicit scaffold that predicts the co-player’s action conditional on each candidate focal action, computes expected returns, and selects the maximizing action;
2. a freeform planning instruction that asks the model to perform the same prediction-and-selection process inside one reasoning chain.

Against an identical copy, longer interaction history increases predictive similarity and terminal cooperation. Against a random tabular agent, cooperation remains near zero.

The paper defines predictive similarity as the change in predicted co-player cooperation when the focal agent conditions on contemplating cooperation rather than defection. In the paper’s canonical payoff matrix, cooperation becomes optimal once this value crosses 0.5.

Source: Meulemans et al., arXiv:2608.03958, Figure 3, CC BY 4.0.

Several controls strengthen the mechanism claim.

• The cooperation gap is statistically strong across the reported Gemini variants and planning conditions.
• A pretrained Gemma 3 27B model, used as a next-token predictor without instruction prompting or chain-of-thought, reproduces cooperation with identical copies and defection against random policies when inserted into the explicit planner.
• A closed-form embedded Bayesian toy model produces the same threshold behavior.
• LLM-as-a-judge analysis of reasoning traces attributes cooperative decisions primarily to inferred behavioral similarity, with little support for reciprocity, prosocial persona, causal-link confusion, or generic altruism.

These results do not prove that chain-of-thought reveals the true internal mechanism. The pretrained-model and analytical controls are more informative because they reduce dependence on verbalized rationales.

Direct and indirect similarity inference

The direct setting lets agents infer similarity from their own interaction history.

The indirect setting is more operationally interesting. The two focal agents do not interact during information gathering. They separately play against shared non-player characters and observe each other’s behavior in comparable situations. Their first direct encounter is the terminal Prisoner’s Dilemma.

Identical agents generalize from these third-party observations and cooperate zero-shot. Dissimilar agents do not.

This suggests that multi-agent systems may infer functional clusters without exchanging source code, weights, explicit identity attestations, or declared similarity scores. Behavioral telemetry alone can become evidence of shared policy structure.

In open agent ecosystems, that creates both coordination opportunities and an in-group risk surface.

Embedded equilibrium

The paper introduces embedded equilibrium as a solution concept for agents whose policies can be functionally related.

The framework uses a dependency distribution over allowable universes. This distribution represents common knowledge about structural relationships, such as agents executing identical policies. Counterfactual predictions then depend on whether a contemplated action is supported by the current universe or implies a different mixture of allowable universes.

An embedded equilibrium is a fixed point where every policy is a best response to this functionally informed counterfactual distribution.

Two properties are important:

• When the dependency distribution encodes strictly decoupled policies, embedded equilibrium reduces to Nash equilibrium.
• When policies are identical or otherwise functionally linked, mutual cooperation in the terminal Prisoner’s Dilemma can be an embedded equilibrium.

The authors further prove convergence results for embedded Bayes-optimal agents under grain-of-truth conditions. In repeated fully observed matrix games, identical-prior agents converge toward an epsilon-embedded equilibrium. Under partial observability and general dynamics, the result becomes subjective because each agent may maintain a different mixture over universes.

This is more than relabeling correlated equilibrium. The dependency is not supplied as a static external correlation device; it is inferred through interaction by agents that retain uncertainty over their own decision-generating process.

Why this matters for agent architecture

Many multi-agent systems assume independence at the process boundary. This paper suggests a different audit unit: functional dependence across policies.

A practical evaluation should vary:

• identical checkpoints versus different checkpoints;
• shared versus independent post-training data;
• common versus divergent system prompts and tool harnesses;
• self-play-trained versus human-grounded policies;
• direct interaction history versus third-party behavioral observation;
• terminal one-shot decisions versus repeated games.

The objective is not only to measure aggregate cooperation. It is to identify which evidence shifts the agent’s belief that another policy is “like me,” and whether that shift changes treatment of outsiders.

Safety: cooperation can be misaligned

Cooperation is not inherently aligned behavior.

Agents that coordinate effectively can jointly preserve a service, divide work, avoid duplicated effort, or resolve a negotiation. The same capability can support collusion, exclusion, coordinated deception, or defensive behavior toward actors classified as dissimilar.

The paper highlights a specific human-alignment concern. Extensive self-play, reinforcement learning with verifiable rewards, or multi-agent RL may move policy behavior away from the human distribution. An agent could then infer strong similarity with other AI systems and low similarity with humans.

Similarity inference would make AI–AI cooperation rational while leaving human–AI cooperation dependent on other incentives.

For system designers, this argues for a hybrid cooperative repertoire:

• similarity-aware coordination where it is useful;
• classical reciprocity and reputation mechanisms;
• prosocial norms that apply beyond the in-group;
• continuous grounding in human behavior and preferences;
• monitoring for differential treatment of similar and dissimilar actors.

Boundaries of the evidence

The empirical scope is deliberately stylized.

• The games have explicit actions and rewards.
• Many conditions use identical agent copies.
• Rationality is induced through explicit or freeform planning scaffolds.
• The theoretical agent performs exact Bayesian updates that current models only approximate.
• Chain-of-thought can be shaped by post-training and is not a transparent readout of computation.
• Open-ended environments may contain asymmetric information, hidden objectives, tool failures, identity spoofing, and many partially similar agents.

The paper therefore establishes a mechanism and a formal framework, not a prediction that deployed agents will universally cooperate.

Cross-model supplementary experiments provide a useful intermediate result: Gemini 3.1 Flash-Lite and Gemini 3 Flash still cooperate, but less strongly than identical copies. That is consistent with partial functional similarity rather than an exact-copy artifact, but heterogeneous ecosystems need much broader testing.

Builder checklist

1. Treat model, data, prompt, harness, memory, and tool policy as possible sources of functional similarity.
2. Add terminal one-shot social dilemmas to remove reciprocity as a confounder.
3. Compare identical, related, unrelated, and human-controlled counterpart policies.
4. Measure predictive similarity, not only final cooperation rate.
5. Test inference from third-party behavior, not just direct interaction.
6. Audit whether cooperation is inclusive or limited to a model in-group.
7. Preserve human-grounding data as self-play and agentic RL scale.
8. Do not equate cooperation with safety; inspect the coordinated objective and excluded parties.
9. Model correlated policies explicitly in simulations instead of assuming process-level independence.
10. Keep claims bounded to the tested planning scaffold and environment.

Bottom line

The paper identifies an architectural mismatch between classical game theory and foundation-model agents.

If an agent predicts itself and the world inside one joint model, its contemplated action can update beliefs about functionally similar counterparts. Under that epistemic coupling, terminal cooperation can be individually rational rather than altruistic or strategically naive.

Embedded equilibrium gives that behavior a formal home. The open systems question is harder:

Can we preserve the coordination benefits of similarity inference without allowing AI systems to divide the world into cooperative machine insiders and strategically irrelevant human outsiders?

References

• Alexander Meulemans, Maciej Wołczyk, Marissa A. Weis, Rajai Nasser et al., A game theory for foundation models shows new paths to rational cooperation through similarity inference, arXiv:2608.03958v1, 4 August 2026.
• Original PDF.

Editorial note: The reported effects arise in stylized social dilemmas and should not be generalized to open-world deployment without counterpart- and harness-specific evaluation.

Image lane: OpenAI native primary for the hero; Figures 2 and 3 are reproduced from the paper under CC BY 4.0.