Microsoft Research’s JoyTwin, announced on November 1, 2026, tackles a practical problem for AI agents: a live environment keeps changing while the model decides what to do. In the open-source 3D role-playing game Veloren, JoyTwin runs a continuous action executive alongside an asynchronous quest planner, allowing it to dodge, fight and navigate while higher-level reasoning continues. It operates through player-visible screen frames and standard mouse-and-keyboard controls, without privileged access to the game’s internal state, according to the project announcement.
Microsoft reports that JoyTwin completed 14 of 25 tasks on its first episode and 19 after three episodes. The strongest listed comparison system completed seven. Those are Microsoft’s results, not independently reproduced measurements, but the architecture offers a useful explanation for why some visual agents struggle in real time: their controller waits for reasoning to finish while the environment keeps moving.
The important distinction is between deciding the next objective and keeping control of the current situation. JoyTwin treats those as concurrent jobs.
Why Waiting for a Plan Can Lose the Fight
A synchronous agent follows a simple sequence: observe the screen, reason about the situation, issue an action, then repeat. That sequence can work when the environment waits between turns. In a live action game, several seconds of deliberation can leave the character exposed.
Microsoft illustrates this failure with a combat replay in which its DeepSeek Harness baseline pauses between reasoning turns while an enemy approaches. The project reports that the character takes 16 melee hits while frozen and dies. In a separate demonstration against the same enemy type, JoyTwin continues tracking, spacing and striking while the planner supervises progress, winning in an in-game duration of 52.4 seconds. The page labels these videos as accelerated 4× replays, not continuous live recordings. These demonstrations illustrate the mechanism; the broader evidence comes from the task evaluation. Microsoft’s project page describes both.
The timing problem has two sides. A slow planner can leave a character defenseless, but an entirely reactive controller can survive a fight without remembering why it entered the area or what a quest requires next. Veloren combines combat with dialogue, trading, crafting, skill selection and map navigation. Immediate survival and long-term progress need different kinds of decisions.
My view is that this is the most useful way to read JoyTwin: as a system for coordinating decisions with different time horizons. A better plan has limited value if the agent loses control while producing it.
How JoyTwin Splits Planning From Execution
JoyTwin’s published architecture has three connected parts: visual perception and working memory, concurrent planning and action, and experience carried across episodes.
| Component | Published role | Practical purpose |
|---|---|---|
| Visual perception and working memory | Reads screen frames and heads-up display cues, maintaining incremental memory | Gives the agent a working account of what it can see |
| Plan loop | Produces high-level guidance for quest stages | Keeps longer objectives in view |
| Act loop | Continuously controls movement, dodging and combat | Responds while planning remains in progress |
| Cross-episode experience | Retains rules and reusable skills from verified outcomes | Makes later attempts benefit from earlier play |
The Plan loop is the planner; the Act loop is the action executive. They exchange guidance and feedback without forcing execution to wait for every planning turn. Microsoft describes them as operating “without a turn barrier.” The architecture description does not specify exact control frequencies, hardware requirements or a complete model configuration, so it supports an explanation of the system design, not a reproducible implementation guide.
Asynchronous operation means the planner and controller can progress independently. The planner might be reasoning about the next stage of a quest while the action executive handles the enemy currently in front of the character. Feedback from execution can then inform further planning.
The screen-only interface matters, too. JoyTwin must interpret visible information instead of querying an authoritative internal record of enemy health, inventory or quest state. That makes maintaining useful working memory part of the challenge. It also keeps the interaction channel recognizably close to what a person uses, although that alone does not make the agent’s abilities equivalent to a human player’s.
The architectural benefit is straightforward: slow reasoning no longer has to impose an equally long interruption in control. It does not follow that reasoning becomes instantaneous or that every control decision is correct.
What Microsoft’s Benchmark Results Show
Microsoft evaluates JoyTwin against five visual-agent baselines: Tool-Calling Agent, DeepSeek Harness, SmolAgents, GamingAgent and GameScaling. The benchmark contains 11 Basic Action & Survival tasks and 14 Open-World RPG Missions, with each method evaluated over a three-episode horizon per task.
The published results are:
| System or stage | Evaluation horizon | Tasks solved | Completion rate |
|---|---|---|---|
| Strongest listed baseline overall | Three episodes | 7 of 25 | 28% |
| JoyTwin, first episode | One episode | 14 of 25 | 56% |
| JoyTwin, with accumulated experience | Three episodes | 19 of 25 | 76% |
JoyTwin’s first-episode result breaks down into eight of 11 basic tasks and six of 14 RPG missions. After three episodes, those totals rise to 10 basic tasks and nine RPG missions.
The mission split is particularly revealing. Microsoft reports that Tool-Calling Agent, GamingAgent and GameScaling completed none of the 14 RPG missions. DeepSeek Harness and SmolAgents each completed one. JoyTwin’s nine completed missions after three episodes therefore represent a larger separation on extended, multistage work than the overall score alone conveys.
Several distinctions keep those numbers in proportion:
- The improvement across JoyTwin episodes is 20 percentage points, from 56% to 76%, representing five additional completed tasks.
- The approximately 2.7× comparison is about tasks solved, calculated from 19 divided by seven. It is not a claim that JoyTwin runs every task 2.7 times faster.
- Six tasks remain unsolved after three episodes, including five of the 14 RPG missions.
The results are consistent with the proposed benefit of maintaining control during deliberation. They do not isolate exactly how much of the advantage comes from concurrency, memory, reusable routines or other differences between systems. That would require controlled comparisons that vary those components separately.
I would give the long-horizon mission results more attention than the headline multiplier. They directly address the project’s central challenge: staying effective in the immediate situation while preserving progress toward a larger objective.
How Gameplay Becomes a Persistent Playbook
JoyTwin’s second mechanism operates across attempts. During play, it binds and composes multistep action routines. Across episodes, Microsoft says it distills verified gameplay outcomes into persistent playbook rules and reusable skills.
That description gives “self-evolving” a concrete meaning here:
- The agent attempts a task and observes the outcome.
- Useful lessons and action routines are derived from gameplay.
- Persistent memory carries those lessons into subsequent episodes.
The published description centers on retained experience and reusable behavior. It should not be read as evidence of continuous neural-network weight updates or unrestricted self-modification.
Microsoft’s Ogre bounty example makes the distinction tangible. JoyTwin initially attacks a 320-health-point Ogre with a novice sword dealing approximately 0.6–1.5 health points of damage per hit. Even at the stated upper damage value, removing 320 health would require at least 214 successful hits, calculated by rounding up 320 divided by 1.5. That simple arithmetic helps explain why persisting with the same direct attack is a poor strategy.
After dying in its first attempt, Microsoft says JoyTwin learns to hit the Ogre to qualify for quest credit, then lure it toward a high-level town guard. The guard helps resolve the fight, allowing the agent to claim the 1,000-coin bounty. describes a change in strategy based on the environment’s mechanics.
Frequently Asked Questions
4 questions
1What is Microsoft Research’s JoyTwin?
JoyTwin is a research agent that plays the open-source 3D RPG Veloren using screen observations and standard mouse-and-keyboard inputs.
