Train the AI

What we learned

The mistakes and successes of months training and running an AI that plays against people, with the numbers.

Everything on this page was measured, almost always the hard way. Read it before changing a reward, a scenario or the room: it saves weeks.

1. The yardstick is the human, not the simulator

Three times the simulator said "it got better" and, against people, the AI got worse:

ChangeIn the simulatorAgainst humans
Strongly imitating humansGot worse(dropped before testing)
Fixed per-touch bonusNormal goalsStopped kicking (kicks near the ball: 24% → 2%)
Sophisticated rewards + "anti-circling"Better at almost everythingEasier to beat: reckless, easy to dribble past, defends badly

The order of trust we adopted:

  1. You playing against it (the most reliable).
  2. Statistics from the real matches recorded in the room.
  3. The AI in real situations (positions taken from the matches).
  4. The simulator (useful to rule out what's bad, not to prove something is good).

2. Perception delay is the biggest factor live

  • The automatic score runs 200 ms behind, on purpose. A model trained seeing the perfect game, playing with 200 ms of delay, lost all 20 test kickoffs to a striker who shoots first-time (without delay: it won all 20).
  • First fix: project the state (position + velocity × 0.2 s). It improved a lot.
  • Final fix: the "eyes", a second simulation without delay, and training while seeing the game the same way. Measured on real matches, decisions started matching the state from 0–100 ms before, and players felt the difference right away.

Moral: training and the room must see the game the same way. Any difference (delay, scale, a sign) makes the AI play badly without any error.

3. "Circling the ball" wasn't about the ball's size

The AI circled around the ball without touching it. We measured: its turning radius was the same as humans' (3.8 vs 3.6 units). The problem was dynamics: it chased where the ball was, overshot and curved back, and the delay made it all worse. The new observation numbers (ball velocity relative to it, distance to the walls) helped more than any reward.

4. Zigzag: decide axis by axis

With argmax (the highest-scoring action), the AI alternated nearly tied "diagonal up" and "diagonal down": 42 flips per minute in real matches, versus ~1 for humans. The fix was to decide each axis separately, with a margin for reversing direction, plus a small reward cost for flipping. It started walking straight.

5. Decide at the same pace as in training

We tried making the AI decide every frame (33 ms) in the room, to "react faster". Trained at 100 ms, it started sending keys that lasted a single frame and "fell over without knowing how to play". Reverted within minutes. Training pace = room pace.

6. The simulator doesn't have the right opponents

  • Scripts, older versions and even a human clone (a network trained to imitate human plays) are weak: the AI beat the clone 95–100% of the time. Humans won most real matches.
  • Nobody in the simulator dribbles like people do, so the simulator never showed that the AI was "easy to dribble past".
  • Human kickoff goals almost always come from whoever touches first and shoots first-time: we created an opponent that does exactly that.
  • The most exploited hole: the AI carries the ball into the attacking corner, it bounces off the wall next to the post, the AI is left behind and the human breaks away with an open field. Within a few days players learned the play and it went from 4–10% to 27% of human goals. Players learn fast: the AI needs scenarios taken from real matches to keep up.

7. Production

  • Leaking connections bring a room down slowly. WebRTC connections from players who left stayed open: CPU from 12% to 86% and memory from 259 to 472 MB in 40 minutes, with the bot "flying" from lag. Fixed in the library (0.2.7).
  • Swap the model without restarting the room (reread the file every minute); new code, only with the room empty.
  • Keep every version of the model. Several times we went back to an earlier model that played better against people.
  • A/B test per player: to compare two models, alternate the model per player (A, B, A, B in each person's matches), not per match. Alternating per match, with a queue, two players taking turns always got the same model each.

8. What we haven't solved yet

Left as a challenge for whoever carries on:

  • Dribbling and defending dribbles like humans.
  • Real 2v2 (passing, covering, goalkeeper): our 2v2 room used the 1v1 AI in attack and a rule-based goalkeeper.
  • An automatic yardstick that predicts the result against humans (ours never replaced testing by playing).

If you make progress on any of these, tell us on our Discord!