Train the AI
What the AI sees and how it earns points
Observation, actions, canonical axis, rewards and curriculum: the design choices that weigh most on the result.
Three choices define what your AI will learn: what it sees (observation), what it can do (actions) and what it is rewarded for (reward). This page explains the kit's choices and why each one is there.
Canonical axis: the AI always attacks to the right
Instead of learning to play on both sides, the AI learns one side. When it plays red (which attacks to the left), the kit mirrors the field before showing it (x becomes 2 × 33.18 − x, x velocities flip sign) and swaps left and right in the key it chooses. Half the work, twice the experience per situation.
The observation (39 numbers)
All numbers stay roughly between −1 and 1 (divided by half the field or by the top speed) and are clamped to ±2. Neural networks learn much better with numbers in that range.
| # | What |
|---|---|
| 1–4 | My position and velocity |
| 5–8 | Ball position and velocity |
| 9–12 | Opponent position and velocity |
| 13–16 | Ball relative to me and to the opponent |
| 17–20 | Ball relative to the opponent's goal and to my goal |
| 21–22 | My distance and the opponent's distance to the ball |
| 23–24 | Can I kick now? Can the opponent? |
| 25 | How long until the kickoff is released |
| 26–29 | Opponent's goal posts and my goal line, from me |
| 30–33 | Distance to the four walls |
| 34–35 | Ball's distance to the top and bottom walls (bounces) |
| 36 | Is the ball between me and the opponent's goal? (I can shoot) |
| 37 | Am I between the ball and my goal? (I'm defending) |
| 38–39 | Ball velocity relative to me (to brake and not overrun the ball) |
Numbers 1 to 25 are "version 1" (the first models); 26 to 39 were added after we noticed the AI didn't "see" walls and posts and kept overrunning the ball. A clear gain.
The actions (18)
9 directions (still, ↑, ↗, →, ↘, ↓, ↙, ←, ↖) × {no kick, kick}. One decision every 3 frames (100 ms).
- Why not let the AI press any key combination? Because many make no sense (left + right together) and only slow learning down.
- Why 100 ms and not every frame? We tested deciding every frame in the room: the AI, trained at 100 ms, started sending keys that lasted 1 frame and "fell over without knowing how to play". Decide in the room at the same pace as in training.
The reward
At each decision, the AI gets a small score. In the end, PPO teaches it to maximize the sum of the scores.
| Term | Weight (stage 3) | Why |
|---|---|---|
| Goal | +1 | The goal. Dominates everything else |
| Goal conceded | −1 | With −0.8 it barely bothered to defend |
| My velocity toward the ball | 0.005 | Helps early on to "find" the ball |
| Ball velocity toward the opponent's goal | 0.015 | Rewards taking the ball to the right place |
| Touch that speeds the ball toward the goal | 0.02 × acceleration | Can be negative: touching toward its own goal costs |
| Flipping up ↔ down | −0.005 | Without it, it zigzagged |
Stages 1 and 2 use larger "reach the ball" weights and a fixed bonus per touch, so the AI discovers the basics quickly.
Traps that got us
The full code:
# recompensas.py
"""Reward per decision (10 per second), on the canonical axis (the agent always attacks to the right).
This is the "classic" set: the one that trained the AI that played best against humans at BonkTools.
- VELOCITY terms, not position: standing still earns 0 (a bonus for "being near the ball" teaches it to park);
- touch that speeds the ball toward the opponent's goal (can be negative: ball toward its own goal);
- goal +1, goal conceded -1;
- a tiny cost for flipping up/down between two decisions (without it the AI zigzags).
"""
from __future__ import annotations
import math
from dataclasses import dataclass
from .features import (ACTION_TABLE, BALL_MAX_SPEED, CENTER_Y, DOWN, GOAL_X, KICK_REACH, PLAYER_MAX_SPEED, UP,
body, canon)
@dataclass(frozen=True)
class Pesos:
jogador_bola: float # agent's velocity toward the ball
bola_gol: float # ball's velocity toward the opponent's goal
toque: float # fixed bonus per touch (only early on: later it turns into "farming" little touches)
toque_gol: float # touch weighted by how much it sped the ball toward the goal
gol: float = 1.0
sofrido: float = 1.0
inverter: float = 0.005
PESOS = {
1: Pesos(jogador_bola=0.01, bola_gol=0.02, toque=0.05, toque_gol=0.1), # empty goal: learn to reach the ball and push
2: Pesos(jogador_bola=0.005, bola_gol=0.02, toque=0.02, toque_gol=0.05), # idle opponent in the way
3: Pesos(jogador_bola=0.005, bola_gol=0.015, toque=0.0, toque_gol=0.02), # real game: the goal dominates
}
TOQUE_DV = 1.5 # minimum change in ball velocity in one step to count as a touch (friction alone gives < 1)
TOQUE_INTERVALO = 5 # steps between fixed touch bonuses
TELEPORTE = 8.0 # nothing moves more than ~1.5 in one step; more than that is a reset (goal/restart)
def _unit(dx: float, dy: float) -> tuple[float, float]:
n = math.hypot(dx, dy)
return (dx / n, dy / n) if n > 1e-6 else (0.0, 0.0)
def toque(prev: dict, curr: dict, eu: int, outro: int, espelho: bool) -> float | None:
"""If `eu` touched the ball between `prev` and `curr`: the ball's acceleration toward the opponent's goal (-1..1). Otherwise None."""
pbx, pby, pbxv, pbyv = canon(body(prev, "ball"), espelho)
bx, by, bxv, byv = canon(body(curr, "ball"), espelho)
dvx, dvy = bxv - pbxv, byv - pbyv
if math.hypot(dvx, dvy) < TOQUE_DV:
return None
sx, sy, _, _ = canon(body(curr, "discs", eu), espelho)
ox, oy, _, _ = canon(body(curr, "discs", outro), espelho)
d_eu, d_outro = math.hypot(bx - sx, by - sy), math.hypot(bx - ox, by - oy)
if d_eu > KICK_REACH + 1.0 or d_eu > d_outro:
return None
gx, gy = _unit(GOAL_X - pbx, CENTER_Y - pby)
return max(-1.0, min(1.0, (dvx * gx + dvy * gy) / BALL_MAX_SPEED))
def teleportou(prev: dict, curr: dict, ids: tuple[int, ...]) -> bool:
def andou(a: dict, b: dict) -> float:
return math.hypot(a["x"] - b["x"], a["y"] - b["y"])
if andou(body(prev, "ball"), body(curr, "ball")) > TELEPORTE:
return True
return any(andou(body(prev, "discs", i), body(curr, "discs", i)) > TELEPORTE for i in ids)
class Recompensa:
def __init__(self, pesos: Pesos):
self.p = pesos
self.reset()
def reset(self) -> None:
self._desde_toque = TOQUE_INTERVALO
self._vertical = 0
def passo(self, prev: dict, curr: dict, eu: int, outro: int, espelho: bool, marcou: bool, sofreu: bool,
acao: int) -> tuple[float, bool]:
"""Returns (reward, whether there was a touch)."""
p = self.p
r = 0.0
bits = ACTION_TABLE[int(acao)]
vertical = -1 if bits & UP else (1 if bits & DOWN else 0)
if vertical * self._vertical == -1:
r -= p.inverter
self._vertical = vertical
if marcou:
r += p.gol
if sofreu:
r -= p.sofrido
self._desde_toque += 1
if teleportou(prev, curr, (eu, outro)):
return r, False
sx, sy, sxv, syv = canon(body(curr, "discs", eu), espelho)
bx, by, bxv, byv = canon(body(curr, "ball"), espelho)
ux, uy = _unit(bx - sx, by - sy)
r += p.jogador_bola * (sxv * ux + syv * uy) / PLAYER_MAX_SPEED
gx, gy = _unit(GOAL_X - bx, CENTER_Y - by)
r += p.bola_gol * max(-1.0, min(1.0, (bxv * gx + byv * gy) / BALL_MAX_SPEED))
t = toque(prev, curr, eu, outro, espelho)
if t is not None:
r += p.toque_gol * t
if self._desde_toque >= TOQUE_INTERVALO:
r += p.toque
self._desde_toque = 0
return r, t is not NoneThe curriculum
| Stage | Scenarios | Opponent | Advances at |
|---|---|---|---|
| 1 | 60% attack (ball in the opponent's half, AI close, almost always behind it), 40% random; no initial freeze | Idle, far away (against the top wall) | 70% goals over 400 plays |
| 2 | 40% kickoff, 30% random, 30% attack | Idle, in the way | 60% goals |
| 3 | 40% kickoff, 30% random, 30% defense (ball heading to the AI's goal, opponent behind it) | 50% script (skill 0.4–1), 50% an older version of the AI itself | — |
A play ends at the first goal (either side) or at the time limit (15, 30 and 45 s in stages 1, 2 and 3).
The scripted opponents
The kit's Perseguidor ("chaser") gets behind the ball, lines up with the goal and kicks. "Skill" controls how often it makes a wrong decision and how slowly it reacts. It matters at the start of stage 3, when the AI's older versions are still too weak.
The room's delay
In the room, the AI doesn't see the game exactly "now": there's the 100 ms pace and, without the "eyes", the score simulation's 200 ms delay. If training shows the perfect game and the room shows a delayed game, the AI plays badly with no visible error (with 200 ms, a model trained without delay lost every kickoff to a striker who shoots first-time).
That's why, on every play, the environment draws how many decisions behind the AI sees:
--visao | Drawn delay | Projection | Use in the room |
|---|---|---|---|
agora ("now", default) | 0 (75%) or 1 decision (25%) | no | With the "eyes" (lagFrames: 0) |
atrasada ("delayed") | 1, 2 or 3 decisions (10/60/30%) | 6 frames ahead | Score tracker only + projectState |
See Put the AI in your room for the room side.