Train the AI

What the AI sees and how it earns points

Observation, actions, canonical axis, rewards and curriculum: the design choices that weigh most on the result.

Three choices define what your AI will learn: what it sees (observation), what it can do (actions) and what it is rewarded for (reward). This page explains the kit's choices and why each one is there.

Canonical axis: the AI always attacks to the right

Instead of learning to play on both sides, the AI learns one side. When it plays red (which attacks to the left), the kit mirrors the field before showing it (x becomes 2 × 33.18 − x, x velocities flip sign) and swaps left and right in the key it chooses. Half the work, twice the experience per situation.

The observation (39 numbers)

All numbers stay roughly between −1 and 1 (divided by half the field or by the top speed) and are clamped to ±2. Neural networks learn much better with numbers in that range.

#What
1–4My position and velocity
5–8Ball position and velocity
9–12Opponent position and velocity
13–16Ball relative to me and to the opponent
17–20Ball relative to the opponent's goal and to my goal
21–22My distance and the opponent's distance to the ball
23–24Can I kick now? Can the opponent?
25How long until the kickoff is released
26–29Opponent's goal posts and my goal line, from me
30–33Distance to the four walls
34–35Ball's distance to the top and bottom walls (bounces)
36Is the ball between me and the opponent's goal? (I can shoot)
37Am I between the ball and my goal? (I'm defending)
38–39Ball velocity relative to me (to brake and not overrun the ball)

Numbers 1 to 25 are "version 1" (the first models); 26 to 39 were added after we noticed the AI didn't "see" walls and posts and kept overrunning the ball. A clear gain.

The actions (18)

9 directions (still, ↑, ↗, →, ↘, ↓, ↙, ←, ↖) × {no kick, kick}. One decision every 3 frames (100 ms).

  • Why not let the AI press any key combination? Because many make no sense (left + right together) and only slow learning down.
  • Why 100 ms and not every frame? We tested deciding every frame in the room: the AI, trained at 100 ms, started sending keys that lasted 1 frame and "fell over without knowing how to play". Decide in the room at the same pace as in training.

The reward

At each decision, the AI gets a small score. In the end, PPO teaches it to maximize the sum of the scores.

TermWeight (stage 3)Why
Goal+1The goal. Dominates everything else
Goal conceded−1With −0.8 it barely bothered to defend
My velocity toward the ball0.005Helps early on to "find" the ball
Ball velocity toward the opponent's goal0.015Rewards taking the ball to the right place
Touch that speeds the ball toward the goal0.02 × accelerationCan be negative: touching toward its own goal costs
Flipping up ↔ down−0.005Without it, it zigzagged

Stages 1 and 2 use larger "reach the ball" weights and a fixed bonus per touch, so the AI discovers the basics quickly.

Traps that got us

The full code:

python
# recompensas.py
"""Reward per decision (10 per second), on the canonical axis (the agent always attacks to the right).

This is the "classic" set: the one that trained the AI that played best against humans at BonkTools.
  - VELOCITY terms, not position: standing still earns 0 (a bonus for "being near the ball" teaches it to park);
  - touch that speeds the ball toward the opponent's goal (can be negative: ball toward its own goal);
  - goal +1, goal conceded -1;
  - a tiny cost for flipping up/down between two decisions (without it the AI zigzags).
"""
from __future__ import annotations

import math
from dataclasses import dataclass

from .features import (ACTION_TABLE, BALL_MAX_SPEED, CENTER_Y, DOWN, GOAL_X, KICK_REACH, PLAYER_MAX_SPEED, UP,
                       body, canon)


@dataclass(frozen=True)
class Pesos:
    jogador_bola: float  # agent's velocity toward the ball
    bola_gol: float  # ball's velocity toward the opponent's goal
    toque: float  # fixed bonus per touch (only early on: later it turns into "farming" little touches)
    toque_gol: float  # touch weighted by how much it sped the ball toward the goal
    gol: float = 1.0
    sofrido: float = 1.0
    inverter: float = 0.005


PESOS = {
    1: Pesos(jogador_bola=0.01, bola_gol=0.02, toque=0.05, toque_gol=0.1),  # empty goal: learn to reach the ball and push
    2: Pesos(jogador_bola=0.005, bola_gol=0.02, toque=0.02, toque_gol=0.05),  # idle opponent in the way
    3: Pesos(jogador_bola=0.005, bola_gol=0.015, toque=0.0, toque_gol=0.02),  # real game: the goal dominates
}

TOQUE_DV = 1.5  # minimum change in ball velocity in one step to count as a touch (friction alone gives < 1)
TOQUE_INTERVALO = 5  # steps between fixed touch bonuses
TELEPORTE = 8.0  # nothing moves more than ~1.5 in one step; more than that is a reset (goal/restart)


def _unit(dx: float, dy: float) -> tuple[float, float]:
    n = math.hypot(dx, dy)
    return (dx / n, dy / n) if n > 1e-6 else (0.0, 0.0)


def toque(prev: dict, curr: dict, eu: int, outro: int, espelho: bool) -> float | None:
    """If `eu` touched the ball between `prev` and `curr`: the ball's acceleration toward the opponent's goal (-1..1). Otherwise None."""
    pbx, pby, pbxv, pbyv = canon(body(prev, "ball"), espelho)
    bx, by, bxv, byv = canon(body(curr, "ball"), espelho)
    dvx, dvy = bxv - pbxv, byv - pbyv
    if math.hypot(dvx, dvy) < TOQUE_DV:
        return None
    sx, sy, _, _ = canon(body(curr, "discs", eu), espelho)
    ox, oy, _, _ = canon(body(curr, "discs", outro), espelho)
    d_eu, d_outro = math.hypot(bx - sx, by - sy), math.hypot(bx - ox, by - oy)
    if d_eu > KICK_REACH + 1.0 or d_eu > d_outro:
        return None
    gx, gy = _unit(GOAL_X - pbx, CENTER_Y - pby)
    return max(-1.0, min(1.0, (dvx * gx + dvy * gy) / BALL_MAX_SPEED))


def teleportou(prev: dict, curr: dict, ids: tuple[int, ...]) -> bool:
    def andou(a: dict, b: dict) -> float:
        return math.hypot(a["x"] - b["x"], a["y"] - b["y"])

    if andou(body(prev, "ball"), body(curr, "ball")) > TELEPORTE:
        return True
    return any(andou(body(prev, "discs", i), body(curr, "discs", i)) > TELEPORTE for i in ids)


class Recompensa:
    def __init__(self, pesos: Pesos):
        self.p = pesos
        self.reset()

    def reset(self) -> None:
        self._desde_toque = TOQUE_INTERVALO
        self._vertical = 0

    def passo(self, prev: dict, curr: dict, eu: int, outro: int, espelho: bool, marcou: bool, sofreu: bool,
              acao: int) -> tuple[float, bool]:
        """Returns (reward, whether there was a touch)."""
        p = self.p
        r = 0.0
        bits = ACTION_TABLE[int(acao)]
        vertical = -1 if bits & UP else (1 if bits & DOWN else 0)
        if vertical * self._vertical == -1:
            r -= p.inverter
        self._vertical = vertical
        if marcou:
            r += p.gol
        if sofreu:
            r -= p.sofrido
        self._desde_toque += 1
        if teleportou(prev, curr, (eu, outro)):
            return r, False
        sx, sy, sxv, syv = canon(body(curr, "discs", eu), espelho)
        bx, by, bxv, byv = canon(body(curr, "ball"), espelho)
        ux, uy = _unit(bx - sx, by - sy)
        r += p.jogador_bola * (sxv * ux + syv * uy) / PLAYER_MAX_SPEED
        gx, gy = _unit(GOAL_X - bx, CENTER_Y - by)
        r += p.bola_gol * max(-1.0, min(1.0, (bxv * gx + byv * gy) / BALL_MAX_SPEED))
        t = toque(prev, curr, eu, outro, espelho)
        if t is not None:
            r += p.toque_gol * t
            if self._desde_toque >= TOQUE_INTERVALO:
                r += p.toque
            self._desde_toque = 0
        return r, t is not None

The curriculum

StageScenariosOpponentAdvances at
160% attack (ball in the opponent's half, AI close, almost always behind it), 40% random; no initial freezeIdle, far away (against the top wall)70% goals over 400 plays
240% kickoff, 30% random, 30% attackIdle, in the way60% goals
340% kickoff, 30% random, 30% defense (ball heading to the AI's goal, opponent behind it)50% script (skill 0.4–1), 50% an older version of the AI itself—

A play ends at the first goal (either side) or at the time limit (15, 30 and 45 s in stages 1, 2 and 3).

The scripted opponents

The kit's Perseguidor ("chaser") gets behind the ball, lines up with the goal and kicks. "Skill" controls how often it makes a wrong decision and how slowly it reacts. It matters at the start of stage 3, when the AI's older versions are still too weak.

The room's delay

In the room, the AI doesn't see the game exactly "now": there's the 100 ms pace and, without the "eyes", the score simulation's 200 ms delay. If training shows the perfect game and the room shows a delayed game, the AI plays badly with no visible error (with 200 ms, a model trained without delay lost every kickoff to a striker who shoots first-time).

That's why, on every play, the environment draws how many decisions behind the AI sees:

--visaoDrawn delayProjectionUse in the room
agora ("now", default)0 (75%) or 1 decision (25%)noWith the "eyes" (lagFrames: 0)
atrasada ("delayed")1, 2 or 3 decisions (10/60/30%)6 frames aheadScore tracker only + projectState

See Put the AI in your room for the room side.