Train the AI

Train and export

Run training with PPO and self-play, follow it with charts, evaluate and take the model to your room.

Running the training

bash
python treinar.py --passos 5000000 --envs 4

What shows up in the terminal (salvo = saved, passos = steps, estágio = stage, currículo = curriculum):

text
[salvo] 100000 passos, estágio 1
[currículo] passou para o estágio 2 (822320 passos)
[currículo] passou para o estágio 3 (870212 passos)
...
OptionDefaultWhat it does
--passos5 millionHow many steps to train this round
--envs4Parallel simulations (use 2, 4 or 8)
--visaoagoraHow the AI sees the game; it must match the room (why)
--novo—Starts from scratch (the old checkpoint is ignored, not deleted)
--salvar100 thousandEvery how many steps it saves a checkpoint
--rival-a-cada250 thousandEvery how many steps the current AI joins the opponents' "league"

You can stop it with Ctrl+C whenever you want: it saves before exiting. Running python treinar.py again resumes where it stopped (the curriculum stage is kept in checkpoints/estado.json).

How long it takes

Measured on an ordinary PC with --envs 4 (~750 steps/s):

MilestoneStepsTime
Leaves stage 1 (finds the ball and scores with no goalkeeper)~820 thousand~18 min
Leaves stage 2 (idle opponent in the way)~870 thousand~1 min later
Against the strongest script: 78 goals, 21 conceded in 100 plays2 million~45 min
Published model: 95 goals, 2 conceded182 milliondays (ours ran on several machines)

The numbers above come from a real training run with the kit, on an ordinary PC. Yours may vary a bit (the luck of the first plays changes the path), but the order of magnitude is this.

Following it with charts

bash
tensorboard --logdir checkpoints/tb

Open the address it shows (usually http://localhost:6006):

ChartWhat to expect
bonk/gols (goals)Rises to the target and drops when the stage changes (the exercise got harder). Normal
bonk/sofridos (conceded)In stage 3, it should drop over time
bonk/estagio (stage)1 → 2 → 3
rollout/ep_rew_meanThe average score per play. Rises slowly
train/entropy_lossHow much the AI still "experiments". It keeps decreasing; if it collapses early, it stopped exploring

Evaluating

bash
python avaliar.py checkpoints/policy.json --lances 200

It plays against the strongest script, deciding exactly like the room (axis by axis, with the chosen delay). For reference, the published model gives (gols = goals, sofridos = conceded, saldo = balance):

text
$ python avaliar.py football-1x1.policy.json --lances 100
100 lances: 95 gols, 2 sofridos, 3 sem gol (saldo +0.93 por lance)

Taking it to the room

Training already writes checkpoints/policy.json at every checkpoint. Just copy it to the room's folder and use it instead of football-1x1.policy.json (or point IA_MODELO to it). Remember to use the same --visao in the room as in training.

The export takes only the part of the network that decides (the value network is only useful for training) and stores the weights as float32 in base64:

python
# politica.py
"""Exports the trained network to `policy.json` (the format the room runs in JavaScript) and runs it back in NumPy
(to use old versions as self-play opponents without loading another PyTorch)."""
from __future__ import annotations

import base64
import time

import numpy as np

from .decisao import decide
from .features import ACTION_TABLE, N_ACTIONS, version_for_size

FORMATO = "bonktools-rl-policy/1"


def exportar(model) -> dict:
    """PPO (stable-baselines3 MlpPolicy) -> policy.json dict. Only the part that decides (the value network stays behind)."""
    import torch

    p = model.policy
    lineares = [m for m in p.mlp_extractor.policy_net if isinstance(m, torch.nn.Linear)] + [p.action_net]
    ativacoes = {type(m).__name__ for m in p.mlp_extractor.policy_net if not isinstance(m, torch.nn.Linear)}
    if ativacoes != {"Tanh"}:
        raise ValueError(f"ativação inesperada: {ativacoes} (o formato só conhece Tanh)")

    def b64(t) -> str:  # little-endian float32 in base64: exact and ~4x smaller than text
        return base64.b64encode(t.detach().cpu().numpy().astype("<f4").tobytes()).decode("ascii")

    camadas = [{"in": l.in_features, "out": l.out_features, "w": b64(l.weight), "b": b64(l.bias)} for l in lineares]
    return {
        "format": FORMATO,
        "obsSize": camadas[0]["in"],
        "obsVersion": version_for_size(camadas[0]["in"]),
        "actionTable": ACTION_TABLE,
        "activation": "tanh",
        "layers": camadas,
        "timesteps": int(model.num_timesteps),
        "exportedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    }


class PoliticaNP:
    """Runs a policy.json in NumPy: tanh in the hidden layers, linear in the last one (18 logits)."""

    def __init__(self, data: dict):
        if data.get("format") != FORMATO:
            raise ValueError(f"formato desconhecido: {data.get('format')}")
        self.obs_version = int(data["obsVersion"])
        self.timesteps = int(data.get("timesteps", 0))
        self.camadas = []
        for c in data["layers"]:
            w = np.frombuffer(base64.b64decode(c["w"]), dtype="<f4").reshape(c["out"], c["in"])
            b = np.frombuffer(base64.b64decode(c["b"]), dtype="<f4")
            self.camadas.append((w.astype(np.float64), b.astype(np.float64)))
        if self.camadas[-1][0].shape[0] != N_ACTIONS:
            raise ValueError("a rede não termina em 18 ações")

    def logits(self, obs: np.ndarray) -> np.ndarray:
        x = np.asarray(obs, dtype=np.float64)
        for i, (w, b) in enumerate(self.camadas):
            x = w @ x + b
            if i < len(self.camadas) - 1:
                x = np.tanh(x)
        return x

    def agir(self, obs: np.ndarray, anterior: int | None = None) -> int:
        """Axis-by-axis decision (the same as the room's), not plain argmax."""
        return decide(self.logits(obs), anterior)

The full training script

python
# treinar.py
"""Trains the 1v1 football AI with PPO (stable-baselines3) and self-play.

    python treinar.py                     # starts from scratch or resumes from checkpoints/latest.zip
    python treinar.py --passos 20000000   # how many steps to train this round
    python treinar.py --envs 4            # parallel simulations (1 Node process each; use 2, 4 or 8)
    python treinar.py --novo              # ignores the checkpoint (deletes nothing)
    python treinar.py --visao atrasada    # for rooms WITHOUT the "eyes" (see the docs); default: agora (now)
    tensorboard --logdir checkpoints/tb   # charts

Every --salvar steps it writes checkpoints/latest.zip, checkpoints/estado.json (curriculum stage) and
checkpoints/policy.json (the model ready for the room).
"""
from __future__ import annotations

import argparse
import json
from collections import deque
from pathlib import Path

import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.vec_env import DummyVecEnv

from bonk_ia.ambiente import BonkFootballEnv, Liga
from bonk_ia.politica import exportar

PASTA = Path("checkpoints")
# To advance a stage: goal rate over the last 400 plays.
META = {1: 0.7, 2: 0.6}

# Based on the RLGym-PPO-Guide: lr 2e-4, few epochs, big batches, entropy 0.01. 256x256 network with tanh.
PPO_ARGS = dict(
    learning_rate=2e-4,
    n_steps=1024,  # per environment
    batch_size=2048,
    n_epochs=4,
    gamma=0.99,  # at 10 decisions/s, looks ~10 s ahead
    gae_lambda=0.95,
    clip_range=0.2,
    ent_coef=0.01,
    policy_kwargs=dict(net_arch=dict(pi=[256, 256], vf=[256, 256])),
)


class VecEmParalelo(DummyVecEnv):
    """Sends the step to ALL bridges before reading the responses: the simulations (Node processes) run together,
    with a single Python process (and a single PyTorch in memory)."""

    def step_wait(self):
        for env, acao in zip(self.envs, self.actions):
            env.step_send(acao)
        for i, env in enumerate(self.envs):
            obs, r, terminou, cortou, info = env.step_recv()
            self.buf_rews[i] = r
            self.buf_dones[i] = terminou or cortou
            info = dict(info)
            info["TimeLimit.truncated"] = cortou and not terminou
            if self.buf_dones[i]:
                info["terminal_observation"] = obs
                obs, self.reset_infos[i] = env.reset()
            self.buf_infos[i] = info
            self._save_obs(i, obs)
        return self._obs_from_buf(), np.copy(self.buf_rews), np.copy(self.buf_dones), list(self.buf_infos)


class Curriculo(BaseCallback):
    """Advances the stage by goal rate, saves checkpoints and feeds the self-play league."""

    def __init__(self, estagio: int, liga: Liga, salvar: int, rival_a_cada: int):
        super().__init__()
        self.estagio = estagio
        self.liga = liga
        self.salvar_a_cada = salvar
        self.rival_a_cada = rival_a_cada
        self.janela: deque[int] = deque(maxlen=400)
        self.prox_salvar = 0
        self.prox_rival = 0

    def _on_training_start(self) -> None:
        # when resuming a run, count from the steps the model already has
        self.prox_salvar = self.num_timesteps + self.salvar_a_cada
        self.prox_rival = self.num_timesteps + self.rival_a_cada

    def _on_step(self) -> bool:
        for info in self.locals["infos"]:
            if "gol" in info and info.get("estagio") == self.estagio:
                self.janela.append(info["gol"])
        if len(self.janela) == self.janela.maxlen:
            gols = sum(g == 1 for g in self.janela) / len(self.janela)
            sofridos = sum(g == -1 for g in self.janela) / len(self.janela)
            self.logger.record("bonk/gols", gols)
            self.logger.record("bonk/sofridos", sofridos)
            self.logger.record("bonk/estagio", self.estagio)
            if self.estagio in META and gols >= META[self.estagio]:
                self.estagio += 1
                self.janela.clear()
                self.training_env.env_method("set_estagio", self.estagio)
                print(f"[currículo] passou para o estágio {self.estagio} ({self.num_timesteps} passos)")
        if self.estagio == 3 and self.num_timesteps >= self.prox_rival:
            self.prox_rival = self.num_timesteps + self.rival_a_cada
            self.liga.adicionar(exportar(self.model))  # the current version becomes an opponent for the next ones
        if self.num_timesteps >= self.prox_salvar:
            self.prox_salvar = self.num_timesteps + self.salvar_a_cada
            self.salvar()
        return True

    def salvar(self) -> None:
        PASTA.mkdir(exist_ok=True)
        self.model.save(PASTA / "latest.zip")
        (PASTA / "estado.json").write_text(json.dumps({"estagio": self.estagio, "passos": self.num_timesteps}))
        (PASTA / "policy.json").write_text(json.dumps(exportar(self.model), separators=(",", ":")))
        print(f"[salvo] {self.num_timesteps} passos, estágio {self.estagio}")


def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--passos", type=int, default=5_000_000)
    ap.add_argument("--envs", type=int, default=4)
    ap.add_argument("--novo", action="store_true")
    ap.add_argument("--salvar", type=int, default=100_000)
    ap.add_argument("--rival-a-cada", type=int, default=250_000)
    ap.add_argument("--visao", choices=["agora", "atrasada"], default="agora")
    a = ap.parse_args()

    continuar = (PASTA / "latest.zip").exists() and not a.novo
    estagio = json.loads((PASTA / "estado.json").read_text())["estagio"] if continuar else 1
    liga = Liga()
    venv = VecEmParalelo([lambda i=i: BonkFootballEnv(estagio=estagio, liga=liga, seed=1000 + i, visao=a.visao) for i in range(a.envs)])
    if continuar:
        model = PPO.load(PASTA / "latest.zip", env=venv, device="cpu")
        print(f"continuando de {model.num_timesteps} passos (estágio {estagio})")
    else:
        model = PPO("MlpPolicy", venv, device="cpu", tensorboard_log=str(PASTA / "tb"), verbose=0, **PPO_ARGS)
    cb = Curriculo(estagio, liga, a.salvar, a.rival_a_cada)
    try:
        model.learn(total_timesteps=a.passos, callback=cb, reset_num_timesteps=not continuar, progress_bar=False)
    finally:
        cb.salvar()
        venv.close()


if __name__ == "__main__":
    main()

The hyperparameters

ParameterValueWhy
learning_rate2e-4Default from the PPO guide for Rocket League (RLGym); higher gets unstable
n_steps × envs1024 × 4~4 thousand steps (7 minutes of play) per update
batch_size, n_epochs2048, 4Big batches and few passes: stable
gamma0.99At 10 decisions/s, the AI "sees" ~10 s ahead
ent_coef0.01Keeps the AI experimenting
Network256 × 256, tanhSmall enough to run in plain JS, big enough for the problem

Next challenges

Once the basics run, these are the improvements that paid off most for us (in order):

  1. Scenarios from real plays. Record your room's matches against humans (rec) and start training plays from the situations where the AI conceded. The bridge's reset accepts overrides with positions and velocities.
  2. An "explorer". Train a second AI whose only goal is to beat the main one, and use its versions as the main one's opponents. It finds the holes before the players do.
  3. Opponents that imitate humans, like a striker who shoots first-time at kickoff (most human kickoff goals come from that).