Train the AI
Train and export
Run training with PPO and self-play, follow it with charts, evaluate and take the model to your room.
Running the training
python treinar.py --passos 5000000 --envs 4What shows up in the terminal (salvo = saved, passos = steps, estágio = stage, currículo = curriculum):
[salvo] 100000 passos, estágio 1
[currículo] passou para o estágio 2 (822320 passos)
[currículo] passou para o estágio 3 (870212 passos)
...| Option | Default | What it does |
|---|---|---|
--passos | 5 million | How many steps to train this round |
--envs | 4 | Parallel simulations (use 2, 4 or 8) |
--visao | agora | How the AI sees the game; it must match the room (why) |
--novo | — | Starts from scratch (the old checkpoint is ignored, not deleted) |
--salvar | 100 thousand | Every how many steps it saves a checkpoint |
--rival-a-cada | 250 thousand | Every how many steps the current AI joins the opponents' "league" |
You can stop it with Ctrl+C whenever you want: it saves before exiting. Running python treinar.py again resumes where it stopped (the curriculum stage is kept in checkpoints/estado.json).
How long it takes
Measured on an ordinary PC with --envs 4 (~750 steps/s):
| Milestone | Steps | Time |
|---|---|---|
| Leaves stage 1 (finds the ball and scores with no goalkeeper) | ~820 thousand | ~18 min |
| Leaves stage 2 (idle opponent in the way) | ~870 thousand | ~1 min later |
| Against the strongest script: 78 goals, 21 conceded in 100 plays | 2 million | ~45 min |
| Published model: 95 goals, 2 conceded | 182 million | days (ours ran on several machines) |
The numbers above come from a real training run with the kit, on an ordinary PC. Yours may vary a bit (the luck of the first plays changes the path), but the order of magnitude is this.
Following it with charts
tensorboard --logdir checkpoints/tbOpen the address it shows (usually http://localhost:6006):
| Chart | What to expect |
|---|---|
bonk/gols (goals) | Rises to the target and drops when the stage changes (the exercise got harder). Normal |
bonk/sofridos (conceded) | In stage 3, it should drop over time |
bonk/estagio (stage) | 1 → 2 → 3 |
rollout/ep_rew_mean | The average score per play. Rises slowly |
train/entropy_loss | How much the AI still "experiments". It keeps decreasing; if it collapses early, it stopped exploring |
Evaluating
python avaliar.py checkpoints/policy.json --lances 200It plays against the strongest script, deciding exactly like the room (axis by axis, with the chosen delay). For reference, the published model gives (gols = goals, sofridos = conceded, saldo = balance):
$ python avaliar.py football-1x1.policy.json --lances 100
100 lances: 95 gols, 2 sofridos, 3 sem gol (saldo +0.93 por lance)Taking it to the room
Training already writes checkpoints/policy.json at every checkpoint. Just copy it to the room's folder and use it instead of football-1x1.policy.json (or point IA_MODELO to it). Remember to use the same --visao in the room as in training.
The export takes only the part of the network that decides (the value network is only useful for training) and stores the weights as float32 in base64:
# politica.py
"""Exports the trained network to `policy.json` (the format the room runs in JavaScript) and runs it back in NumPy
(to use old versions as self-play opponents without loading another PyTorch)."""
from __future__ import annotations
import base64
import time
import numpy as np
from .decisao import decide
from .features import ACTION_TABLE, N_ACTIONS, version_for_size
FORMATO = "bonktools-rl-policy/1"
def exportar(model) -> dict:
"""PPO (stable-baselines3 MlpPolicy) -> policy.json dict. Only the part that decides (the value network stays behind)."""
import torch
p = model.policy
lineares = [m for m in p.mlp_extractor.policy_net if isinstance(m, torch.nn.Linear)] + [p.action_net]
ativacoes = {type(m).__name__ for m in p.mlp_extractor.policy_net if not isinstance(m, torch.nn.Linear)}
if ativacoes != {"Tanh"}:
raise ValueError(f"ativação inesperada: {ativacoes} (o formato só conhece Tanh)")
def b64(t) -> str: # little-endian float32 in base64: exact and ~4x smaller than text
return base64.b64encode(t.detach().cpu().numpy().astype("<f4").tobytes()).decode("ascii")
camadas = [{"in": l.in_features, "out": l.out_features, "w": b64(l.weight), "b": b64(l.bias)} for l in lineares]
return {
"format": FORMATO,
"obsSize": camadas[0]["in"],
"obsVersion": version_for_size(camadas[0]["in"]),
"actionTable": ACTION_TABLE,
"activation": "tanh",
"layers": camadas,
"timesteps": int(model.num_timesteps),
"exportedAt": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
}
class PoliticaNP:
"""Runs a policy.json in NumPy: tanh in the hidden layers, linear in the last one (18 logits)."""
def __init__(self, data: dict):
if data.get("format") != FORMATO:
raise ValueError(f"formato desconhecido: {data.get('format')}")
self.obs_version = int(data["obsVersion"])
self.timesteps = int(data.get("timesteps", 0))
self.camadas = []
for c in data["layers"]:
w = np.frombuffer(base64.b64decode(c["w"]), dtype="<f4").reshape(c["out"], c["in"])
b = np.frombuffer(base64.b64decode(c["b"]), dtype="<f4")
self.camadas.append((w.astype(np.float64), b.astype(np.float64)))
if self.camadas[-1][0].shape[0] != N_ACTIONS:
raise ValueError("a rede não termina em 18 ações")
def logits(self, obs: np.ndarray) -> np.ndarray:
x = np.asarray(obs, dtype=np.float64)
for i, (w, b) in enumerate(self.camadas):
x = w @ x + b
if i < len(self.camadas) - 1:
x = np.tanh(x)
return x
def agir(self, obs: np.ndarray, anterior: int | None = None) -> int:
"""Axis-by-axis decision (the same as the room's), not plain argmax."""
return decide(self.logits(obs), anterior)The full training script
# treinar.py
"""Trains the 1v1 football AI with PPO (stable-baselines3) and self-play.
python treinar.py # starts from scratch or resumes from checkpoints/latest.zip
python treinar.py --passos 20000000 # how many steps to train this round
python treinar.py --envs 4 # parallel simulations (1 Node process each; use 2, 4 or 8)
python treinar.py --novo # ignores the checkpoint (deletes nothing)
python treinar.py --visao atrasada # for rooms WITHOUT the "eyes" (see the docs); default: agora (now)
tensorboard --logdir checkpoints/tb # charts
Every --salvar steps it writes checkpoints/latest.zip, checkpoints/estado.json (curriculum stage) and
checkpoints/policy.json (the model ready for the room).
"""
from __future__ import annotations
import argparse
import json
from collections import deque
from pathlib import Path
import numpy as np
from stable_baselines3 import PPO
from stable_baselines3.common.callbacks import BaseCallback
from stable_baselines3.common.vec_env import DummyVecEnv
from bonk_ia.ambiente import BonkFootballEnv, Liga
from bonk_ia.politica import exportar
PASTA = Path("checkpoints")
# To advance a stage: goal rate over the last 400 plays.
META = {1: 0.7, 2: 0.6}
# Based on the RLGym-PPO-Guide: lr 2e-4, few epochs, big batches, entropy 0.01. 256x256 network with tanh.
PPO_ARGS = dict(
learning_rate=2e-4,
n_steps=1024, # per environment
batch_size=2048,
n_epochs=4,
gamma=0.99, # at 10 decisions/s, looks ~10 s ahead
gae_lambda=0.95,
clip_range=0.2,
ent_coef=0.01,
policy_kwargs=dict(net_arch=dict(pi=[256, 256], vf=[256, 256])),
)
class VecEmParalelo(DummyVecEnv):
"""Sends the step to ALL bridges before reading the responses: the simulations (Node processes) run together,
with a single Python process (and a single PyTorch in memory)."""
def step_wait(self):
for env, acao in zip(self.envs, self.actions):
env.step_send(acao)
for i, env in enumerate(self.envs):
obs, r, terminou, cortou, info = env.step_recv()
self.buf_rews[i] = r
self.buf_dones[i] = terminou or cortou
info = dict(info)
info["TimeLimit.truncated"] = cortou and not terminou
if self.buf_dones[i]:
info["terminal_observation"] = obs
obs, self.reset_infos[i] = env.reset()
self.buf_infos[i] = info
self._save_obs(i, obs)
return self._obs_from_buf(), np.copy(self.buf_rews), np.copy(self.buf_dones), list(self.buf_infos)
class Curriculo(BaseCallback):
"""Advances the stage by goal rate, saves checkpoints and feeds the self-play league."""
def __init__(self, estagio: int, liga: Liga, salvar: int, rival_a_cada: int):
super().__init__()
self.estagio = estagio
self.liga = liga
self.salvar_a_cada = salvar
self.rival_a_cada = rival_a_cada
self.janela: deque[int] = deque(maxlen=400)
self.prox_salvar = 0
self.prox_rival = 0
def _on_training_start(self) -> None:
# when resuming a run, count from the steps the model already has
self.prox_salvar = self.num_timesteps + self.salvar_a_cada
self.prox_rival = self.num_timesteps + self.rival_a_cada
def _on_step(self) -> bool:
for info in self.locals["infos"]:
if "gol" in info and info.get("estagio") == self.estagio:
self.janela.append(info["gol"])
if len(self.janela) == self.janela.maxlen:
gols = sum(g == 1 for g in self.janela) / len(self.janela)
sofridos = sum(g == -1 for g in self.janela) / len(self.janela)
self.logger.record("bonk/gols", gols)
self.logger.record("bonk/sofridos", sofridos)
self.logger.record("bonk/estagio", self.estagio)
if self.estagio in META and gols >= META[self.estagio]:
self.estagio += 1
self.janela.clear()
self.training_env.env_method("set_estagio", self.estagio)
print(f"[currículo] passou para o estágio {self.estagio} ({self.num_timesteps} passos)")
if self.estagio == 3 and self.num_timesteps >= self.prox_rival:
self.prox_rival = self.num_timesteps + self.rival_a_cada
self.liga.adicionar(exportar(self.model)) # the current version becomes an opponent for the next ones
if self.num_timesteps >= self.prox_salvar:
self.prox_salvar = self.num_timesteps + self.salvar_a_cada
self.salvar()
return True
def salvar(self) -> None:
PASTA.mkdir(exist_ok=True)
self.model.save(PASTA / "latest.zip")
(PASTA / "estado.json").write_text(json.dumps({"estagio": self.estagio, "passos": self.num_timesteps}))
(PASTA / "policy.json").write_text(json.dumps(exportar(self.model), separators=(",", ":")))
print(f"[salvo] {self.num_timesteps} passos, estágio {self.estagio}")
def main() -> None:
ap = argparse.ArgumentParser()
ap.add_argument("--passos", type=int, default=5_000_000)
ap.add_argument("--envs", type=int, default=4)
ap.add_argument("--novo", action="store_true")
ap.add_argument("--salvar", type=int, default=100_000)
ap.add_argument("--rival-a-cada", type=int, default=250_000)
ap.add_argument("--visao", choices=["agora", "atrasada"], default="agora")
a = ap.parse_args()
continuar = (PASTA / "latest.zip").exists() and not a.novo
estagio = json.loads((PASTA / "estado.json").read_text())["estagio"] if continuar else 1
liga = Liga()
venv = VecEmParalelo([lambda i=i: BonkFootballEnv(estagio=estagio, liga=liga, seed=1000 + i, visao=a.visao) for i in range(a.envs)])
if continuar:
model = PPO.load(PASTA / "latest.zip", env=venv, device="cpu")
print(f"continuando de {model.num_timesteps} passos (estágio {estagio})")
else:
model = PPO("MlpPolicy", venv, device="cpu", tensorboard_log=str(PASTA / "tb"), verbose=0, **PPO_ARGS)
cb = Curriculo(estagio, liga, a.salvar, a.rival_a_cada)
try:
model.learn(total_timesteps=a.passos, callback=cb, reset_num_timesteps=not continuar, progress_bar=False)
finally:
cb.salvar()
venv.close()
if __name__ == "__main__":
main()The hyperparameters
| Parameter | Value | Why |
|---|---|---|
learning_rate | 2e-4 | Default from the PPO guide for Rocket League (RLGym); higher gets unstable |
n_steps × envs | 1024 × 4 | ~4 thousand steps (7 minutes of play) per update |
batch_size, n_epochs | 2048, 4 | Big batches and few passes: stable |
gamma | 0.99 | At 10 decisions/s, the AI "sees" ~10 s ahead |
ent_coef | 0.01 | Keeps the AI experimenting |
| Network | 256 × 256, tanh | Small enough to run in plain JS, big enough for the problem |
Next challenges
Once the basics run, these are the improvements that paid off most for us (in order):
- Scenarios from real plays. Record your room's matches against humans (rec) and start training plays from the situations where the AI conceded. The bridge's
resetacceptsoverrideswith positions and velocities. - An "explorer". Train a second AI whose only goal is to beat the main one, and use its versions as the main one's opponents. It finds the holes before the players do.
- Opponents that imitate humans, like a striker who shoots first-time at kickoff (most human kickoff goals come from that).