hybrid_aif.reference.raif.run#

The resumable multi-seed Mountain Car run, and its report and benchmark modes.

One chunk is CHUNK_STEPS pooled agent steps per seed, jitted and vmapped over seeds: n_blocks × [train_every pooled collect steps → one train_step()]. Each seed collects from LANES independent environment streams that share its parameters and pool its step count, so a block is train_every // lanes steps per lane. Collection auto-resets in the scan and records the fixture format.

Between chunks, on the host: episodes are split out per lane at is_last (a chunk boundary’s partial episode is carried), labelled by label() and routed (success → both buffers, failure → M). Every world-model batch comes from M unless --alternate. Both buffers are StreamBuffer rings of CAPACITY steps, and each seed’s are seeded once from its oracle fixture through the same routing.

A checkpoint (every --checkpoint-every minutes and at shutdown, SIGINT/SIGTERM included) holds every numeric carrier, the buffers, the keys, the counters and the partial episodes, so --resume continues the run; bitwise continuation is not asserted.

<out>/scores.jsonl

Per finished episode: {step, seed, lane, return, length, success, stability, log_entropy, coverage}. step is the pooled agent step it ended on; return the raw summed reward; stability R-S’s C; log_entropy the episode’s mean act-time policy entropy (act_entropy()); coverage the lane’s cumulative grid fraction (Coverage).

<out>/metrics.jsonl

Per chunk, per seed: the trainers’ losses and probe()’s diagnostics.

<out>/checkpoint/

The resumable state.

<out>/run.json

The resolved configuration and the launching commit.

Modes: no flag = train; --report = the §7 table from scores.jsonl; --benchmark = the throughput table (benchmark()):

python -m hybrid_aif.reference.raif.run --seeds 0,1,2,3 --steps 1000000 --out runs/mc_2d
python -m hybrid_aif.reference.raif.run --out runs/mc_2d --resume
python -m hybrid_aif.reference.raif.run --out runs/mc_2d --report

Design: docs/design/amortization.md P1′ chunk 2d; docs/design/amortization-p0-addendum.md §§5–7.

Module Attributes

CHUNK_STEPS

Pooled agent steps per device chunk: their batch_steps, and 64 gradient steps.

LANES

Environment lanes per seed: their envs: {amount: 4}, four streams behind one agent with the step count pooled.

CAPACITY

Their replay_size 5e5 steps per buffer (§5); StreamBuffer counts steps.

DEFAULT_STEPS

Their Mountain Car budget: 1M agent steps per seed, 4 seeds (§5).

PROBE_BATCH

The probe batch the distillation gap is measured on — smaller than a training batch because it is a diagnostic paid every chunk.

SMALL

Test/smoke sizes.

CollectCarry

The collection carry an agent-step scan threads: SeedState minus the trained things and the training key, leaves on a leading lane axis.

ACT_SAMPLE_KEY_TAG

The tag the act-time code sample's key is folded in with: large, so it cannot coincide with any split(k_act, n)[i] a planner might take from the same key (fold_in(k, i) equals split(k, n)[i] for small i; see train.imagine.SCORING_KEY_TAG).

OPTIONAL_STREAM_COLUMNS

Stream columns a resumed pending may lack: telemetry, backfilled with NaN.

COVERAGE_BINS

Their bin_size for the gym suite (wrappers.py:795): a 10 × 10 grid over (position, velocity), so 100 cells.

PendingGrid

One pending-episode slot per seed per lane: the type the host's boundary carry is.

Functions

NamedTuple(typename[, fields])

Typed version of namedtuple.

act_entropy(system, params, beliefs)

The act-time policy entropy H[pi(a | s_t)] at beliefs: a scalar.

alternate(a, b)

a, b, a, b, … — the paper's strict M/M⁺ alternation as an iterator (A10 disposition 9).

alternation(grad_index, n_blocks, ready)

The --alternate schedule: (seeds, n_blocks) bool.

as_collected(cols)

One host-split episode as a labelled Episode.

assemble(system, params[, roles])

The world-model assembly behind a declared System: (core, slot addresses).

benchmark(cfg, *[, reps, log])

Jitted steps/s for the full train cycle, one seed against len(seeds) vmapped.

build(cfg)

Step 0: (system, crspp, env, state, buffers, probe_batch, seed_reports, horizon).

crspp_recognize(system, params, embeds, *, ...)

The CRSPP posterior over a batch's embeddings, as belief tuples.

embeddings(system, params, obs)

The agent's encoder output on a batch of observations: (..., embed_dim).

empty_pending(cfg)

[seeds][lanes] of None — nothing in flight.

episode_start_beliefs(system, params)

The belief carry an episode's first filtering step reads.

extend(buffer, ep)

One episode's steps spliced onto the stream: a new buffer, nothing mutated.

fixture_path(seed)

reference/raif/fixtures/oracle_mc_seed<N>.npz — the oracle's own collect stream.

label(ep, **kwargs)

ep with both columns lookback_on_episode writes: ρ and the imitation mask.

load_checkpoint(cfg, state, buffers)

(state, buffers, pending, counters) restored onto freshly built shapes.

load_coverage(cfg, env)

[seeds][lanes] of Coverage, restored from the sidecar if it is there.

load_fixture(path)

Load an oracle-export fixture: (episodes, provenance).

m_only(n_seeds, n_blocks)

The parity default's use_positive schedule: all False, every batch from M.

main([argv])

make_chunk_fn(system, crspp, env, *[, ...])

The jitted, seed-vmapped chunk: (state, positive, general, use_positive) → ….

make_collect_fn(system, env, steps, *[, ...])

(params, carry) → (carry, Stream) over steps steps per lane.

parse([argv])

posterior_states(system, params, batch, *, key)

The recognition model's posterior states over a replay batch: (B, T, ...).

probe(system, crspp, params, batch, key, *)

Diagnostics on a fixed probe batch, per seed: (ret, critic, gap, value, mu_act, mu_train, sigma_act, sigma_train, mu_gap), batch means.

raif_episode(*, obs, action, reward, ...[, ...])

An Episode from their recorded columns: cont = 1 − is_success, the four task columns in extras (ρ zero until revise.label()).

raif_mountain_car(key, *[, obs_dim, ...])

R-AIF's agent for state-based Mountain Car: Agent(system, params, crspp, …, learn_state).

read_scores(path)

report(cfg[, log])

Print the §7-comparable table from scores.jsonl.

rollout_returns(system, params, ...[, ...])

The λ-returns and critic values an imagine() step would fit, without stepping.

route(positive, general, ep[, add_fn])

Buffer routing: success → both buffers, failure → M only.

sample_windows(buffer, key, batch_size)

A uniform batch of seq_len-step windows: leaves (batch_size, seq_len, ...).

save_checkpoint(cfg, state, buffers, ...[, ...])

Serialise everything numeric, then the sidecar, whose presence means the checkpoint is valid.

seed_report(rows, seed, *[, window, span, ...])

The §7-comparable numbers for one seed, from scores.jsonl rows.

seed_stats(episodes[, success])

Mean return / success rate / mean length over a list of episodes.

soft_clip(action, bounds)

Bring an action into its box by a stop-gradiented rescale, not a clip.

split_stream(pending, chunk)

(finished episodes, the new pending tail): one lane's stream split on is_last.

succeeded(ep)

Whether the episode reached the goal: its final has_succeeded.

to_world_batch(batch)

A batched Episode as a WorldBatch.

train(cfg, *[, resume, log])

The run.

train_step(agent, batch, *, key[, horizon, ...])

One gradient step of all five parameter sets, in their order: (agent, StepLosses).

Classes

Agent

Everything a training loop threads: the declared agent and its optimizer carry.

Array

Array base class for JAX

CollectCarry

The collection carry an agent-step scan threads: SeedState minus the trained things and the training key, leaves on a leading lane axis.

Config

The resolved run configuration — written to run.json and re-read on resume.

Counters

What the JSON sidecar carries: the loop's position, not its arrays.

Coverage

Their MeasureCoverage for Mountain Car, one instance per lane.

Episode

One episode's per-step columns — the format the whole training side speaks.

LearnState

Optimizer state for an agent's gradient-learnable parts: learn's explicit carry.

Mapping

A Mapping is a generic container for associating key/value pairs.

MountainCar

Sparse-reward continuous Mountain Car.

SeedReport

One seed's §7 row.

SeedState

One seed's entire numeric state, the thing a checkpoint holds; leaves are stacked over seeds.

Stream

A chunk's recorded collection stream in the fixture format, one row per driver transition.

StreamBuffer

A uniform replay buffer over a flat step stream.

System

Static structure of a System: discrete, continuous or mixed.

SystemParams

All numeric state of a System: generative tensors, designations and the closed-loop factors.

hybrid_aif.reference.raif.run.CHUNK_STEPS = 1024#

Pooled agent steps per device chunk: their batch_steps, and 64 gradient steps. Episodes finished inside a chunk reach the buffers at its end (--chunk-steps).

hybrid_aif.reference.raif.run.LANES = 4#

Environment lanes per seed: their envs: {amount: 4}, four streams behind one agent with the step count pooled. --lanes 1 collects from a single stream.

hybrid_aif.reference.raif.run.CAPACITY = 500000#

Their replay_size 5e5 steps per buffer (§5); StreamBuffer counts steps.

hybrid_aif.reference.raif.run.DEFAULT_STEPS = 1000000#

Their Mountain Car budget: 1M agent steps per seed, 4 seeds (§5). One counted step is one driver transition, reset and terminal rows included, so a 100-action episode counts 101.

hybrid_aif.reference.raif.run.PROBE_BATCH = 4#

The probe batch the distillation gap is measured on — smaller than a training batch because it is a diagnostic paid every chunk. See probe().

hybrid_aif.reference.raif.run.SMALL = {'deter': 16, 'embed': 16, 'head_layers': 2, 'head_units': 32, 'horizon': 4, 'mlp_units': 16, 'num_classes': 6, 'num_vars': 6, 'units': 8}#

Test/smoke sizes. Not parity: –small exists so the resume round trip and the throughput ratio are testable in seconds. Their tiny is the default.

class hybrid_aif.reference.raif.run.SeedState[source]#

Bases: NamedTuple

One seed’s entire numeric state, the thing a checkpoint holds; leaves are stacked over seeds.

params/learn_state carry only the seed axis. The collection carriers carry a further leading lane axis (LANES). key is per lane (env noise, action draw, reset); train_key is per seed, one gradient step per seed however many lanes fed it. The CRSPP’s parameters live in params.scoring, not in a field of their own.

params: SystemParams#

Alias for field number 0

learn_state: LearnState#

Alias for field number 1

beliefs: dict[str, Array]#

Alias for field number 2

env_state: tuple[Array, ...]#

Alias for field number 3

incoming_reward: Array#

Alias for field number 4

t_in_episode: Array#

Alias for field number 5

succeeded: Array#

Alias for field number 6

key: Array#

Alias for field number 7

train_key: Array#

Alias for field number 8

collect_carry()[source]#

This seed’s collection carry: the tuple make_collect_fn() threads.

static __new__(
_cls,
params,
learn_state,
beliefs,
env_state,
incoming_reward,
t_in_episode,
succeeded,
key,
train_key,
)#

Create new instance of SeedState(params, learn_state, beliefs, env_state, incoming_reward, t_in_episode, succeeded, key, train_key)

class hybrid_aif.reference.raif.run.Stream[source]#

Bases: NamedTuple

A chunk’s recorded collection stream in the fixture format, one row per driver transition.

The columns are hybrid_aif.train.buffers.EPISODE_FIELDS minus can_self_imitate, plus the telemetry column log_entropy (dropped by as_collected()). From make_chunk_fn() the leaves are (seeds, lanes, chunk_steps // lanes, ...). obs is the row’s observation, reward the reward that produced it, action the raw sample drawn at it and is_success the goal predicate on it (make_collect_fn()).

obs: Array#

Alias for field number 0

action: Array#

Alias for field number 1

reward: Array#

Alias for field number 2

is_success: Array#

Alias for field number 3

has_succeeded: Array#

Alias for field number 4

is_first: Array#

Alias for field number 5

is_last: Array#

Alias for field number 6

log_entropy: Array#

The act-time differential entropy of the distribution this row’s action was drawn from. Telemetry, not replay data, and computed without a key; its per-episode mean is their stats/mean_log_entropy.

static __new__(
_cls,
obs,
action,
reward,
is_success,
has_succeeded,
is_first,
is_last,
log_entropy,
)#

Create new instance of Stream(obs, action, reward, is_success, has_succeeded, is_first, is_last, log_entropy)

hybrid_aif.reference.raif.run.CollectCarry#

The collection carry an agent-step scan threads: SeedState minus the trained things and the training key, leaves on a leading lane axis.

hybrid_aif.reference.raif.run.episode_start_beliefs(system, params)[source]#

The belief carry an episode’s first filtering step reads.

The episode-start state the world model’s training restart uses (initial_state(): the learned carry and the prior’s mode at it, a one-hot code), advanced one GRU step with a zero action, as their obs_step does at is_first. Pure; the collection scan hoists it and broadcasts it across lanes.

Design: docs/experiments/raif-mountain-car/execution-ledger-acting.md D6.

hybrid_aif.reference.raif.run.act_entropy(system, params, beliefs)[source]#

The act-time policy entropy H[pi(a | s_t)] at beliefs: a scalar.

The entropy of the actor’s raw Normal head at the features habit_action() uses, summed over action dimensions. Takes no key and draws nothing. Its per-episode mean is their stats/mean_log_entropy; the imagined hybrid_aif.train.imagine.ImagineMetrics.policy_entropy is a different quantity.

Design: docs/experiments/raif-mountain-car/series-sweep.md §5.3.

hybrid_aif.reference.raif.run.ACT_SAMPLE_KEY_TAG = 94444773#

The tag the act-time code sample’s key is folded in with: large, so it cannot coincide with any split(k_act, n)[i] a planner might take from the same key (fold_in(k, i) equals split(k, n)[i] for small i; see train.imagine.SCORING_KEY_TAG).

hybrid_aif.reference.raif.run.make_collect_fn(system, env, steps, *, act_sampling=True)[source]#

(params, carry) → (carry, Stream) over steps steps per lane. Pure.

carry’s leaves have a leading lane axis of any width; the lanes are vmapped with params and the episode-start belief broadcast. The returned Stream leaves are (steps, lanes, ...).

Each lane-step is one transition of their driver. A row is (obs_t, reward_t, flags_t, a_t) with reward_t caused by a_{t−1}, so an episode of T applied actions is T + 1 rows: a reset row with a real action at zero reward, and a terminal row whose observation is the final state, whose action is zeroed and after which the env is not stepped.

reward

the incoming reward of the previous step’s action; 0 on a reset row.

is_success

the goal predicate on this row’s observation; has_succeeded is its running OR.

action

the raw actor sample, zeroed on is_last. The env receives it clipped and the transition soft_clip()-ed; the recorded column is raw.

is_last

the goal predicate on this row, or t_in_episode reaching the time limit.

On is_last a jnp.where swaps in a fresh env state, episode_start_beliefs(), zero incoming reward and t_in_episode = 0, so episodes straddle chunk boundaries and the host recovers them from the flags. With act_sampling the recognized code is sampled (infer_states(key=)) and the actor, log_entropy and the transition read that sample; without, the lane acts on the mean-field posterior.

Design: docs/experiments/raif-mountain-car/execution-ledger-acting.md D1–D6.

hybrid_aif.reference.raif.run.make_chunk_fn(
system,
crspp,
env,
*,
chunk_steps=1024,
train_every=16,
lanes=4,
batch_size=16,
horizon=8,
world_lr=0.0001,
actor_lr=3e-05,
ensemble_lr=0.0001,
sample_scoring=False,
act_sampling=True,
)[source]#

The jitted, seed-vmapped chunk: (state, positive, general, use_positive) → ….

Returns (state, stream, losses), the stream’s leaves (seeds, lanes, chunk_steps // lanes, ...) with each lane’s steps contiguous, and the losses’ (seeds, n_blocks). The outer scan is the blocks, with use_positive (the host’s schedule) as its xs; the inner is the train_every // lanes lane-steps, so a gradient step follows every train_every pooled steps. positive/general are constant through the chunk.

hybrid_aif.reference.raif.run.OPTIONAL_STREAM_COLUMNS = frozenset({'log_entropy'})#

Stream columns a resumed pending may lack: telemetry, backfilled with NaN. A pending tail missing any other column is a corrupt checkpoint.

hybrid_aif.reference.raif.run.split_stream(pending, chunk)[source]#

(finished episodes, the new pending tail): one lane’s stream split on is_last.

An episode a chunk boundary cut is the tail here and the head of the next chunk’s pending.

hybrid_aif.reference.raif.run.COVERAGE_BINS = 10#

Their bin_size for the gym suite (wrappers.py:795): a 10 × 10 grid over (position, velocity), so 100 cells.

class hybrid_aif.reference.raif.run.Coverage[source]#

Bases: object

Their MeasureCoverage for Mountain Car, one instance per lane.

The observation box (position [-1.2, 0.6], velocity [-0.07, 0.07]) is cut into COVERAGE_BINS per axis at the interior edges, the cell id is COVERAGE_BINS · bin(position) + bin(velocity), and coverage is the fraction of cells visited. The counter is cumulative over the lane’s life, not per episode, and is carried across chunks and checkpoints. It reads recorded obs on the host only.

Design: docs/experiments/raif-mountain-car/series-sweep.md §5.3.

__init__(env, bins=10, seen=None)[source]#
update(obs)[source]#

Count one episode’s rows in order, then return the coverage.

obs is (steps, 2), reset and terminal rows included.

hybrid_aif.reference.raif.run.as_collected(cols)[source]#

One host-split episode as a labelled Episode.

can_self_imitate is zero as recorded; hybrid_aif.reference.raif.revise.label() then replaces it with the imitation mask.

hybrid_aif.reference.raif.run.route(positive, general, ep, add_fn=<function extend>)[source]#

Buffer routing: success → both buffers, failure → M only. Fixture episodes included.

hybrid_aif.reference.raif.run.alternate(a, b)[source]#

a, b, a, b, … — the paper’s strict M/M⁺ alternation as an iterator (A10 disposition 9). Not the parity default; see alternation().

hybrid_aif.reference.raif.run.alternation(grad_index, n_blocks, ready)[source]#

The --alternate schedule: (seeds, n_blocks) bool. Not the default.

The paper’s strict i % 2 alternation, drawing from M while a seed’s M⁺ is not ready. The parity default is m_only(), because the oracle’s alternation is inert.

Design: docs/design/amortization-debt-ledger.md entry 16.

hybrid_aif.reference.raif.run.m_only(n_seeds, n_blocks)[source]#

The parity default’s use_positive schedule: all False, every batch from M.

hybrid_aif.reference.raif.run.probe(system, crspp, params, batch, key, *, sample_scoring=False, act_sampling=True)[source]#

Diagnostics on a fixed probe batch, per seed: (ret, critic, gap, value, mu_act, mu_train, sigma_act, sigma_train, mu_gap), batch means.

gap is the distillation gap ret − critic: the λ-return of the explicit rollout against the g_tail critic’s value at the same states, both from rollout_returns(). The actor is also read on two feature paths over the same windows: the training features, and the collection lane’s filter run from a cold start at each window’s first step (episode_start_beliefs at is_first, infer_states, keyed when act_sampling, then the prior pushed by the recorded action). mu_* are its mean actions, sigma_* its per-dimension scales, and mu_gap the mean absolute difference of the two means.

Design: docs/experiments/raif-mountain-car/actor-entropy-diagnostic.md §8.

class hybrid_aif.reference.raif.run.Config[source]#

Bases: object

The resolved run configuration — written to run.json and re-read on resume.

__init__(
out,
seeds=(0, 1, 2, 3),
steps=1000000,
chunk_steps=1024,
train_every=16,
lanes=4,
batch_size=16,
seq_len=64,
capacity=500000,
horizon=8,
time_limit=100,
probe_batch=4,
checkpoint_every=15.0,
metrics_every=1,
small=False,
seed_offset=0,
fixtures=True,
alternate=False,
winit='normal',
sample_scoring=False,
act_sampling=True,
)#
hybrid_aif.reference.raif.run.build(cfg)[source]#

Step 0: (system, crspp, env, state, buffers, probe_batch, seed_reports, horizon).

Deterministic in the config alone, so --resume rebuilds the shapes by calling it and then overwrites the leaves. horizon is resolved once, for both the agent and make_chunk_fn().

hybrid_aif.reference.raif.run.fixture_path(seed)[source]#

reference/raif/fixtures/oracle_mc_seed<N>.npz — the oracle’s own collect stream.

class hybrid_aif.reference.raif.run.Counters[source]#

Bases: NamedTuple

What the JSON sidecar carries: the loop’s position, not its arrays.

step: int#

Alias for field number 0

chunk: int#

Alias for field number 1

grad_index: int#

Alias for field number 2

episodes: tuple[int, ...]#

Alias for field number 3

wall_seconds: float#

Alias for field number 4

static __new__(_cls, step, chunk, grad_index, episodes, wall_seconds)#

Create new instance of Counters(step, chunk, grad_index, episodes, wall_seconds)

hybrid_aif.reference.raif.run.save_checkpoint(cfg, state, buffers, pending, counters, coverage=None)[source]#

Serialise everything numeric, then the sidecar, whose presence means the checkpoint is valid.

Files land as .tmp and are renamed, and checkpoint.json is written last, so a process killed mid-checkpoint leaves the previous one intact.

hybrid_aif.reference.raif.run.PendingGrid#

One pending-episode slot per seed per lane: the type the host’s boundary carry is.

alias of list[list[dict[str, ndarray] | None]]

hybrid_aif.reference.raif.run.empty_pending(cfg)[source]#

[seeds][lanes] of None — nothing in flight. One place builds the grid.

hybrid_aif.reference.raif.run.load_coverage(cfg, env)[source]#

[seeds][lanes] of Coverage, restored from the sidecar if it is there.

Without a coverage key the counters start empty.

hybrid_aif.reference.raif.run.load_checkpoint(cfg, state, buffers)[source]#

(state, buffers, pending, counters) restored onto freshly built shapes.

hybrid_aif.reference.raif.run.train(cfg, *, resume=False, log=<function _console>)[source]#

The run. Returns the final counters; writes everything else to cfg.out.

class hybrid_aif.reference.raif.run.SeedReport[source]#

Bases: NamedTuple

One seed’s §7 row. Every field is a window, not an endpoint — §7’s ruling.

seed: int#

Alias for field number 0

episodes: int#

Alias for field number 1

steps: int#

Alias for field number 2

best_window: float#

Alias for field number 3

best_window_at: str#

Alias for field number 4

final_acr: float#

Alias for field number 5

last_500k: float#

Alias for field number 6

sr_window: float#

Alias for field number 7

sr_oracle: float#

Alias for field number 8

rs: float#

Alias for field number 9

static __new__(
_cls,
seed,
episodes,
steps,
best_window,
best_window_at,
final_acr,
last_500k,
sr_window,
sr_oracle,
rs,
)#

Create new instance of SeedReport(seed, episodes, steps, best_window, best_window_at, final_acr, last_500k, sr_window, sr_oracle, rs)

hybrid_aif.reference.raif.run.seed_report(
rows,
seed,
*,
window=100,
span=200000,
tail=500000,
offset=1.1,
threshold=-100.0,
)[source]#

The §7-comparable numbers for one seed, from scores.jsonl rows.

best_window

The best mean over complete tiled span-step windows (the log reaching within 1% of a window’s end); a partial trailing window is not counted.

final_acr

§6’s ACR: the mean over the last window episodes (an endpoint read).

last_500k

The mean over episodes in the final tail steps.

sr_window

§6’s SR: the fraction of the last window episodes reaching the goal.

sr_oracle

The oracle table’s SR: the fraction of the last tail steps’ episodes returning above threshold, a different statistic from §6’s.

rs

§6’s R-S at the Mountain Car offset, over each logged episode’s C (_stability()).

Design: docs/design/amortization-p0-addendum.md §7.

hybrid_aif.reference.raif.run.report(cfg, log=<function _console>)[source]#

Print the §7-comparable table from scores.jsonl.

hybrid_aif.reference.raif.run.benchmark(cfg, *, reps=3, log=<function _console>)[source]#

Jitted steps/s for the full train cycle, one seed against len(seeds) vmapped.

Measured at the run’s shapes with the run’s chunk function. Reports both rates, their ratio, the projected wall clock for cfg.steps × seeds, and the collection half’s own time (collect_seconds) from the same make_collect_fn().