hybrid_aif.reference.raif.run#
The resumable multi-seed Mountain Car run, and its report and benchmark modes.
One chunk is CHUNK_STEPS pooled agent steps per seed, jitted and vmapped over
seeds: n_blocks × [train_every pooled collect steps → one train_step()].
Each seed collects from LANES independent environment streams that share its
parameters and pool its step count, so a block is train_every // lanes steps per lane.
Collection auto-resets in the scan and records the fixture format.
Between chunks, on the host: episodes are split out per lane at is_last (a chunk
boundary’s partial episode is carried), labelled by
label() and routed (success → both buffers, failure
→ M). Every world-model batch comes from M unless --alternate. Both buffers are
StreamBuffer rings of CAPACITY steps, and each
seed’s are seeded once from its oracle fixture through the same routing.
A checkpoint (every --checkpoint-every minutes and at shutdown, SIGINT/SIGTERM
included) holds every numeric carrier, the buffers, the keys, the counters and the partial
episodes, so --resume continues the run; bitwise continuation is not asserted.
<out>/scores.jsonlPer finished episode:
{step, seed, lane, return, length, success, stability, log_entropy, coverage}.stepis the pooled agent step it ended on;returnthe raw summed reward;stabilityR-S’sC;log_entropythe episode’s mean act-time policy entropy (act_entropy());coveragethe lane’s cumulative grid fraction (Coverage).<out>/metrics.jsonlPer chunk, per seed: the trainers’ losses and
probe()’s diagnostics.<out>/checkpoint/The resumable state.
<out>/run.jsonThe resolved configuration and the launching commit.
Modes: no flag = train; --report = the §7 table from scores.jsonl;
--benchmark = the throughput table (benchmark()):
python -m hybrid_aif.reference.raif.run --seeds 0,1,2,3 --steps 1000000 --out runs/mc_2d
python -m hybrid_aif.reference.raif.run --out runs/mc_2d --resume
python -m hybrid_aif.reference.raif.run --out runs/mc_2d --report
Design: docs/design/amortization.md P1′ chunk 2d;
docs/design/amortization-p0-addendum.md §§5–7.
Module Attributes
Pooled agent steps per device chunk: their |
|
Environment lanes per seed: their |
|
Their |
|
Their Mountain Car budget: 1M agent steps per seed, 4 seeds (§5). |
|
The probe batch the distillation gap is measured on — smaller than a training batch because it is a diagnostic paid every chunk. |
|
Test/smoke sizes. |
|
The collection carry an agent-step scan threads: |
|
The tag the act-time code sample's key is folded in with: large, so it cannot coincide with any |
|
Stream columns a resumed |
|
Their |
|
One pending-episode slot per seed per lane: the type the host's boundary carry is. |
Functions
|
Typed version of namedtuple. |
|
The act-time policy entropy |
|
|
|
The |
|
One host-split episode as a labelled |
|
The world-model assembly behind a declared System: |
|
Jitted steps/s for the full train cycle, one seed against |
|
Step 0: |
|
The CRSPP posterior over a batch's embeddings, as belief tuples. |
|
The agent's encoder output on a batch of observations: |
|
|
|
The belief carry an episode's first filtering step reads. |
|
One episode's steps spliced onto the stream: a new buffer, nothing mutated. |
|
|
|
|
|
|
|
|
|
Load an oracle-export fixture: |
|
The parity default's |
|
|
|
The jitted, seed-vmapped chunk: |
|
|
|
|
|
The recognition model's posterior states over a replay batch: |
|
Diagnostics on a fixed probe batch, per seed: |
|
An |
|
R-AIF's agent for state-based Mountain Car: |
|
|
|
Print the §7-comparable table from |
|
The λ-returns and critic values an |
|
Buffer routing: success → both buffers, failure → |
|
A uniform batch of |
|
Serialise everything numeric, then the sidecar, whose presence means the checkpoint is valid. |
|
The §7-comparable numbers for one seed, from |
|
Mean return / success rate / mean length over a list of episodes. |
|
Bring an action into its box by a stop-gradiented rescale, not a clip. |
|
|
|
Whether the episode reached the goal: its final |
|
A batched |
|
The run. |
|
One gradient step of all five parameter sets, in their order: |
Classes
|
Everything a training loop threads: the declared agent and its optimizer carry. |
|
Array base class for JAX |
The collection carry an agent-step scan threads: |
|
The resolved run configuration — written to |
|
What the JSON sidecar carries: the loop's position, not its arrays. |
|
Their |
|
|
One episode's per-step columns — the format the whole training side speaks. |
|
Optimizer state for an agent's gradient-learnable parts: |
|
A Mapping is a generic container for associating key/value pairs. |
|
Sparse-reward continuous Mountain Car. |
One seed's §7 row. |
|
One seed's entire numeric state, the thing a checkpoint holds; leaves are stacked over seeds. |
|
A chunk's recorded collection stream in the fixture format, one row per driver transition. |
|
|
A uniform replay buffer over a flat step stream. |
|
Static structure of a System: discrete, continuous or mixed. |
|
All numeric state of a System: generative tensors, designations and the closed-loop factors. |
- hybrid_aif.reference.raif.run.CHUNK_STEPS = 1024#
Pooled agent steps per device chunk: their
batch_steps, and 64 gradient steps. Episodes finished inside a chunk reach the buffers at its end (--chunk-steps).
- hybrid_aif.reference.raif.run.LANES = 4#
Environment lanes per seed: their
envs: {amount: 4}, four streams behind one agent with the step count pooled.--lanes 1collects from a single stream.
- hybrid_aif.reference.raif.run.CAPACITY = 500000#
Their
replay_size 5e5steps per buffer (§5);StreamBuffercounts steps.
- hybrid_aif.reference.raif.run.DEFAULT_STEPS = 1000000#
Their Mountain Car budget: 1M agent steps per seed, 4 seeds (§5). One counted step is one driver transition, reset and terminal rows included, so a 100-action episode counts 101.
- hybrid_aif.reference.raif.run.PROBE_BATCH = 4#
The probe batch the distillation gap is measured on — smaller than a training batch because it is a diagnostic paid every chunk. See
probe().
- hybrid_aif.reference.raif.run.SMALL = {'deter': 16, 'embed': 16, 'head_layers': 2, 'head_units': 32, 'horizon': 4, 'mlp_units': 16, 'num_classes': 6, 'num_vars': 6, 'units': 8}#
Test/smoke sizes. Not parity: –small exists so the resume round trip and the throughput ratio are testable in seconds. Their
tinyis the default.
- class hybrid_aif.reference.raif.run.SeedState[source]#
Bases:
NamedTupleOne seed’s entire numeric state, the thing a checkpoint holds; leaves are stacked over seeds.
params/learn_statecarry only the seed axis. The collection carriers carry a further leading lane axis (LANES).keyis per lane (env noise, action draw, reset);train_keyis per seed, one gradient step per seed however many lanes fed it. The CRSPP’s parameters live inparams.scoring, not in a field of their own.- params: SystemParams#
Alias for field number 0
- learn_state: LearnState#
Alias for field number 1
- collect_carry()[source]#
This seed’s collection carry: the tuple
make_collect_fn()threads.
- static __new__(
- _cls,
- params,
- learn_state,
- beliefs,
- env_state,
- incoming_reward,
- t_in_episode,
- succeeded,
- key,
- train_key,
Create new instance of SeedState(params, learn_state, beliefs, env_state, incoming_reward, t_in_episode, succeeded, key, train_key)
- class hybrid_aif.reference.raif.run.Stream[source]#
Bases:
NamedTupleA chunk’s recorded collection stream in the fixture format, one row per driver transition.
The columns are
hybrid_aif.train.buffers.EPISODE_FIELDSminuscan_self_imitate, plus the telemetry columnlog_entropy(dropped byas_collected()). Frommake_chunk_fn()the leaves are(seeds, lanes, chunk_steps // lanes, ...).obsis the row’s observation,rewardthe reward that produced it,actionthe raw sample drawn at it andis_successthe goal predicate on it (make_collect_fn()).- log_entropy: Array#
The act-time differential entropy of the distribution this row’s action was drawn from. Telemetry, not replay data, and computed without a key; its per-episode mean is their
stats/mean_log_entropy.
- static __new__(
- _cls,
- obs,
- action,
- reward,
- is_success,
- has_succeeded,
- is_first,
- is_last,
- log_entropy,
Create new instance of Stream(obs, action, reward, is_success, has_succeeded, is_first, is_last, log_entropy)
- hybrid_aif.reference.raif.run.CollectCarry#
The collection carry an agent-step scan threads:
SeedStateminus the trained things and the training key, leaves on a leading lane axis.
- hybrid_aif.reference.raif.run.episode_start_beliefs(system, params)[source]#
The belief carry an episode’s first filtering step reads.
The episode-start state the world model’s training restart uses (
initial_state(): the learned carry and the prior’s mode at it, a one-hot code), advanced one GRU step with a zero action, as theirobs_stepdoes atis_first. Pure; the collection scan hoists it and broadcasts it across lanes.Design:
docs/experiments/raif-mountain-car/execution-ledger-acting.mdD6.
- hybrid_aif.reference.raif.run.act_entropy(system, params, beliefs)[source]#
The act-time policy entropy
H[pi(a | s_t)]atbeliefs: a scalar.The entropy of the actor’s raw
Normalhead at the featureshabit_action()uses, summed over action dimensions. Takes no key and draws nothing. Its per-episode mean is theirstats/mean_log_entropy; the imaginedhybrid_aif.train.imagine.ImagineMetrics.policy_entropyis a different quantity.Design:
docs/experiments/raif-mountain-car/series-sweep.md§5.3.
- hybrid_aif.reference.raif.run.ACT_SAMPLE_KEY_TAG = 94444773#
The tag the act-time code sample’s key is folded in with: large, so it cannot coincide with any
split(k_act, n)[i]a planner might take from the same key (fold_in(k, i)equalssplit(k, n)[i]for smalli; seetrain.imagine.SCORING_KEY_TAG).
- hybrid_aif.reference.raif.run.make_collect_fn(system, env, steps, *, act_sampling=True)[source]#
(params, carry) → (carry, Stream)overstepssteps per lane. Pure.carry’s leaves have a leading lane axis of any width; the lanes are vmapped withparamsand the episode-start belief broadcast. The returnedStreamleaves are(steps, lanes, ...).Each lane-step is one transition of their driver. A row is
(obs_t, reward_t, flags_t, a_t)withreward_tcaused bya_{t−1}, so an episode ofTapplied actions isT + 1rows: a reset row with a real action at zero reward, and a terminal row whose observation is the final state, whose action is zeroed and after which the env is not stepped.rewardthe incoming reward of the previous step’s action; 0 on a reset row.
is_successthe goal predicate on this row’s observation;
has_succeededis its running OR.actionthe raw actor sample, zeroed on
is_last. The env receives it clipped and the transitionsoft_clip()-ed; the recorded column is raw.is_lastthe goal predicate on this row, or
t_in_episodereaching the time limit.
On
is_lastajnp.whereswaps in a fresh env state,episode_start_beliefs(), zero incoming reward andt_in_episode = 0, so episodes straddle chunk boundaries and the host recovers them from the flags. Withact_samplingthe recognized code is sampled (infer_states(key=)) and the actor,log_entropyand the transition read that sample; without, the lane acts on the mean-field posterior.Design:
docs/experiments/raif-mountain-car/execution-ledger-acting.mdD1–D6.
- hybrid_aif.reference.raif.run.make_chunk_fn(
- system,
- crspp,
- env,
- *,
- chunk_steps=1024,
- train_every=16,
- lanes=4,
- batch_size=16,
- horizon=8,
- world_lr=0.0001,
- actor_lr=3e-05,
- ensemble_lr=0.0001,
- sample_scoring=False,
- act_sampling=True,
The jitted, seed-vmapped chunk:
(state, positive, general, use_positive) → ….Returns
(state, stream, losses), the stream’s leaves(seeds, lanes, chunk_steps // lanes, ...)with each lane’s steps contiguous, and the losses’(seeds, n_blocks). The outer scan is the blocks, withuse_positive(the host’s schedule) as itsxs; the inner is thetrain_every // laneslane-steps, so a gradient step follows everytrain_everypooled steps.positive/generalare constant through the chunk.
- hybrid_aif.reference.raif.run.OPTIONAL_STREAM_COLUMNS = frozenset({'log_entropy'})#
Stream columns a resumed
pendingmay lack: telemetry, backfilled with NaN. A pending tail missing any other column is a corrupt checkpoint.
- hybrid_aif.reference.raif.run.split_stream(pending, chunk)[source]#
(finished episodes, the new pending tail): one lane’s stream split onis_last.An episode a chunk boundary cut is the tail here and the head of the next chunk’s
pending.
- hybrid_aif.reference.raif.run.COVERAGE_BINS = 10#
Their
bin_sizefor thegymsuite (wrappers.py:795): a 10 × 10 grid over(position, velocity), so 100 cells.
- class hybrid_aif.reference.raif.run.Coverage[source]#
Bases:
objectTheir
MeasureCoveragefor Mountain Car, one instance per lane.The observation box (position
[-1.2, 0.6], velocity[-0.07, 0.07]) is cut intoCOVERAGE_BINSper axis at the interior edges, the cell id isCOVERAGE_BINS · bin(position) + bin(velocity), and coverage is the fraction of cells visited. The counter is cumulative over the lane’s life, not per episode, and is carried across chunks and checkpoints. It reads recordedobson the host only.Design:
docs/experiments/raif-mountain-car/series-sweep.md§5.3.
- hybrid_aif.reference.raif.run.as_collected(cols)[source]#
One host-split episode as a labelled
Episode.can_self_imitateis zero as recorded;hybrid_aif.reference.raif.revise.label()then replaces it with the imitation mask.
- hybrid_aif.reference.raif.run.route(positive, general, ep, add_fn=<function extend>)[source]#
Buffer routing: success → both buffers, failure →
Monly. Fixture episodes included.
- hybrid_aif.reference.raif.run.alternate(a, b)[source]#
a, b, a, b, …— the paper’s strict M/M⁺ alternation as an iterator (A10 disposition 9). Not the parity default; seealternation().
- hybrid_aif.reference.raif.run.alternation(grad_index, n_blocks, ready)[source]#
The
--alternateschedule:(seeds, n_blocks)bool. Not the default.The paper’s strict
i % 2alternation, drawing fromMwhile a seed’sM⁺is notready. The parity default ism_only(), because the oracle’s alternation is inert.Design:
docs/design/amortization-debt-ledger.mdentry 16.
- hybrid_aif.reference.raif.run.m_only(n_seeds, n_blocks)[source]#
The parity default’s
use_positiveschedule: allFalse, every batch fromM.
- hybrid_aif.reference.raif.run.probe(system, crspp, params, batch, key, *, sample_scoring=False, act_sampling=True)[source]#
Diagnostics on a fixed probe batch, per seed:
(ret, critic, gap, value, mu_act, mu_train, sigma_act, sigma_train, mu_gap), batch means.gapis the distillation gapret − critic: the λ-return of the explicit rollout against theg_tailcritic’s value at the same states, both fromrollout_returns(). The actor is also read on two feature paths over the same windows: the training features, and the collection lane’s filter run from a cold start at each window’s first step (episode_start_beliefsatis_first,infer_states, keyed whenact_sampling, then the prior pushed by the recorded action).mu_*are its mean actions,sigma_*its per-dimension scales, andmu_gapthe mean absolute difference of the two means.Design:
docs/experiments/raif-mountain-car/actor-entropy-diagnostic.md§8.
- class hybrid_aif.reference.raif.run.Config[source]#
Bases:
objectThe resolved run configuration — written to
run.jsonand re-read on resume.- __init__(
- out,
- seeds=(0, 1, 2, 3),
- steps=1000000,
- chunk_steps=1024,
- train_every=16,
- lanes=4,
- batch_size=16,
- seq_len=64,
- capacity=500000,
- horizon=8,
- time_limit=100,
- probe_batch=4,
- checkpoint_every=15.0,
- metrics_every=1,
- small=False,
- seed_offset=0,
- fixtures=True,
- alternate=False,
- winit='normal',
- sample_scoring=False,
- act_sampling=True,
- hybrid_aif.reference.raif.run.build(cfg)[source]#
Step 0:
(system, crspp, env, state, buffers, probe_batch, seed_reports, horizon).Deterministic in the config alone, so
--resumerebuilds the shapes by calling it and then overwrites the leaves.horizonis resolved once, for both the agent andmake_chunk_fn().
- hybrid_aif.reference.raif.run.fixture_path(seed)[source]#
reference/raif/fixtures/oracle_mc_seed<N>.npz— the oracle’s own collect stream.
- class hybrid_aif.reference.raif.run.Counters[source]#
Bases:
NamedTupleWhat the JSON sidecar carries: the loop’s position, not its arrays.
- static __new__(_cls, step, chunk, grad_index, episodes, wall_seconds)#
Create new instance of Counters(step, chunk, grad_index, episodes, wall_seconds)
- hybrid_aif.reference.raif.run.save_checkpoint(cfg, state, buffers, pending, counters, coverage=None)[source]#
Serialise everything numeric, then the sidecar, whose presence means the checkpoint is valid.
Files land as
.tmpand are renamed, andcheckpoint.jsonis written last, so a process killed mid-checkpoint leaves the previous one intact.
- hybrid_aif.reference.raif.run.PendingGrid#
One pending-episode slot per seed per lane: the type the host’s boundary carry is.
- hybrid_aif.reference.raif.run.empty_pending(cfg)[source]#
[seeds][lanes]ofNone— nothing in flight. One place builds the grid.
- hybrid_aif.reference.raif.run.load_coverage(cfg, env)[source]#
[seeds][lanes]ofCoverage, restored from the sidecar if it is there.Without a
coveragekey the counters start empty.
- hybrid_aif.reference.raif.run.load_checkpoint(cfg, state, buffers)[source]#
(state, buffers, pending, counters)restored onto freshly built shapes.
- hybrid_aif.reference.raif.run.train(cfg, *, resume=False, log=<function _console>)[source]#
The run. Returns the final counters; writes everything else to
cfg.out.
- class hybrid_aif.reference.raif.run.SeedReport[source]#
Bases:
NamedTupleOne seed’s §7 row. Every field is a window, not an endpoint — §7’s ruling.
- static __new__(
- _cls,
- seed,
- episodes,
- steps,
- best_window,
- best_window_at,
- final_acr,
- last_500k,
- sr_window,
- sr_oracle,
- rs,
Create new instance of SeedReport(seed, episodes, steps, best_window, best_window_at, final_acr, last_500k, sr_window, sr_oracle, rs)
- hybrid_aif.reference.raif.run.seed_report(
- rows,
- seed,
- *,
- window=100,
- span=200000,
- tail=500000,
- offset=1.1,
- threshold=-100.0,
The §7-comparable numbers for one seed, from
scores.jsonlrows.best_windowThe best mean over complete tiled
span-step windows (the log reaching within 1% of a window’s end); a partial trailing window is not counted.final_acr§6’s ACR: the mean over the last
windowepisodes (an endpoint read).last_500kThe mean over episodes in the final
tailsteps.sr_window§6’s SR: the fraction of the last
windowepisodes reaching the goal.sr_oracleThe oracle table’s SR: the fraction of the last
tailsteps’ episodes returning abovethreshold, a different statistic from §6’s.rs§6’s R-S at the Mountain Car offset, over each logged episode’s
C(_stability()).
Design:
docs/design/amortization-p0-addendum.md§7.
- hybrid_aif.reference.raif.run.report(cfg, log=<function _console>)[source]#
Print the §7-comparable table from
scores.jsonl.
- hybrid_aif.reference.raif.run.benchmark(cfg, *, reps=3, log=<function _console>)[source]#
Jitted steps/s for the full train cycle, one seed against
len(seeds)vmapped.Measured at the run’s shapes with the run’s chunk function. Reports both rates, their ratio, the projected wall clock for
cfg.steps× seeds, and the collection half’s own time (collect_seconds) from the samemake_collect_fn().