hybrid_aif.reference.raif.agent#

The assembled R-AIF agent, and its training step.

The five parameter sets of their training step and where they live:

θ world model

the declared slots (hybrid_aif.reference.raif.world)

group 1

ψ actor

SystemParams.policy_prior

group 2

χ critic + slow copy

scoring["g_tail"]

group 3

ω info-gain ensemble

scoring["model_uncertainty"]

group 3

φ CRSPP

scoring["cosine_sim_to_pref_rollout"]

group 3

Sizes are the oracle’s resolved config: the actor is 3 × 256, the critic and each ensemble member 2 × 32, and the world model is tiny.

Design: docs/experiments/raif-mountain-car/module-correspondence.md.

Module Attributes

ACTOR_LAYERS

The three head sizes, each from its own block of the oracle's resolved config (raif-paper/logs/oracle_state_mc_seed0/config.yaml).

ACTOR_UNITS

The three head sizes, each from its own block of the oracle's resolved config (raif-paper/logs/oracle_state_mc_seed0/config.yaml).

HEAD_LAYERS

The actor's size under its old name.

ENSEMBLE_SIZE

ig_models: 4 (resolved config; configs.yaml:102-106).

ACTOR_MINSTD

The actor head's std band, actor: {minstd: 0.1, maxstd: 1.0}: their head is Normal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), which MLPConditional's head="bounded" implements.

ACTOR_MAXSTD

The actor head's std band, actor: {minstd: 0.1, maxstd: 1.0}: their head is Normal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), which MLPConditional's head="bounded" implements.

ENSEMBLE_STD

The ensemble heads' σ, a constant: ig_head declares no minstd/maxstd, so their defaults (1.0, 1.0) give std ≡ 1.0 and the member NLL is a scaled MSE.

ACTION_LOW

Their action box, Box(-1, 1, (1,)), and so the magnitude of their rssm.action_clip: 1.0.

ACTION_HIGH

Their action box, Box(-1, 1, (1,)), and so the magnitude of their rssm.action_clip: 1.0.

BATCH_SIZE

Their Mountain Car training protocol (addendum §5), for the recipe's defaults.

WORLD_LR

Their optimizer groups (addendum §5): the world model and the CRSPP at adam 1e-4 (clip 1000), the actor and critic at adam 3e-5 (clip 100), the info-gain ensemble at adam 1e-4.

ACTOR_EPS

The rest of those optimizer lines, from the oracle's resolved config (logs/oracle_state_mc_seed0/config.yaml).

BC_SCALE

The self-imitation term's scale (configs.yaml:79; A10 disposition 8).

ACTENT

Paper Eq.

TERMS

The rows this agent mounts: their critic's value reward + sim − IG, closed at the horizon by the critic (paper Eq.

Functions

NamedTuple(typename[, fields])

Typed version of namedtuple.

act(agent, beliefs, key)

The habit's action at beliefs: System.act under planner="habit".

assemble(system, params[, roles])

The world-model assembly behind a declared System: (core, slot addresses).

crspp_rssm(*, key[, policy_len, deter, ...])

R-AIF's CRSPP prior-preference model: tiny_rssm without the action or the heads.

embeddings(system, params, obs)

The agent's encoder output on a batch of observations: (..., embed_dim).

fit_crspp(system, params, learn_state, ...)

One gradient step of preference learning: (crspp_params, learn_state, losses).

fit_ensemble(system, params, learn_state, ...)

One gradient step of ensemble learning: (params, learn_state, losses).

fit_world(system, params, learn_state, ...)

One gradient step of world-model learning: (params, learn_state, losses).

imagine(system, params, start_beliefs, ...)

One actor+critic gradient step on imagined rollouts: (params, learn_state, metrics).

init_of(winit)

The engine's init= for their winit ("default" | "normal").

raif_mountain_car(key, *[, obs_dim, ...])

R-AIF's agent for state-based Mountain Car: Agent(system, params, crspp, …, learn_state).

soft_clip(action, bounds)

Bring an action into its box by a stop-gradiented rescale, not a clip.

tiny_rssm(obs_dim, action_dim, *, key[, ...])

R-AIF's tiny world model as a (System, SystemParams) pair.

to_world_batch(batch)

A batched Episode as a WorldBatch.

train_step(agent, batch, *, key[, horizon, ...])

One gradient step of all five parameter sets, in their order: (agent, StepLosses).

transitions(deter, code, actions)

A batch of posterior states as the ensemble's (s_t, a_t) -> s_{t+1} examples.

Classes

Agent

Everything a training loop threads: the declared agent and its optimizer carry.

Array

Array base class for JAX

CRSPPLosses

The preference model's loss, decomposed.

EnsembleLosses

The ensemble's loss: the sum over members, and each member's own.

Episode

One episode's per-step columns — the format the whole training side speaks.

ImagineMetrics

Plain arrays, one scalar each — what a training loop logs and a test asserts.

LearnState

Optimizer state for an agent's gradient-learnable parts: learn's explicit carry.

MLPConditional

MLP over concatenated parents → diagonal-Gaussian child (see module docstring).

PredictorHead

An MLPPredictor plus the name of the loss that reads its output.

StepLosses

One training step's four loss reports, one per parameter set.

System

Static structure of a System: discrete, continuous or mixed.

SystemParams

All numeric state of a System: generative tensors, designations and the closed-loop factors.

WorldBatch

A batch of replay sequences, all shapes static.

WorldLosses

The objective, decomposed: every field a scalar mean over batch and time.

hybrid_aif.reference.raif.agent.ACTOR_LAYERS = 3#

The three head sizes, each from its own block of the oracle’s resolved config (raif-paper/logs/oracle_state_mc_seed0/config.yaml):

actor:   {layers: 3, units: 256, outscale: 1.0, act: silu, dist: normal}
critic:  {layers: 2, units:  32, outscale: 0.0, dist: symlog_disc, bins: 255}
ig_head: {layers: 2, units:  32, outscale: 1.0, dist: normal}
hybrid_aif.reference.raif.agent.ACTOR_UNITS = 256#

The three head sizes, each from its own block of the oracle’s resolved config (raif-paper/logs/oracle_state_mc_seed0/config.yaml):

actor:   {layers: 3, units: 256, outscale: 1.0, act: silu, dist: normal}
critic:  {layers: 2, units:  32, outscale: 0.0, dist: symlog_disc, bins: 255}
ig_head: {layers: 2, units:  32, outscale: 1.0, dist: normal}
hybrid_aif.reference.raif.agent.HEAD_LAYERS = 3#

The actor’s size under its old name. raif_mountain_car(head_units=…, head_layers=…) still overrides all three at once, which is what the test and smoke sizes want; the parity build reads the three pairs above.

hybrid_aif.reference.raif.agent.ENSEMBLE_SIZE = 4#

ig_models: 4 (resolved config; configs.yaml:102-106).

hybrid_aif.reference.raif.agent.ACTOR_MINSTD = 0.1#

The actor head’s std band, actor: {minstd: 0.1, maxstd: 1.0}: their head is Normal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), which MLPConditional’s head="bounded" implements.

hybrid_aif.reference.raif.agent.ACTOR_MAXSTD = 1.0#

The actor head’s std band, actor: {minstd: 0.1, maxstd: 1.0}: their head is Normal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), which MLPConditional’s head="bounded" implements.

hybrid_aif.reference.raif.agent.ENSEMBLE_STD = 1.0#

The ensemble heads’ σ, a constant: ig_head declares no minstd/maxstd, so their defaults (1.0, 1.0) give std ≡ 1.0 and the member NLL is a scaled MSE.

hybrid_aif.reference.raif.agent.ACTION_LOW = -1.0#

Their action box, Box(-1, 1, (1,)), and so the magnitude of their rssm.action_clip: 1.0. Read by to_world_batch().

hybrid_aif.reference.raif.agent.ACTION_HIGH = 1.0#

Their action box, Box(-1, 1, (1,)), and so the magnitude of their rssm.action_clip: 1.0. Read by to_world_batch().

hybrid_aif.reference.raif.agent.BATCH_SIZE = 16#

Their Mountain Car training protocol (addendum §5), for the recipe’s defaults.

hybrid_aif.reference.raif.agent.WORLD_LR = 0.0001#

Their optimizer groups (addendum §5): the world model and the CRSPP at adam 1e-4 (clip 1000), the actor and critic at adam 3e-5 (clip 100), the info-gain ensemble at adam 1e-4. Defaults of train_step()’s arguments.

hybrid_aif.reference.raif.agent.ACTOR_EPS = 1e-05#

The rest of those optimizer lines, from the oracle’s resolved config (logs/oracle_state_mc_seed0/config.yaml):

model_opt:  {clip: 1000.0, eps: 1e-08, lr: 0.0001, opt: adam, wd: 0.0}
actor_opt:  {clip:  100.0, eps: 1e-05, lr: 3e-05,  opt: adam, wd: 0.0}
critic_opt: {clip:  100.0, eps: 1e-05, lr: 3e-05,  opt: adam, wd: 0.0}
ig_opt:     {clip:  100.0, eps: 1e-05, lr: 0.0001, opt: adam, wd: 0.0}

eps differs by group: 1e-8 on the world model (optax’s default) and 1e-5 on the amortized heads, passed explicitly. ACTOR_EPS covers the critic too: imagine() takes one adam_eps for the pair.

hybrid_aif.reference.raif.agent.BC_SCALE = 0.5#

The self-imitation term’s scale (configs.yaml:79; A10 disposition 8).

hybrid_aif.reference.raif.agent.ACTENT = 0.0003#

Paper Eq. 10’s η and ζ: the policy-entropy bonus (their actent) and the state-prior-entropy bonus (their wment), both 3e-4 in the resolved config.

hybrid_aif.reference.raif.agent.TERMS = ('predicted_reward:obs:reward', 'cosine_sim_to_pref_rollout', 'model_uncertainty:stoch', 'g_tail')#

The rows this agent mounts: their critic’s value reward + sim − IG, closed at the horizon by the critic (paper Eq. 6, all G_scales 1.0). model_uncertainty is the −IG: the ensemble-disagreement estimator under a term of sign +1, so it enters G as a cost and the curiosity row is not mounted. occupancy is absent because the state-prior entropy is the actor’s wment bonus, not part of the critic’s value. Design: docs/design/amortization-p0-addendum.md §3.

class hybrid_aif.reference.raif.agent.Agent[source]#

Bases: NamedTuple

Everything a training loop threads: the declared agent and its optimizer carry.

system/params are the agent with the G side mounted. crspp/crspp_params are the preference model’s pair, also reachable as system.scoring/params.scoring under the CRSPP row and named here for fit_crspp(). learn_state is the one optimizer carry for all five parameter sets.

system: System#

Alias for field number 0

params: SystemParams#

Alias for field number 1

crspp: System#

Alias for field number 2

crspp_params: SystemParams#

Alias for field number 3

learn_state: LearnState#

Alias for field number 4

static __new__(_cls, system, params, crspp, crspp_params, learn_state)#

Create new instance of Agent(system, params, crspp, crspp_params, learn_state)

hybrid_aif.reference.raif.agent.raif_mountain_car(
key,
*,
obs_dim=2,
action_dim=1,
horizon=8,
head_layers=None,
head_units=None,
actor_layers=3,
actor_units=256,
critic_layers=2,
critic_units=32,
ensemble_layers=2,
ensemble_units=32,
ensemble_size=4,
units=32,
num_bins=255,
bin_low=-20.0,
bin_high=20.0,
winit='normal',
**tiny,
)[source]#

R-AIF’s agent for state-based Mountain Car: Agent(system, params, crspp, …, learn_state).

winit is the initializer of the world model, the CRSPP and the critic ("normal" theirs, "default" eqx’s); the actor and ensemble use "normal" either way. tiny keyword arguments go to tiny_core(), and the CRSPP is built at the same sizes with the agent’s embed. head_layers/head_units override all three head sizes at once; None keeps each head’s own. The planner is "habit": G trains the actor through imagine and is not an act-time score. The ensemble enters G as the model_uncertainty cost, so epistemic=False (TERMS).

Design: docs/experiments/raif-mountain-car/divergence-register.md (divergences 17, 26).

class hybrid_aif.reference.raif.agent.StepLosses[source]#

Bases: NamedTuple

One training step’s four loss reports, one per parameter set.

world: WorldLosses#

Alias for field number 0

imagine: ImagineMetrics#

Alias for field number 1

ensemble: EnsembleLosses#

Alias for field number 2

crspp: CRSPPLosses#

Alias for field number 3

static __new__(_cls, world, imagine, ensemble, crspp)#

Create new instance of StepLosses(world, imagine, ensemble, crspp)

hybrid_aif.reference.raif.agent.train_step(
agent,
batch,
*,
key,
horizon=8,
world_lr=0.0001,
actor_lr=3e-05,
critic_lr=3e-05,
ensemble_lr=0.0001,
crspp_lr=0.0001,
bc_scale=0.5,
sample_scoring=False,
actor_eps=1e-05,
ensemble_eps=1e-05,
ensemble_clip=100.0,
)[source]#

One gradient step of all five parameter sets, in their order: (agent, StepLosses).

batch is one replay draw as a batched Episode (leaves (B, T, ...)). to_world_batch() shifts its actions to the a_{t−1} convention for the recurrent trainers; the BC term reads action in the a_t convention with can_self_imitate (by then the imitation mask), and the CRSPP reads pref_label. Both columns come from hybrid_aif.reference.raif.revise.label() at collection time. The order:

1. fit_world       the sequence loss over the declared slots; its forward
                   pass's posterior states are the ONE recognition 2-4 read
2. fit_ensemble    Gaussian NLL on (s_t, a_t) -> s_{t+1}
3. fit_crspp       the rho-gated KL + the contrastive cosine term; its
                   forward pass's posterior is the cosine row's start
4. imagine         actor + critic on sampled rollouts from those states

Steps 2-4 read step 1’s pre-step states, gradient stopped, and do not differentiate the world model; step 4 starts the cosine row from step 3’s pass. Collection, routing and batch sourcing are the caller’s (hybrid_aif.reference.raif.run); every world-model batch is drawn from M in the parity recipe.

Design: docs/experiments/raif-mountain-car/execution-ledger-training.md §21.

hybrid_aif.reference.raif.agent.to_world_batch(batch)[source]#

A batched Episode as a WorldBatch.

Shifts replay’s a_t (the action taken at obs[t]) to a_{t−1} (the action that produced it) with prev_actions(), and carries is_first so observe() restarts the carry where a window crosses an episode boundary. The shifted action is rescaled into the action box with soft_clip(), as their obs_step does; the recorded column stays raw.

Design: docs/experiments/raif-mountain-car/execution-ledger-acting.md D5.

hybrid_aif.reference.raif.agent.act(agent, beliefs, key)[source]#

The habit’s action at beliefs: System.act under planner="habit".

beliefs and the action are keyed as the faces key them. plan_key and the action key are split from key, not shared.