hybrid_aif.reference.raif.agent#
The assembled R-AIF agent, and its training step.
The five parameter sets of their training step and where they live:
θ world model |
the declared slots ( |
group 1 |
ψ actor |
|
group 2 |
χ critic + slow copy |
|
group 3 |
ω info-gain ensemble |
|
group 3 |
φ CRSPP |
|
group 3 |
Sizes are the oracle’s resolved config: the actor is 3 × 256, the critic and each ensemble
member 2 × 32, and the world model is tiny.
Design: docs/experiments/raif-mountain-car/module-correspondence.md.
Module Attributes
The three head sizes, each from its own block of the oracle's resolved config ( |
|
The three head sizes, each from its own block of the oracle's resolved config ( |
|
The actor's size under its old name. |
|
|
|
The actor head's std band, |
|
The actor head's std band, |
|
The ensemble heads' σ, a constant: |
|
Their action box, |
|
Their action box, |
|
Their Mountain Car training protocol (addendum §5), for the recipe's defaults. |
|
Their optimizer groups (addendum §5): the world model and the CRSPP at |
|
The rest of those optimizer lines, from the oracle's resolved config ( |
|
The self-imitation term's scale ( |
|
Paper Eq. |
|
The rows this agent mounts: their critic's value |
Functions
|
Typed version of namedtuple. |
|
The habit's action at |
|
The world-model assembly behind a declared System: |
|
R-AIF's CRSPP prior-preference model: |
|
The agent's encoder output on a batch of observations: |
|
One gradient step of preference learning: |
|
One gradient step of ensemble learning: |
|
One gradient step of world-model learning: |
|
One actor+critic gradient step on imagined rollouts: |
|
The engine's |
|
R-AIF's agent for state-based Mountain Car: |
|
Bring an action into its box by a stop-gradiented rescale, not a clip. |
|
R-AIF's |
|
A batched |
|
One gradient step of all five parameter sets, in their order: |
|
A batch of posterior states as the ensemble's |
Classes
Everything a training loop threads: the declared agent and its optimizer carry. |
|
|
Array base class for JAX |
|
The preference model's loss, decomposed. |
|
The ensemble's loss: the sum over members, and each member's own. |
|
One episode's per-step columns — the format the whole training side speaks. |
|
Plain arrays, one scalar each — what a training loop logs and a test asserts. |
|
Optimizer state for an agent's gradient-learnable parts: |
|
MLP over concatenated parents → diagonal-Gaussian child (see module docstring). |
|
An |
One training step's four loss reports, one per parameter set. |
|
|
Static structure of a System: discrete, continuous or mixed. |
|
All numeric state of a System: generative tensors, designations and the closed-loop factors. |
|
A batch of replay sequences, all shapes static. |
|
The objective, decomposed: every field a scalar mean over batch and time. |
- hybrid_aif.reference.raif.agent.ACTOR_LAYERS = 3#
The three head sizes, each from its own block of the oracle’s resolved config (
raif-paper/logs/oracle_state_mc_seed0/config.yaml):actor: {layers: 3, units: 256, outscale: 1.0, act: silu, dist: normal} critic: {layers: 2, units: 32, outscale: 0.0, dist: symlog_disc, bins: 255} ig_head: {layers: 2, units: 32, outscale: 1.0, dist: normal}
- hybrid_aif.reference.raif.agent.ACTOR_UNITS = 256#
The three head sizes, each from its own block of the oracle’s resolved config (
raif-paper/logs/oracle_state_mc_seed0/config.yaml):actor: {layers: 3, units: 256, outscale: 1.0, act: silu, dist: normal} critic: {layers: 2, units: 32, outscale: 0.0, dist: symlog_disc, bins: 255} ig_head: {layers: 2, units: 32, outscale: 1.0, dist: normal}
- hybrid_aif.reference.raif.agent.HEAD_LAYERS = 3#
The actor’s size under its old name.
raif_mountain_car(head_units=…, head_layers=…)still overrides all three at once, which is what the test and smoke sizes want; the parity build reads the three pairs above.
- hybrid_aif.reference.raif.agent.ENSEMBLE_SIZE = 4#
ig_models: 4(resolved config;configs.yaml:102-106).
- hybrid_aif.reference.raif.agent.ACTOR_MINSTD = 0.1#
The actor head’s std band,
actor: {minstd: 0.1, maxstd: 1.0}: their head isNormal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), whichMLPConditional’shead="bounded"implements.
- hybrid_aif.reference.raif.agent.ACTOR_MAXSTD = 1.0#
The actor head’s std band,
actor: {minstd: 0.1, maxstd: 1.0}: their head isNormal(tanh(out), minstd + (maxstd − minstd)·σ(raw + 2)), whichMLPConditional’shead="bounded"implements.
- hybrid_aif.reference.raif.agent.ENSEMBLE_STD = 1.0#
The ensemble heads’ σ, a constant:
ig_headdeclares nominstd/maxstd, so their defaults (1.0, 1.0) givestd ≡ 1.0and the member NLL is a scaled MSE.
- hybrid_aif.reference.raif.agent.ACTION_LOW = -1.0#
Their action box,
Box(-1, 1, (1,)), and so the magnitude of theirrssm.action_clip: 1.0. Read byto_world_batch().
- hybrid_aif.reference.raif.agent.ACTION_HIGH = 1.0#
Their action box,
Box(-1, 1, (1,)), and so the magnitude of theirrssm.action_clip: 1.0. Read byto_world_batch().
- hybrid_aif.reference.raif.agent.BATCH_SIZE = 16#
Their Mountain Car training protocol (addendum §5), for the recipe’s defaults.
- hybrid_aif.reference.raif.agent.WORLD_LR = 0.0001#
Their optimizer groups (addendum §5): the world model and the CRSPP at
adam 1e-4(clip 1000), the actor and critic atadam 3e-5(clip 100), the info-gain ensemble atadam 1e-4. Defaults oftrain_step()’s arguments.
- hybrid_aif.reference.raif.agent.ACTOR_EPS = 1e-05#
The rest of those optimizer lines, from the oracle’s resolved config (
logs/oracle_state_mc_seed0/config.yaml):model_opt: {clip: 1000.0, eps: 1e-08, lr: 0.0001, opt: adam, wd: 0.0} actor_opt: {clip: 100.0, eps: 1e-05, lr: 3e-05, opt: adam, wd: 0.0} critic_opt: {clip: 100.0, eps: 1e-05, lr: 3e-05, opt: adam, wd: 0.0} ig_opt: {clip: 100.0, eps: 1e-05, lr: 0.0001, opt: adam, wd: 0.0}
epsdiffers by group:1e-8on the world model (optax’s default) and1e-5on the amortized heads, passed explicitly.ACTOR_EPScovers the critic too:imagine()takes oneadam_epsfor the pair.
- hybrid_aif.reference.raif.agent.BC_SCALE = 0.5#
The self-imitation term’s scale (
configs.yaml:79; A10 disposition 8).
- hybrid_aif.reference.raif.agent.ACTENT = 0.0003#
Paper Eq. 10’s η and ζ: the policy-entropy bonus (their
actent) and the state-prior-entropy bonus (theirwment), both 3e-4 in the resolved config.
- hybrid_aif.reference.raif.agent.TERMS = ('predicted_reward:obs:reward', 'cosine_sim_to_pref_rollout', 'model_uncertainty:stoch', 'g_tail')#
The rows this agent mounts: their critic’s value
reward + sim − IG, closed at the horizon by the critic (paper Eq. 6, allG_scales1.0).model_uncertaintyis the−IG: the ensemble-disagreement estimator under a term of sign +1, so it enters G as a cost and the curiosity row is not mounted.occupancyis absent because the state-prior entropy is the actor’swmentbonus, not part of the critic’s value. Design:docs/design/amortization-p0-addendum.md§3.
- class hybrid_aif.reference.raif.agent.Agent[source]#
Bases:
NamedTupleEverything a training loop threads: the declared agent and its optimizer carry.
system/paramsare the agent with the G side mounted.crspp/crspp_paramsare the preference model’s pair, also reachable assystem.scoring/params.scoringunder the CRSPP row and named here forfit_crspp().learn_stateis the one optimizer carry for all five parameter sets.- params: SystemParams#
Alias for field number 1
- crspp_params: SystemParams#
Alias for field number 3
- learn_state: LearnState#
Alias for field number 4
- static __new__(_cls, system, params, crspp, crspp_params, learn_state)#
Create new instance of Agent(system, params, crspp, crspp_params, learn_state)
- hybrid_aif.reference.raif.agent.raif_mountain_car(
- key,
- *,
- obs_dim=2,
- action_dim=1,
- horizon=8,
- head_layers=None,
- head_units=None,
- actor_layers=3,
- actor_units=256,
- critic_layers=2,
- critic_units=32,
- ensemble_layers=2,
- ensemble_units=32,
- ensemble_size=4,
- units=32,
- num_bins=255,
- bin_low=-20.0,
- bin_high=20.0,
- winit='normal',
- **tiny,
R-AIF’s agent for state-based Mountain Car:
Agent(system, params, crspp, …, learn_state).winitis the initializer of the world model, the CRSPP and the critic ("normal"theirs,"default"eqx’s); the actor and ensemble use"normal"either way.tinykeyword arguments go totiny_core(), and the CRSPP is built at the same sizes with the agent’sembed.head_layers/head_unitsoverride all three head sizes at once;Nonekeeps each head’s own. The planner is"habit": G trains the actor throughimagineand is not an act-time score. The ensemble enters G as themodel_uncertaintycost, soepistemic=False(TERMS).Design:
docs/experiments/raif-mountain-car/divergence-register.md(divergences 17, 26).
- class hybrid_aif.reference.raif.agent.StepLosses[source]#
Bases:
NamedTupleOne training step’s four loss reports, one per parameter set.
- world: WorldLosses#
Alias for field number 0
- imagine: ImagineMetrics#
Alias for field number 1
- ensemble: EnsembleLosses#
Alias for field number 2
- crspp: CRSPPLosses#
Alias for field number 3
- static __new__(_cls, world, imagine, ensemble, crspp)#
Create new instance of StepLosses(world, imagine, ensemble, crspp)
- hybrid_aif.reference.raif.agent.train_step(
- agent,
- batch,
- *,
- key,
- horizon=8,
- world_lr=0.0001,
- actor_lr=3e-05,
- critic_lr=3e-05,
- ensemble_lr=0.0001,
- crspp_lr=0.0001,
- bc_scale=0.5,
- sample_scoring=False,
- actor_eps=1e-05,
- ensemble_eps=1e-05,
- ensemble_clip=100.0,
One gradient step of all five parameter sets, in their order:
(agent, StepLosses).batchis one replay draw as a batchedEpisode(leaves(B, T, ...)).to_world_batch()shifts its actions to thea_{t−1}convention for the recurrent trainers; the BC term readsactionin thea_tconvention withcan_self_imitate(by then the imitation mask), and the CRSPP readspref_label. Both columns come fromhybrid_aif.reference.raif.revise.label()at collection time. The order:1. fit_world the sequence loss over the declared slots; its forward pass's posterior states are the ONE recognition 2-4 read 2. fit_ensemble Gaussian NLL on (s_t, a_t) -> s_{t+1} 3. fit_crspp the rho-gated KL + the contrastive cosine term; its forward pass's posterior is the cosine row's start 4. imagine actor + critic on sampled rollouts from those states
Steps 2-4 read step 1’s pre-step states, gradient stopped, and do not differentiate the world model; step 4 starts the cosine row from step 3’s pass. Collection, routing and batch sourcing are the caller’s (
hybrid_aif.reference.raif.run); every world-model batch is drawn fromMin the parity recipe.Design:
docs/experiments/raif-mountain-car/execution-ledger-training.md§21.
- hybrid_aif.reference.raif.agent.to_world_batch(batch)[source]#
A batched
Episodeas aWorldBatch.Shifts replay’s
a_t(the action taken atobs[t]) toa_{t−1}(the action that produced it) withprev_actions(), and carriesis_firstsoobserve()restarts the carry where a window crosses an episode boundary. The shifted action is rescaled into the action box withsoft_clip(), as theirobs_stepdoes; the recorded column stays raw.Design:
docs/experiments/raif-mountain-car/execution-ledger-acting.mdD5.