imagine#
- hybrid_aif.train.imagine(
- system,
- params,
- start_beliefs,
- learn_state,
- *,
- key,
- horizon=None,
- start_actions=None,
- bc_weights=None,
- start_cont=None,
- aux_beliefs=None,
- cont_modality=None,
- discount=0.996996996996997,
- return_lambda=0.95,
- policy_entropy_scale=0.0003,
- prior_entropy_scale=0.0,
- bc_scale=0.0,
- slow_fraction=0.02,
- slow_period=1,
- slowreg=1.0,
- retnorm=(5.0, 95.0, 0.99, 1.0),
- actor_lr=3e-05,
- critic_lr=3e-05,
- lr_by_slot=None,
- clip=100.0,
- adam_eps=1e-05,
- sample_scoring=False,
One actor+critic gradient step on imagined rollouts:
(params, learn_state, metrics).Gradients: the actor’s flows through its reparameterised draws and the dynamics, with the critic and the actor’s conditioning state stop-gradiented; the critic regresses on the stop-gradiented λ-return at stop-gradiented start features.
Pure, jit-able and resumable from its own return value; steps only
ACTOR_KEYandscoring:g_tail, threading every otherLearnStateentry through.start_beliefsis batched beliefs keyed by state factor name (leaves(N, ...)), e.g.posterior_states()flattened over batch and time.horizondefaults topolicy_lenand must be at least 1.start_actions(keyed by the action nodes’ names,System.names.actions; leaves(N, ...)) andbc_weights(N,)(in [0, 1]) are the replay action taken at each start state and its behaviour-cloning weight; withWorldBatch’saction[t] = a_{t−1}that action sits att+1. Pass both or neither.aux_beliefsmaps a scoring row name to the belief carry of that row’s own nested System, keyed by its state factor names (leaves(N, ...)), asplanner_context()takes it.cont_modalitynames the continuation head (None: the uniquebinaryhead).start_cont(N,)is1 − is_terminal(s_0); omitted, every start counts as non-terminal.sample_scoring(static) gives the scoring rows their own key without changing the actor’s draws.actor_lr/critic_lrmay be traced;lr_by_slotoverrides them by address, a static0.0skipping that slot.clipis a per-slot global-norm clip (Nonedisables it);adam_epsbuilds the two optimizers.Raises
ValueErrorwhen the System declares no policy module, whenscoring['g_tail']is not a(critic, slow)pair, whenlearn_stateisNoneor lacks this trainer’s optimizer ormomentsentries (never re-seeded), and on a shape mismatch instart_actions/bc_weights/start_cont.