imagine

Contents

imagine#

hybrid_aif.train.imagine(
system,
params,
start_beliefs,
learn_state,
*,
key,
horizon=None,
start_actions=None,
bc_weights=None,
start_cont=None,
aux_beliefs=None,
cont_modality=None,
discount=0.996996996996997,
return_lambda=0.95,
policy_entropy_scale=0.0003,
prior_entropy_scale=0.0,
bc_scale=0.0,
slow_fraction=0.02,
slow_period=1,
slowreg=1.0,
retnorm=(5.0, 95.0, 0.99, 1.0),
actor_lr=3e-05,
critic_lr=3e-05,
lr_by_slot=None,
clip=100.0,
adam_eps=1e-05,
sample_scoring=False,
)[source]#

One actor+critic gradient step on imagined rollouts: (params, learn_state, metrics).

Gradients: the actor’s flows through its reparameterised draws and the dynamics, with the critic and the actor’s conditioning state stop-gradiented; the critic regresses on the stop-gradiented λ-return at stop-gradiented start features.

Pure, jit-able and resumable from its own return value; steps only ACTOR_KEY and scoring:g_tail, threading every other LearnState entry through.

start_beliefs is batched beliefs keyed by state factor name (leaves (N, ...)), e.g. posterior_states() flattened over batch and time. horizon defaults to policy_len and must be at least 1. start_actions (keyed by the action nodes’ names, System.names.actions; leaves (N, ...)) and bc_weights (N,) (in [0, 1]) are the replay action taken at each start state and its behaviour-cloning weight; with WorldBatch’s action[t] = a_{t−1} that action sits at t+1. Pass both or neither. aux_beliefs maps a scoring row name to the belief carry of that row’s own nested System, keyed by its state factor names (leaves (N, ...)), as planner_context() takes it. cont_modality names the continuation head (None: the unique binary head). start_cont (N,) is 1 − is_terminal(s_0); omitted, every start counts as non-terminal. sample_scoring (static) gives the scoring rows their own key without changing the actor’s draws.

actor_lr/critic_lr may be traced; lr_by_slot overrides them by address, a static 0.0 skipping that slot. clip is a per-slot global-norm clip (None disables it); adam_eps builds the two optimizers.

Raises ValueError when the System declares no policy module, when scoring['g_tail'] is not a (critic, slow) pair, when learn_state is None or lacks this trainer’s optimizer or moments entries (never re-seeded), and on a shape mismatch in start_actions/bc_weights/ start_cont.