Learning#

Parameters are learned either by a conjugate update, where a Dirichlet or matrix-normal posterior absorbs sufficient statistics in closed form, or by gradient on a trainable module. The conjugate route leaves behind a closed-form log evidence, which is the quantity description length is computed from. The training functions fit a whole world model, an actor and critic, or an ensemble from replayed episodes.

Conjugate updates and evidence#

update_likelihood_dirichlet

One conjugate count update per modality, on positional arrays.

log_evidence

The log marginal likelihood of the statistics posterior absorbed from prior, one number per independent family (a table's parent combinations, an LG declaration's modes): the array has the declaration's parent shape, and a slot's evidence is its sum.

absorbed_count

The mass posterior absorbed beyond prior, per independent family and with the same shape as log_evidence(): the concentration a Dirichlet gained, the degrees of freedom an inverse-Wishart or a matrix-normal inverse-Wishart gained, twice the shape an inverse-Gamma gained (read off its first dimension; learning gives every dimension the same mass).

Evidence

The evidence of one learnable slot (a likelihood, transition or initial_state_prior entry) for what its posterior absorbed from its prior, as evidence_by_slot() reports it, summed over the slot's parent combinations.

Training loops#

fit_world

One gradient step of world-model learning: (params, learn_state, losses).

world_loss

The world-model loss on one batch: (total, WorldLosses).

WorldBatch

A batch of replay sequences, all shapes static.

WorldLosses

The objective, decomposed: every field a scalar mean over batch and time.

assemble

The world-model assembly behind a declared System: (core, slot addresses).

posterior_states

The recognition model's posterior states over a replay batch: (B, T, ...).

imagine

One actor+critic gradient step on imagined rollouts: (params, learn_state, metrics).

roll_batch

A batch of sampled imagined trajectories: one Rollout with a leading axis.

rollout_returns

The λ-returns and critic values an imagine() step would fit, without stepping.

lambda_return

DreamerV3's λ-return over an imagined trajectory.

perc_ema

Percentile-EMA return normalization: (new state, offset, invscale).

init_retnorm

A fresh (low, high) percentile-EMA carry: zeros, as seeded under RETNORM_KEY.

term_value

A term tree folded into a value (negative EFE): (value, per term).

ImagineMetrics

Plain arrays, one scalar each — what a training loop logs and a test asserts.

ReturnEstimates

Per-state λ-returns and critic values over a batch of imagined rollouts.

Rollout

One imagined trajectory's per-step quantities.

fit_ensemble

One gradient step of ensemble learning: (params, learn_state, losses).

ensemble_loss

(total, EnsembleLosses): the summed member NLL, plus the per-member vector.

transitions

A batch of posterior states as the ensemble's (s_t, a_t) -> s_{t+1} examples.

EnsembleBatch

The flat (N, ...) batch the members are pointwise functions of.

EnsembleLosses

The ensemble's loss: the sum over members, and each member's own.

Episodes and replay buffers#

Episode

One episode's per-step columns — the format the whole training side speaks.

ColumnSpec

One task-specific column of an Episode.

episode

An Episode from its columns, cast to the engine dtypes and checked.

as_episode

A ReplayBuffer episode dict → an Episode.

episode_columns

An Episode → a column dict for ReplayBuffer.write_episode(): the engine columns and the task columns, flat, as numpy arrays.

stack_episodes

A list of episodes as one batched Episode, each fitted to seq_len (fit_length(), with pad the task columns' rules).

fit_length

One episode at exactly seq_len steps: truncated, or padded by freezing.

ReplayBuffer

Ring-storage replay buffer with whole-episode FIFO eviction.

StaticBuffer

A uniform replay buffer of fixed-length episode-aligned sequences.

StreamBuffer

A uniform replay buffer over a flat step stream.

add

One episode written into a StaticBuffer ring: a new buffer, nothing mutated.

extend

One episode's steps spliced onto the stream: a new buffer, nothing mutated.

sample

A uniform batch of whole sequences: leaves (batch_size, seq_len, ...).

sample_windows

A uniform batch of seq_len-step windows: leaves (batch_size, seq_len, ...).