Learning#
Parameters are learned either by a conjugate update, where a Dirichlet or matrix-normal posterior absorbs sufficient statistics in closed form, or by gradient on a trainable module. The conjugate route leaves behind a closed-form log evidence, which is the quantity description length is computed from. The training functions fit a whole world model, an actor and critic, or an ensemble from replayed episodes.
Conjugate updates and evidence#
One conjugate count update per modality, on positional arrays. |
|
The log marginal likelihood of the statistics |
|
The mass |
|
The evidence of one learnable slot (a |
Training loops#
One gradient step of world-model learning: |
|
The world-model loss on one batch: |
|
A batch of replay sequences, all shapes static. |
|
The objective, decomposed: every field a scalar mean over batch and time. |
|
The world-model assembly behind a declared System: |
|
The recognition model's posterior states over a replay batch: |
|
One actor+critic gradient step on imagined rollouts: |
|
A batch of sampled imagined trajectories: one |
|
The λ-returns and critic values an |
|
DreamerV3's λ-return over an imagined trajectory. |
|
Percentile-EMA return normalization: |
|
A fresh |
|
A term tree folded into a value (negative EFE): |
|
Plain arrays, one scalar each — what a training loop logs and a test asserts. |
|
Per-state λ-returns and critic values over a batch of imagined rollouts. |
|
One imagined trajectory's per-step quantities. |
|
One gradient step of ensemble learning: |
|
|
|
A batch of posterior states as the ensemble's |
|
The flat |
|
The ensemble's loss: the sum over members, and each member's own. |
Episodes and replay buffers#
One episode's per-step columns — the format the whole training side speaks. |
|
One task-specific column of an |
|
An |
|
A |
|
An |
|
A list of episodes as one batched |
|
One episode at exactly |
|
Ring-storage replay buffer with whole-episode FIFO eviction. |
|
A uniform replay buffer of fixed-length episode-aligned sequences. |
|
A uniform replay buffer over a flat step stream. |
|
One episode written into a |
|
One episode's steps spliced onto the stream: a new buffer, nothing mutated. |
|
A uniform batch of whole sequences: leaves |
|
A uniform batch of |