hybrid_aif.reference.raif.revise#
Self-revision: the per-episode backward pass that produces ρ and the imitation mask.
A port of their callbacks.py::lookback_on_episode with their configs.yaml:109
hyperparameters. It runs once per finished episode, before the episode is written to the
buffers (label()). Pure, unbatched and jit-able: one episode in, two (T,) arrays
out.
Design: docs/design/amortization-p0-addendum.md §4.
Module Attributes
Their |
Functions
|
Typed version of namedtuple. |
|
|
|
|
|
|
Classes
|
Array base class for JAX |
|
One episode's per-step columns — the format the whole training side speaks. |
What |
- hybrid_aif.reference.raif.revise.ALPHA = 0.4#
Their
configs.yaml:109triple: the backward discount, the quantization floor below which the carry is zeroed, and the failure branch’s decay.
- class hybrid_aif.reference.raif.revise.PrefLabels[source]#
Bases:
NamedTupleWhat
lookback_on_episodewrites back into the episode. Both(T,).pref_labelis ρ ∈ [−1, 1].can_self_imitateis the propagated imitation mask, a float weight in [0, 1] that replaces the recorded flag and gates the actor’s BC term.- static __new__(_cls, pref_label, can_self_imitate)#
Create new instance of PrefLabels(pref_label, can_self_imitate)
- hybrid_aif.reference.raif.revise.pref_labels(ep, *, alpha=0.4, epsilon=0.001, beta=0.3)[source]#
lookback_on_episode, ported:(pref_label, can_self_imitate).alpha,epsilonandbetaare theirdisc,thrandfail. The steps:the expert flags are gated by episode success,
can_self_imitate *= has_succeeded[-1];a backward carry seeded from the last step’s gated flag credits step
t−1fromis_success[t]geometrically atalpha, zeroed belowepsilon;mask[T−1]is the seed and is not thresholded;the mask is the clipped sum of the gated flags and the carry, shifted one step forward into
positive_mask(slot 0 fromhas_succeeded[0]);in a failed episode each step gets
−beta**(T−1−t), so the last step is at −1; the expert exemption on this branch is a no-op after the success gate and is kept as their code has it;pref_label = clip(positive_mask + neg, −1, 1)withpref_label[0] = 0.
Raises
ValueErrorifalphais outside [0, 1),epsilonis negative orbetais outside (0, 1).
- hybrid_aif.reference.raif.revise.label(ep, **kwargs)[source]#
epwith both columnslookback_on_episodewrites: ρ and the imitation mask.Call it on a finished episode, before
hybrid_aif.train.buffers.add(); on a stored window the last step is not the episode’s last step. Afterwardscan_self_imitateis the float mask, not the recorded expert flag.