hybrid_aif.reference.raif.revise#

Self-revision: the per-episode backward pass that produces ρ and the imitation mask.

A port of their callbacks.py::lookback_on_episode with their configs.yaml:109 hyperparameters. It runs once per finished episode, before the episode is written to the buffers (label()). Pure, unbatched and jit-able: one episode in, two (T,) arrays out.

Design: docs/design/amortization-p0-addendum.md §4.

Module Attributes

ALPHA

Their configs.yaml:109 triple: the backward discount, the quantization floor below which the carry is zeroed, and the failure branch's decay.

Functions

NamedTuple(typename[, fields])

Typed version of namedtuple.

label(ep, **kwargs)

ep with both columns lookback_on_episode writes: ρ and the imitation mask.

positive_gate(rho)

max(0, sgn ρ): the CRSPP KL's hard mask to positive states (paper Eq.

pref_labels(ep, *[, alpha, epsilon, beta])

lookback_on_episode, ported: (pref_label, can_self_imitate).

Classes

Array

Array base class for JAX

Episode

One episode's per-step columns — the format the whole training side speaks.

PrefLabels

What lookback_on_episode writes back into the episode.

hybrid_aif.reference.raif.revise.ALPHA = 0.4#

Their configs.yaml:109 triple: the backward discount, the quantization floor below which the carry is zeroed, and the failure branch’s decay.

class hybrid_aif.reference.raif.revise.PrefLabels[source]#

Bases: NamedTuple

What lookback_on_episode writes back into the episode. Both (T,).

pref_label is ρ ∈ [−1, 1]. can_self_imitate is the propagated imitation mask, a float weight in [0, 1] that replaces the recorded flag and gates the actor’s BC term.

pref_label: Array#

Alias for field number 0

can_self_imitate: Array#

Alias for field number 1

static __new__(_cls, pref_label, can_self_imitate)#

Create new instance of PrefLabels(pref_label, can_self_imitate)

hybrid_aif.reference.raif.revise.pref_labels(ep, *, alpha=0.4, epsilon=0.001, beta=0.3)[source]#

lookback_on_episode, ported: (pref_label, can_self_imitate).

alpha, epsilon and beta are their disc, thr and fail. The steps:

  • the expert flags are gated by episode success, can_self_imitate *= has_succeeded[-1];

  • a backward carry seeded from the last step’s gated flag credits step t−1 from is_success[t] geometrically at alpha, zeroed below epsilon; mask[T−1] is the seed and is not thresholded;

  • the mask is the clipped sum of the gated flags and the carry, shifted one step forward into positive_mask (slot 0 from has_succeeded[0]);

  • in a failed episode each step gets −beta**(T−1−t), so the last step is at −1; the expert exemption on this branch is a no-op after the success gate and is kept as their code has it;

  • pref_label = clip(positive_mask + neg, −1, 1) with pref_label[0] = 0.

Raises ValueError if alpha is outside [0, 1), epsilon is negative or beta is outside (0, 1).

hybrid_aif.reference.raif.revise.label(ep, **kwargs)[source]#

ep with both columns lookback_on_episode writes: ρ and the imitation mask.

Call it on a finished episode, before hybrid_aif.train.buffers.add(); on a stored window the last step is not the episode’s last step. Afterwards can_self_imitate is the float mask, not the recorded expert flag.

hybrid_aif.reference.raif.revise.positive_gate(rho)[source]#

max(0, sgn ρ): the CRSPP KL’s hard mask to positive states (paper Eq. 5).