From FP4 Stability to INT2 Prediction

The hypothesis that a stabilized low-bit envelope can make two-bit coarse prediction useful for residual neural computation

Abstract

INT2 is usually evaluated as a replacement precision:

Can a two-bit representation carry enough information to reproduce an entire neural operation?

For many training workloads, that question may be too strict and aimed at the wrong role.

This research note presents a different hypothesis developed in IDA’s native C++/CUDA training stack:

A stabilized FP4 representation can provide a bounded target surface on which an INT2 predictor learns a coarse estimate, leaving the full model body to compute only the residual it cannot predict.

The proposal does not require INT2 to reproduce the final transformation independently. It requires INT2 to predict a useful, measurable fraction of a transformation whose lower-precision behavior has already been stabilized.

FP4 supplies the envelope.

INT2 supplies the coarse prediction.

The full body remains the source of truth and correction.

The current implementation is training on NVIDIA Hopper hardware. Hopper does not provide the native FP4 execution path available on Blackwell-class hardware, so this stage primarily tests representational learnability, state correctness, calibration, and governance.

The live experiment is training the INT2 prediction path under a reduced learning rate after an earlier optimizer-regime mismatch caused overshoot. No acceleration claim is made until prediction confidence is calibrated and real body computation is measurably reduced.

---

1. The question may be wrong

The conventional low-bit question is:

Can INT2 represent the whole operation accurately enough?

That is the correct question when INT2 is expected to replace the operation.

It is not necessarily the correct question when INT2 is assigned a predictive role.

A prediction does not have to be the final answer. It has to remove enough uncertainty that the remaining correction becomes cheaper.

Let the full body transformation be:

$$y=F(x)y\; =\; F(x)$$

A coarse predictor produces:

$${\widehat{y}}_2=P_2(x)\backslash hat\{y\}\_\{2\}\; =\; P\_\{2\}(x)$$

Here, the subscript denotes an INT2 predictive surface rather than a claim that every physical instruction executes natively at two bits.

The residual is:

$$r=F(x)-{\widehat{y}}_{2}r\; =\; F(x)\; -\; \backslash hat\{y\}\_\{2\}$$

The final result remains:

$$y={\widehat{y}}_2+ry\; =\; \backslash hat\{y\}\_\{2\}\; +\; r$$

At the beginning of the experiment, the full body still computes (F(x)). That is necessary to train and score the predictor.

The architectural payoff arrives only if later execution can avoid some of the work already represented by (\hat{y}_{2}), while preserving the full body as verifier and correction path.

This changes the research question:

Can INT2 predict enough of a stabilized operation that the full model no longer needs to rediscover all of it at full resolution?

That is the hypothesis being tested.

---

2. Why FP4 stabilization comes first

Two-bit prediction is not useful when its target is contaminated by unrelated numerical state errors.

Before an INT2 head can learn a stable approximation, the representation below it must have reliable answers to four questions:

1. Which layer owns the low-bit state?
2. Which scale and descale metadata belong to a given quantization event?
3. When is delayed state allowed to advance?
4. What happens when activation checkpointing re-enters the forward path?

IDA’s packed-FP4 path exposed failures in both state timing and state ownership.

In the FP8 path, a tensor could be quantized with one scale and interpreted with a descale from a later state. Under activation checkpointing, recomputation could also enter the quantizer a second time and incorrectly advance delayed state again.

In the packed-FP4 attention path, Q/K quantizer state was initially shared across layers that required independent histories. Even a locally reasonable quantizer becomes globally wrong when one layer mutates another layer’s state.

The repaired invariant is:

real forward:
    snapshot exact quantization metadata
    quantize and consume using that snapshot
    advance mutable delayed state exactly once

recomputation:
    reuse the frozen snapshot
    reproduce computation
    perform no quantizer-state transition

The ownership rule is:

one independently evolving quantized surface
    → one independently owned state history

These repairs do not make FP4 perfect.

They make its error attributable.

That difference is essential. An INT2 predictor can learn a coarse, repeatable target. It cannot learn around scale provenance that changes between forward and recompute, or around hidden state shared by unrelated layers.

The central dependency is therefore:

unstable FP4 surface
    → INT2 learns target motion, ownership leaks, and metadata error

stabilized FP4 surface
    → INT2 can learn repeatable coarse structure

FP4 stabilization is not merely a lower loss curve.

It is a contract about state, provenance, and repeatability.

---

3. A nested precision hierarchy

The architecture is not “run the whole model at INT2.”

It is a hierarchy of responsibilities:

FP8 trunk
    provides range and the main trained representation

packed FP4 surface
    provides a lower-resolution but stabilized envelope

INT2 predictor
    produces the cheapest coarse estimate

full body residual path
    corrects what the predictor did not capture

This can also be expressed as:

INT2
    nested inside FP4
        nested inside FP8

The nesting is not merely a storage hierarchy.

It is an authority hierarchy.

FP8

FP8 carries the primary low-precision training body and supplies the broader representational envelope.

Packed FP4

Packed FP4 provides a more compressed surface whose state ownership and recomputation behavior have been explicitly stabilized.

INT2

INT2 is not treated as a free-floating quantizer responsible for the entire transformation.

It predicts within the lower-level envelope.

Full body

The complete body remains responsible for correcting the prediction and preserving the final result.

Each level is assigned work appropriate to its precision.

The coarsest representation predicts.

The finer representations bound and correct.

In the current Hopper implementation, packed FP4 and INT2 are software-managed representational formats.

The present experiment therefore separates two questions:

1. Can the nested representation learn and remain stable?
2. Can hardware with native lower-bit execution convert that learning into a favorable compute trade?

The first can be tested now.

The second requires the appropriate kernel and hardware path.

This hardware distinction is not a disclaimer added after the fact. It is part of the experimental design.

Representational viability should be established before claiming a speedup from hardware that is not executing the intended format natively.

---

4. Prediction has a lower precision requirement than final computation

An ultra-low-bit replacement path must preserve nearly everything the full operation needs.

A predictive path has a narrower obligation:

Capture the portion that is sufficiently regular to avoid recomputing it at full cost.

This is analogous in spirit—but not implementation—to speculative decoding.

A smaller draft model proposes work, and the larger model verifies it.

The IDA hypothesis moves the draft-and-correct pattern inside the body:

speculative decoding:
    cheap model predicts tokens
    full model verifies token proposals

coarse internal prediction:
    INT2 head predicts an internal transformation
    full body verifies and computes the residual

The intended advantage is not that INT2 becomes magically expressive.

The advantage is that the task assigned to INT2 is deliberately limited.

The predictor should become most useful where the body is already repetitive:

• frequent tokens;
• recurring local contexts;
• stable tail-layer transformations;
• low-surprisal hidden-state transitions;
• and patterns repeatedly encountered across a model lineage.

Novel or high-surprisal positions should continue receiving the full computation.

This suggests that success should not be measured only with one global confidence value.

A global average can hide the structure that matters.

The eventual engagement policy should be per pattern, initially per token and later per context hash:

familiar pattern:
    coarse prediction has demonstrated calibrated coverage
    → allow partial predictive authority

novel pattern:
    prediction error or surprisal rises
    → immediately return to the full body

The speed mechanism is therefore learned fluency, not indiscriminate quantization.

---

5. The predictive head

The current head is trained to estimate a downstream body transformation from an earlier hidden state.

Its basic form is a low-rank predictor:

$$\widehat{y}=U\sigma (Vh)\backslash hat\{y\}\; =\; U\backslash ,\backslash sigma(Vh)$$

Here:

• (h) is the saved pre-target hidden state;
• (V) projects into a low-rank space;
• (\sigma) is the nonlinearity;
• (U) reconstructs the predicted body output;
• and (\hat{y}) is the coarse estimate.

The current INT2 path treats the prediction as a coarse surface nested inside the stabilized FP4/FP8 representation.

Training begins in shadow mode:

saved hidden state
    → INT2 predictive head
    → coarse estimated output

real body
    → true output

prediction loss
    → trains the predictive head

main task loss
    → continues training the body

Shadow mode is important because it separates two claims:

• Learnability: the predictor can approximate the transformation.
• Authority: the predictor is safe and useful enough to influence execution.

A predictor should not grant itself authority merely because its auxiliary loss falls.

Its claim has to be evaluated externally.

---

6. Learning-rate reduction is part of the hypothesis test

The live experiment is currently training with a reduced learning rate on the INT2 predictive head.

That change is not cosmetic tuning.

The predictor’s earlier learning-rate scale had been chosen before the production optimizer changed to Lion.

Under a sign-based optimizer, multiplying the predictor learning rate does not simply make a weak signal learn faster. It increases the size of every signed parameter update.

In the observed run, the result was predictor overshoot rather than faster convergence.

The practical fingerprint was consistent with an update scale that was too large:

• the predicted output expanded beyond the target scale;
• relative prediction error rose instead of settling;
• and confidence remained uninformative.

Reducing the INT2 head’s learning rate tests a specific causal claim:

The coarse predictor is learnable, but it must settle inside the stabilized FP4 target rather than oscillate across it.

The current run is therefore not a generic hyperparameter sweep.

It tests whether a lower update magnitude allows the two-bit predictive surface to track the repeatable component of the body transformation.

The result should be judged over more than one favorable window.

Evidence should include:

• prediction error over time;
• calibration of asserted coverage against delivered coverage;
• target and predictor magnitude;
• gradient ownership;
• trunk loss and gradient behavior;
• held-out capability;
• predictor overhead;
• and eventual residual-work reduction.

---

7. Confidence is a claim, not permission

A predictive head can report that it covers a fraction of the target.

That report is only a claim.

Let the head assert coverage (c_t) at time (t). The system can compare that assertion with the coverage actually delivered in a later window:

$$E_{\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{b}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}}=\mid c_t-{\tilde{c}}_{t+1}\mid E\_\{\backslash mathrm\{calibration\}\}\; =\; \backslash left|c\_t\; -\; \backslash tilde\{c\}\_\{t+1\}\backslash right|$$

This creates an external score for the head’s honesty.

The governor—not the predictor—owns engagement authority.

The intended policy has three properties.

Gradual engagement

Crossing a threshold should not immediately replace a large portion of body computation.

Authority should rise in small steps.

Hysteresis

The engagement threshold should be higher than the disengagement threshold.

That prevents the system from rapidly switching in and out near one noisy boundary.

Immediate retreat under surprise

When delivered coverage falls, prediction error rises, or a pattern becomes novel, the governor should return that position to the full body.

Conceptually:

confidence low:
    shadow prediction only

confidence calibrated and rising:
    limited blend or residual authority

surprisal rises:
    snap back to full computation

The head estimates.

The evaluator measures.

The governor authorizes.

That separation is necessary because a self-scoring predictor can become confidently wrong.

---

8. What “enabled by FP4 stabilization” means

The title does not claim that FP4 mathematically guarantees successful INT2 prediction.

It makes a narrower systems claim:

The INT2 experiment became meaningful only after FP4 behavior was stable enough that prediction error could be separated from substrate error.

Before stabilization, a failed INT2 run would be ambiguous.

The failure could come from:

• insufficient two-bit capacity;
• scale/descale mismatch;
• delayed-state double advancement;
• recomputation mutation;
• cross-layer state contamination;
• gradient instability;
• optimizer mismatch;
• or actual predictor failure.

After stabilization, those confounders are reduced or independently testable.

The INT2 hypothesis can finally be falsified on its own terms.

This is a general research lesson:

Extreme quantization should not be evaluated on top of an uncharacterized lower-precision substrate.

Otherwise, the experiment measures the stack’s hidden state errors rather than the representation being tested.

---

9. Global confidence is probably insufficient

A single global confidence score can hide the structure that matters.

Some tokens and contexts appear constantly.

Others are rare.

A model should not wait until every pattern becomes equally predictable before receiving any benefit from predictive computation.

The expected acquisition order is frequency-driven:

common patterns:
    seen repeatedly
    learned first
    become predictable first

rare patterns:
    remain uncertain
    retain full computation

This motivates a fluency table.

The first implementation can track confidence by token ID.

A later version can track confidence by a context hash or another local pattern identifier.

fluency[token]
    = moving estimate of delivered predictive coverage

At each position:

fluency high:
    permit limited predictive engagement

fluency low:
    execute the full body

unexpected prediction error:
    immediately revoke engagement

This is similar to reading fluency.

A beginning reader decodes each letter.

A fluent reader recognizes familiar word patterns and reserves slow decoding for unfamiliar words.

The model’s speed mechanism would follow the same principle:

Familiar computational patterns become cheap, while novel patterns continue receiving full attention.

---

10. Coarse prediction of forward and backward work

The intended INT2 surface is not limited to predicting one forward activation.

The broader design is to estimate portions of both forward and backward work:

INT2 predictor:
    estimates forward transformation
    estimates backward structure

real body:
    computes correction
    supplies unresolved residual

This is the frame-generation analogy made precise.

A generated frame does not eliminate rendering responsibility.

It predicts enough of the next frame that the renderer can concentrate on correction, newly revealed geometry, motion error, and detail.

Likewise, the INT2 head is not intended to replace the body’s reasoning.

It is intended to predict the portion of the transformation that no longer needs to be recomputed from scratch.

The full body remains responsible for anything the prediction cannot justify.

---

11. Blending is not yet acceleration

A blended output can test authority transfer:

$$y_{\mathrm{b}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{d}}=(1-\alpha )F(x)+\alpha P_2(x)y\_\{\backslash mathrm\{blend\}\}\; =\; (1-\backslash alpha)F(x)\; +\; \backslash alpha\; P\_\{2\}(x)$$

Here, (\alpha) is the governor-controlled engagement fraction.

But if both (F(x)) and (P_{2}(x)) are fully computed, blending saves no body FLOPs.

It may prove that:

• the governor works;
• the prediction is numerically compatible;
• gradient routing behaves correctly;
• and partial authority does not destabilize the body.

It does not prove acceleration.

Actual speed begins only when the body performs less work.

That may require:

• skipping an authorized subcomponent;
• executing only the unresolved residual;
• reducing selected tiles or blocks;
• conditionally narrowing a tail computation;
• or designing a kernel that consumes the prediction as a starting point.

The relevant cost equation is:

Anything less is an architectural demonstration, not a throughput result.

---

12. Current implementation status

At the time of writing, the following elements have been implemented and exercised in the IDA native trainer:

• recomputation-safe FP8 delayed-state snapshots;
• per-layer packed-FP4 state ownership;
• paired quantization and interpretation metadata;
• an active low-rank body predictor;
• INT2 coarse prediction in the live training path;
• shadow scoring against the true body output;
• an independent predictor-training objective;
• an external confidence-calibration path;
• governor logic with ramp, hold, disengage, and hysteresis behavior;
• and a reduced predictor learning rate for the current INT2 training run.

The distinction between implementation and demonstrated payoff remains important.

Implemented and exercised:

coarse prediction
training
scoring
calibration
governor mechanics
learning-rate intervention

Still being established experimentally:

stable long-horizon INT2 convergence
naturally earned predictive authority
per-pattern engagement
reduced residual computation
net throughput improvement
held-out capability parity under compute reduction

All mechanisms attempted so far have been built and run.

The remaining question is not whether the system exists.

It is whether the live low-learning-rate INT2 head earns enough calibrated coverage to reduce body work safely.

---

13. Experimental protocol

The experiment should be evaluated as a sequence of gates rather than one headline number.

Gate A — substrate stability

The FP8/FP4 body must remain within its established loss, gradient, and trajectory bounds.

Quantizer state must not advance during recomputation, and independently evolving layers must not share mutable histories.

Gate B — INT2 learnability

Under the reduced learning rate:

• normalized prediction error should decline;
• predictor magnitude should remain bounded relative to the target;
• confidence should move rather than saturate on a telemetry artifact;
• and the predictor should not destabilize the trunk.

Gate C — calibration

A confidence claim made in one window should predict delivered coverage in a later window.

A high confidence number without low calibration error is not useful.

Gate D — pattern-local evidence

Coverage should be broken down by token or context bucket.

The hypothesis predicts that frequent patterns become predictable first.

A useful signature would be:

high-frequency patterns:
    confidence rises first
    residual magnitude falls first

rare or novel patterns:
    confidence stays low
    full computation remains active

Gate E — controlled authority

The governor should engage only where the predictor has demonstrated calibrated coverage.

Engagement should be gradual, reversible, and attributable.

Gate F — real compute reduction

Blending a predicted output with a fully computed output does not save FLOPs.

The experiment reaches its central systems claim only when the real body performs less work.

That may take the form of:

• reduced tail computation;
• conditional subcomponent execution;
• block- or tile-level residual work;
• or another kernel path designed around the predicted residual.

Gate G — capability parity

Held-out capability must remain within the accepted range.

Training loss alone is not sufficient.

Gate H — net benefit

The predictor, scoring path, calibration, governor, format conversion, and residual computation all consume resources.

The final comparison must include their cost.

The hypothesis succeeds operationally only when:

while capability and stability remain within the accepted range.

---

14. Failure conditions

A research hypothesis needs explicit ways to fail.

This hypothesis is weakened or rejected if:

1. INT2 prediction error does not fall under an appropriate learning rate.
2. The predictor repeatedly overshoots the stabilized FP4 target.
3. The predictor can improve its own objective only by influencing or destabilizing the trunk.
4. Confidence cannot be calibrated out of sample.
5. Predictability is too global or diffuse to support local compute reduction.
6. Frequent patterns do not become predictable before rare patterns.
7. The residual remains as expensive as the original operation.
8. Predictor and scoring overhead exceed the compute saved.
9. FP4 target motion remains too large for the INT2 head to track.
10. Compute reduction changes held-out capability beyond the accepted bound.
11. The system engages on familiar training patterns but fails under distribution shift.
12. Hardware conversion, unpacking, or expansion eliminates the theoretical low-bit advantage.
13. The system cannot define a safe unit of partial execution.

A failed result would still be useful.

It would identify whether the limiting factor is:

• representational capacity;
• target motion;
• optimizer behavior;
• calibration;
• engagement granularity;
• predictor architecture;
• residual-kernel structure;
• or hardware support.

---

15. Relationship to existing work

This hypothesis intersects several established research directions but does not reduce cleanly to any one of them.

Ultra-low-bit models

BitNet b1.58 demonstrates that ternary-weight language models can be trained as native low-bit models.

That work asks whether an entire model can operate with extremely low-bit weights.

The present hypothesis assigns an even coarser representation a narrower role:

Prediction within a higher-precision corrective envelope.

Low-bit transformer stability

Prior work on quantization variation identifies module sensitivity, outliers, and dynamic parameter oscillation as central problems in low-bit transformer training.

The IDA experiments add mutable quantizer-state timing, recomputation semantics, metadata provenance, and layer ownership as systems-level sources of instability.

Speculative decoding

Speculative decoding uses a cheaper draft model and a full-model verifier to accelerate autoregressive generation.

The conceptual overlap is draft and verification.

The difference is location:

speculative decoding:
    draft model predicts output tokens
    full model verifies token proposals

INT2 coarse prediction:
    low-bit head predicts internal body work
    full body computes the residual

Conditional computation

Conditional-computation research uses routing, gating, or low-rank estimates to avoid work predicted to be unnecessary.

The current proposal adds:

• a nested low-bit prediction surface;
• explicit confidence calibration;
• pattern-local authority;
• and a governor separating prediction quality from execution authority.

Low-rank approximation

The predictive head uses low-rank structure to estimate a larger transformation.

The distinctive question is whether that estimate can also be represented and trained as an INT2 coarse surface bounded by stabilized FP4 and FP8 representations.

Hidden-state prediction

Recent work has explored predicting future hidden states as a way to measure the complexity or novelty of internal computation.

That supports the broader idea that predictable hidden-state evolution contains information about where computation is routine.

The IDA hypothesis asks whether that predictability can become a governed low-bit compute surface.

---

16. Hardware scope

The current implementation runs on Hopper.

NVIDIA’s Hopper architecture exposes native FP8 Tensor Core computation, while NVIDIA’s Blackwell generation adds FP4 support through its newer Transformer Engine and Tensor Core capabilities.

Accordingly:

• the current Hopper run tests INT2/FP4 representation, learning, state management, and control;
• it does not establish native INT2 or FP4 Tensor Core acceleration;
• software packing may reduce some storage or bandwidth costs, but expansion into a supported compute format can dominate;
• and a future Blackwell-class implementation is the appropriate place to test the complete hardware-speed hypothesis.

The distinction should remain visible in every result:

representational success
    does not automatically imply

kernel-level speedup
    does not automatically imply

end-to-end training speedup
    does not automatically imply

capability preservation

All must be measured separately.

---

17. Predictions made by the hypothesis

A good hypothesis should produce observations that were not inserted after the result.

This one predicts:

1. The INT2 head should improve after the learning-rate reduction if the earlier failure was dominated by optimizer overshoot rather than inadequate representational capacity.
2. Frequent patterns should become predictable before rare patterns.
3. Prediction error should correlate with novelty or surprisal.
4. A single global confidence scalar should underperform per-pattern confidence.
5. The residual should become smaller or more structured as prediction improves.
6. A stable FP4 target should produce smoother INT2 learning than an unstable or state-contaminated target.
7. Compute savings should appear first in repeatedly mastered regions, not uniformly across the sequence.
8. Native low-bit hardware should improve the economic value of a predictor whose representational value can already be demonstrated on Hopper.

These predictions can be tested independently.

---

18. The broader idea

Quantization is usually presented as compression:

Represent the same computation with fewer bits.

Predictive supersampling suggests another role:

Use lower precision to forecast the predictable portion of computation, then spend higher precision only on the residual information that remains.

Under that view, precision is not one global model setting.

It becomes a hierarchy:

high precision:
    final responsibility and correction

middle precision:
    stable representation and bounding envelope

extreme low precision:
    coarse prediction of recurring structure

The two-bit surface is useful not because two bits suddenly contain everything.

It is useful because they are no longer asked to.

The full model remains responsible for truth.

The predictor is responsible for making repeated work cheaper.

---

19. Current status

The active experiment is training the INT2 predictive head with a reduced learning rate on the live server.

The key measurements from the run include:

normalized prediction error
predictor-to-target magnitude ratio
confidence
future delivered coverage
calibration error
dominant gradient ownership
trunk loss
trunk gradient behavior
held-out capability
predictor overhead

The result should not be summarized as successful merely because training remains finite.

The stronger questions are:

• Does the predictor settle?
• Does prediction error decline?
• Does confidence rise for the right reason?
• Does confidence accurately predict future coverage?
• Do common patterns become fluent first?
• Can authority be granted without destabilization?
• Can real body work be removed?
• Does the net system become faster?

Those measurements will determine whether the INT2 path becomes part of the production architecture, is redesigned, or is rejected.

---

Conclusion

The hypothesis is straightforward:

FP4 stabilization can make INT2 prediction useful by turning a noisy low-bit target into a bounded, repeatable surface whose predictable component can be learned and corrected residually.

The proposal does not claim that INT2 can replace the full body.

It does not claim that Hopper provides native execution for the entire format stack.

It does not claim that the current live run has already produced an end-to-end speedup.

It claims something narrower and testable:

1. Stabilize the FP8/FP4 substrate.
2. Preserve exact low-bit state through recomputation.
3. Train an INT2 coarse predictor inside that envelope.
4. Measure its future coverage externally.
5. Grant it only calibrated and reversible authority.
6. Reduce real body work where prediction has been earned.
7. Reject the design if the residual or overhead remains too expensive.

At the time of writing, the low-learning-rate INT2 training run is active.

Its value will not be decided by whether the idea sounds plausible.

It will be decided by whether prediction error falls, confidence calibrates, familiar patterns become cheap, capability holds, and the full body performs measurably less work.

INT2 is not being asked to replace the model’s reasoning.

It is being trained to predict the portion of the computation that no longer needs to be rediscovered at full resolution.

That is the experiment.

---

References

1. NVIDIA, NVIDIA Hopper Architecture In-Depth.
   https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/

2. NVIDIA, NVIDIA Blackwell Architecture.
   https://www.nvidia.com/en-us/data-center/technologies/blackwell-architecture/

3. Shuming Ma et al., The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits.
   https://arxiv.org/abs/2402.17764

4. Xijie Huang et al., Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precision.
   https://arxiv.org/abs/2307.00331

5. Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast Inference from Transformers via Speculative Decoding.
   https://arxiv.org/abs/2211.17192

6. Andrew Davis and Itamar Arel, Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks.
   https://arxiv.org/abs/1312.4461

7. Vincent Herrmann, Róbert Csordás, and Jürgen Schmidhuber, Measuring In-Context Computation Complexity via Hidden State Prediction.
   https://arxiv.org/abs/2503.13431

---

Scope note

The FP8, packed-FP4, and INT2 paths discussed here are independent software-managed experiments in the IDA native C++/CUDA trainer.

This article does not claim that NVIDIA Transformer Engine, cuDNN, PyTorch, bitsandbytes, or another external framework contains the same implementation, state model, or failure modes.