The Number That Was Never Measured

The Number That Was Never Measured

A learned estimator sitting inside a control loop cannot distinguish the instrument reported zero from the instrument reported nothing. Both leave as a confident number, and the loop acts on both. This is a measurement of that failure across three independent settings, including the control arm that killed the first version of the claim and the objection that forced the last experiment.

Everything below is reproducible from fixed seeds. Where something was not measured, it says so.

Start with the arm that failed

The original claim was that learned receivers, specifically, cannot represent an absence — that a neural decoder handed a channel with nothing in it renders its training prior into the silence and emits no flag.

The evidence looked strong. A forty-second synthetic source was generated by integrating a damped parametric-resonance equation across an instability threshold, giving a signal with landmarks fixed by construction: a subharmonic line that exists only above threshold, a known threshold-crossing time, and an inharmonic mode bank that is not a harmonic series. Four frequency bands are empty by construction. Push it through a neural codec at 3 kbps and measure band by band:

Two learned codecs from different labs, with different architectures and different training data, both manufacture 45–75 dB of energy into bands that contain nothing, while tracking the real content within 1 dB. That reproduces, and it is a real effect.

And then the third row. Opus is a classical transform codec — no learning anywhere in it — running at twice the bitrate. It fills empty bands too.

So the claim as stated was false. Empty-band lift is not diagnostic of a learned receiver. Any lossy codec does it, and a reviewer would have said so in one sentence. The control arm was not requested by anyone; it was added specifically to give the boring explanation a chance to win, and it won.

What survives is narrower. The separator is not the lift, it is the pairing. The classical codec degrades visibly where you can check it — 3.3 dB of error in the occupied bands — and that visible degradation is itself a warning that something was lost. The learned codecs are four to five times more faithful where you can check, and equally inventive where you cannot. Invention per unit of visible error: Mimi 100×, EnCodec 69×, Opus 17×.

> Fidelity where verifiable is not evidence of provenance where unverifiable. For learned receivers, the two move in opposite directions.

Less quotable than the claim I started with. Actually true.

Out of audio, where it matters

Codecs are a proxy. The setting that matters is an estimator recovering the state of a channel from a lossy measurement — the shape of estimating a wavefront directly from images, or a polarization state from a drifting fibre.

Sixteen modes, forty-eight quadratic intensity measurements, sign destroyed by the squaring, and a measurement operator built blind to four of the modes. Then four conditions, crossing was absence in the training data with does the instrument see this mode at all:

The estimator emits amplitude 0.186 into modes that are exactly zero, against a typical real mode of 0.351. Half a real signal, invented, with nothing marking it.

Training on absence does not fix it. Mixing genuine zeros into the training distribution moves the failure from 105.4 to 101.4 dB — a roughly uniform shrinkage of everything, not an acquired ability to say empty. There is no mechanism in the model that distinguishes told zero from told nothing, and more data does not create one.

The part that should worry you

The standard answer to how do I know the model is making this up is uncertainty quantification: train an ensemble, watch the members disagree.

Measured here, the five-member ensemble disagrees less on the modes it is entirely fabricating (std 0.015) than on the modes it actually measures (0.049).

Confidence inverts. The members share a prior, so in the directions where the data says nothing they all fall back to the same place and agree beautifully. Used as a gate, ensemble disagreement selects the wrong values to distrust.

"That is circular" — and it was

The first version of the channel experiment deserved the obvious objection, and got it from an independent reviewer: the operator was built blind to those modes, so detecting the blindness proves nothing. Detecting an absence you encoded is not prediction.

Correct. So the experiment was rebuilt to remove the circularity rather than argued with:

• the estimator trains only on a healthy instrument, with no blind modes anywhere;
• at test time a random, undisclosed subset of modes loses its measurement gain — a dead actuator, the ordinary failure of a real adaptive-optics system — drawn fresh on each of 400 trials;
• the mask is hidden from the estimator and from the detector;
• the detector receives only a noisy estimate of the live instrument, because no real calibration is exact.

Detection scored by AUC, so no threshold is tuned:

Error on the drifted modes runs 1.7–1.8× that on healthy modes, so the failure genuinely damages the estimate — and neither standard signal finds it.

Correction, added the same day: that 1.000 does not generalise

The table above uses one operator family — dense, independent, identically distributed. That is the friendly case, and publishing it alone would have been the third mistake in this piece.

Re-run across two structurally different families, three seeds each, retraining every time:

A banded operator — each sensor element seeing only nearby modes, which is what real subaperture geometry looks like — knocks the detector down to 0.91–0.93. Still useful. Not perfect. The honest range is 0.910 to 1.000, mean 0.960, and the degradation tracks column correlation.

The same run also softens the sharpest sentence in this piece. Ensemble variance is below chance on the independent family and merely at chance on the banded one, once nudging 0.512. So the claim is not "uncertainty quantification is anti-diagnostic." It is:

> Ensemble disagreement carried no usable signal for this failure in any configuration tested — 0.459 to 0.512 across six runs. It is never a detector. It is simply not measuring this.

Weaker sentence. Survives contact.

What it costs to run

The obvious implementation builds a three-index tensor and would have made the honest answer "too slow." It collapses to a single matrix-vector product against the elementwise-squared operator — verified numerically identical to the naive form to 2×10⁻¹⁶ before being timed, because an optimisation not checked against the thing it replaces is just a different function.

Against a 333 µs budget, the per-frame cost at a 3 kHz control rate:

That last row said "does not fit at the largest apertures," and I wrote that single precision and sparsity would probably close it. Both guesses were then measured. Both were wrong, and so was the conclusion:

Single precision bought nothing, so the bottleneck was never the memory bandwidth I asserted it was. And the original 19×-over figure was a dense kernel run against a structurally sparse operator — two implementation artifacts stacked into something that sounded like a physical limit.

It is real-time at every scale tested, provided the operator is banded — which real subaperture geometry is. A genuinely dense operator that size does not fit in any variant measured. The limit belongs to the operator's structure, not to the test.

What the fix is, and what it is not

The test is deliberately unremarkable: how much does the measurement respond to this mode, relative to noise. A Jacobian column norm. It is a standard observability check and I claim no novelty in the mathematics.

What is worth something is the discipline: requiring the estimator to emit, per coefficient, a type — measured, or no signal — computed from the instrument rather than from the model's opinion of itself. The separation is not delicate. Blind-mode response 0.0027 against observable-mode response 0.176: the populations do not overlap, with a 65× margin, so the verdict does not rest on a tuned constant.

Because the test is small, its soundness core can be checked by machine. Formalised in Lean 4, with no mathematics library in the trusted base: if a measurement is invariant to a coordinate then every estimator's output is independent of the truth in that coordinate, so a number emitted there is prior, not observation. Nothing in the proof mentions neural networks, which is why no larger model escapes it. The proof checks; a deliberately broken copy is rejected — a checker that has never rejected anything is not a checker.

And what it does not do. It types provenance, not accuracy. In conditions 2 and 3 above, on modes that are genuinely observable but badly estimated, the test correctly stays silent and most of the error remains. It answers was this measured. It never answers is this right. That limit is recorded as a theorem with an explicit counterexample rather than as a footnote, because the version of this idea that quietly claims both is the version that gets someone hurt.

What has not been established

Every measurement here is synthetic or audio. No real polarization trace and no real wavefront-sensor frame has been touched. No latency budget exists for the test at realistic control-loop rates, where deformable mirrors run to several kilohertz — it is a Jacobian column norm and it should be cheap, but should-be is not measured. The drift result is one operator family, one noise model, one seed.

Those are the four things I would attack first, and they are listed here rather than left to be found.