# Does the eps knee follow W_c, the init norm, or the logit scale? Width/embedding-dim sweep at p = 97

Date: 2026-10-02. Scripts: `clamp_gen.py` (new flags `--d_hid`, `--d_emb`; tags get `_h{H}` / `_e{E}` only when non-default, so older tags are unchanged), `width_summarize.py` (writes `width_dose_response.csv` and `width_tables.md`), `fig_width_knee.py` (writes `fig_width_knee.png`). Run stdout: `run_width.log`. Queue/worker: `width_queue.txt`, `width_worker.sh`.

## Question
At p = 59/97/113 the eps-induced knee in time-to-grok vs clamped norm N sat within ~10% of the free-run critical norm W_c and just above the default init norm. All three candidate quantities (W_c, init norm, logit scale per unit norm) moved together with p. Changing the architecture at fixed p = 97 separates them. With PyTorch default init the init norm is dominated by the N(0,1) embedding (norm ≈ sqrt(97·d_emb)), so:
- changing d_hid (A: 128, B: 512) leaves the init norm at ~80 but changes the logit scale per unit norm;
- changing d_emb (C: 32, D: 128) moves the init norm to 57 / 112, while the free-run W_c stays at 79-94.

## Method
- Same as the modulus sweep: 2-layer MLP, (a+b) mod 97, 50% train, full batch, AdamW lr 1e-3, betas (0.9, 0.98), fp32, one thread. Clamped runs use `--clamp total --wd 0`, with all parameters rescaled to total L2 norm N after every step, `--steps 12000`, and early stop 250 steps after test >= 0.99.
- Free run per variant: `--clamp none --wd 1 --steps 8000`, seed 0. W_c is norm_total at the first logged step with test acc >= 0.5 and >= 0.99.
- T = first logged step (50-step resolution) with test acc >= 0.99. A run that never reaches it is censored at its last step ('>').
- Knee = smallest N where mean T(eps 1e-8) / mean T(eps 1e-16) >= 1.5, or where eps 1e-8 is censored and 1e-16 is not. Seed 0 is used at every N. Seeds 1 (and 2) are added at the norms around each knee.
- grad_rms = raw gradient norm / sqrt(nparam) before the step. "min grad_rms" is its minimum over the run, and logit_std is the train-logit std at that step.
- Reference (d_hid 256, d_emb 64): T values come from `../temp-clamp/dose_response.csv`, seeds 0-2; its eps 1e-8 entries at N <= 82 came from the older `norm-clamp` script, which logged no gradients. Gradient stats come from the `../temp-clamp/logs` files that have grad_norm. Added here: N=82 eps 1e-8 seed 0 and N=85 eps 1e-16 seed 0, both run with clamp_gen.py. The free run is `../temp-clamp/logs/none0_lr0.001_T1_seed0_wd1.csv`.

## Main table

| variant | d_hid | d_emb | nparam | init norm | W_c free run (test 0.5 / 0.99) | free-run grok step | knee N (eps 1e-8) | last N below knee (ratio) | knee / W_c(0.5) | knee / W_c(0.99) | knee / init | min grad_rms, eps 1e-8: below / at knee | logit_std there: below / at knee |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ref | 256 | 64 | 64161 | 79.6 | 88.4 / 78.7 | 2450 | 88 (2.65x) | 85 (1.34x) | 1.00 | 1.12 | 1.10 | 4.0e-10 / 4.1e-10 | 5.8 / 6.0 |
| A | 128 | 64 | 35233 | 79.4 | 80.1 / 76.2 | 2350 | 82 (1.67x) | 79 (1.15x) | 1.02 | 1.08 | 1.03 | 9.8e-10 / 4.8e-10 | 6.7 / 7.1 |
| B | 512 | 64 | 122017 | 80.2 | 89.1 / 80.9 | 2500 | 91 (2.42x) | 88 (1.10x) | 1.02 | 1.13 | 1.13 | 7.2e-10 / 3.5e-10 | 4.9 / 5.3 |
| C | 256 | 32 | 44673 | 56.9 | 93.4 / 79.8 | 3200 | 95 (2.73x) | 90 (1.37x, seeds 1.09/2.00/1.07) | 1.02 | 1.19 | 1.67 | 4.1e-10 / 3.8e-10 | 8.2 / 8.6 |
| D | 256 | 128 | 103137 | 112.4 | 89.6 / 78.6 | 2650 | 88 (3.42x) | 84 (1.35x) | 0.98 | 1.12 | 0.78 | 3.5e-10 / 3.4e-10 | 4.8 / 5.2 |

Init norm is norm_total at step 0 of the seed-0 free run, which is also the default-init norm of the clamped model before the first projection. Seed 1 init norms are within about 1 (e.g. 80.3 for A, 58.1 for C, 112.5 for D). Grid spacing near the knees is 3-6, so each knee is located only to within that bracket.

![fig](fig_width_knee.png)

`fig_width_knee.png`: one panel per variant showing T vs N for eps 1e-8 (blue) and 1e-16 (orange). Dots are individual seeds, lines are means, and triangles are censored runs. The aqua band spans the free-run W_c (test 0.99 to test 0.5), and the yellow dashed line marks the default init norm.

## Mechanism check: when the gradient collapses, not how far

The minimum grad_rms does **not** separate runs below the knee from runs above it. Every run near the knee reaches 3-10e-10, including runs that are not slowed (B N=88: 6.5-8.0e-10 with ratio 1.1; ref N=85: 4.5e-10 with 1.36x). What separates them is **when** grad_rms first falls below 1e-9 (10x under eps), measured relative to the time the same run needs to grok with eps 1e-16.

| variant | N | seed | T eps 1e-8 | T eps 1e-16 | ratio | first step grad_rms < 1e-9 (eps 1e-8) | crossing / T(1e-16) | min grad_rms (eps 1e-8) |
|---|---|---|---|---|---|---|---|---|
| ref | 82 | 0 | 1900 | 1850 | 1.03 | 2150 | 1.16 | 9.27e-10 |
| ref | 85 | 0 | 2650 | 1950 | 1.36 | 2150 | 1.10 | 4.45e-10 |
| ref | 88 | 0 | 5200 | 2100 | 2.48 | 2150 | 1.02 | 4.57e-10 |
| ref | 88 | 1 | 6400 | 2300 | 2.78 | 2400 | 1.04 | 3.62e-10 |
| A | 79 | 0 | 2300 | 2200 | 1.05 | - | - | 1.31e-09 |
| A | 79 | 1 | 2800 | 2250 | 1.24 | 2900 | 1.29 | 7.28e-10 |
| A | 82 | 0 | 3550 | 2350 | 1.51 | 2450 | 1.04 | 5.02e-10 |
| A | 82 | 1 | 4900 | 2700 | 1.81 | 2750 | 1.02 | 4.58e-10 |
| A | 88 | 0 | 7350 | 2600 | 2.83 | 2400 | 0.92 | 5.21e-10 |
| A | 88 | 1 | 9300 | 2750 | 3.38 | 2550 | 0.93 | 4.49e-10 |
| A | 96 | 0 | >12000 | 2850 | 4.21 | 2250 | 0.79 | 5.28e-10 |
| A | 104 | 0 | >12000 | 3200 | 3.75 | 2050 | 0.64 | 5.71e-10 |
| B | 88 | 0 | 1800 | 1650 | 1.09 | 2000 | 1.21 | 6.45e-10 |
| B | 88 | 1 | 1950 | 1750 | 1.11 | 2200 | 1.26 | 8.03e-10 |
| B | 91 | 0 | 4350 | 1800 | 2.42 | 1950 | 1.08 | 3.53e-10 |
| B | 94 | 0 | 7000 | 1900 | 3.68 | 1950 | 1.03 | 3.88e-10 |
| B | 94 | 1 | 9150 | 2050 | 4.46 | 2050 | 1.00 | 3.3e-10 |
| B | 104 | 0 | 11100 | 2500 | 4.44 | 1650 | 0.66 | 4.82e-10 |
| C | 90 | 0 | 3050 | 2800 | 1.09 | 3000 | 1.07 | 3.79e-10 |
| C | 90 | 1 | 5300 | 2650 | 2.00 | 2800 | 1.06 | 3.97e-10 |
| C | 90 | 2 | 3200 | 3000 | 1.07 | 3300 | 1.10 | 4.67e-10 |
| C | 95 | 0 | 9150 | 3350 | 2.73 | 3250 | 0.97 | 3.75e-10 |
| C | 100 | 0 | >12000 | 3700 | 3.24 | 3350 | 0.91 | 4.18e-10 |
| C | 100 | 1 | >12000 | 3400 | 3.53 | 3050 | 0.90 | 4.69e-10 |
| C | 110 | 0 | >12000 | 4300 | 2.79 | 4050 | 0.94 | 6.1e-10 |
| D | 84 | 0 | 2450 | 1750 | 1.40 | 1950 | 1.11 | 3.44e-10 |
| D | 84 | 1 | 2100 | 1700 | 1.24 | 1850 | 1.09 | 3.8e-10 |
| D | 84 | 2 | 2250 | 1600 | 1.41 | 1800 | 1.12 | 3.39e-10 |
| D | 88 | 0 | 7000 | 1950 | 3.59 | 1850 | 0.95 | 3.45e-10 |
| D | 88 | 1 | 6150 | 1900 | 3.24 | 1750 | 0.92 | 3.42e-10 |
| D | 92 | 0 | 9350 | 2100 | 4.45 | 1700 | 0.81 | 3.35e-10 |
| D | 92 | 1 | 8750 | 2100 | 4.17 | 1650 | 0.79 | 3.44e-10 |
| D | 100 | 0 | 11400 | 2300 | 4.96 | 1550 | 0.67 | 3.59e-10 |

Rule from this table: the run is strongly slowed (ratio >= 2) whenever the eps 1e-8 run's gradient crosses 1e-9 at or before ~1.05 x T(eps 1e-16). It is unaffected or mildly slowed (1.0-1.4x) when the crossing comes >= ~1.08-1.3 x T(1e-16), i.e. after generalization is essentially done. The boundary is fuzzy in the 1.06-1.10 band (C N=90: crossings at 1.06/1.07/1.10 give ratios 2.00/1.09/1.07; B N=91 at 1.08 gives 2.42). The crossing step moves earlier as N grows (saturation comes faster at larger norm), while T(1e-16) moves later. The knee is the N where the two cross. This refines the "min grad_rms < 1e-9" criterion from the modulus sweep: the gradient falling 10x below eps is necessary, but it slows grokking only if it happens before the test-set circuit has formed.

## Conclusion
- **The knee does not follow the init norm.** With d_emb 32 the init norm drops to 57 but the knee stays at N ≈ 90-95 (knee/init 1.67). With d_emb 128 the init norm rises to 112 and the knee is at 84-88, *below* the init norm (knee/init 0.78). At d_emb 128 the clamped runs with N = 66-84, far under the init norm, are fine with both eps values.
- **The knee follows the free-run norm at test 0.5 (W_c(0.5)) closely.** knee/W_c(0.5) = 1.00, 1.02, 1.02, 1.02, 0.98 across all five architectures. The knee moves from 82 (A) to 95 (C), and W_c(0.5) moves in the same order: A 80 < ref 88 ≈ B 89 ≈ D 90 < C 93. The norm at test 0.99 (76-81) varies less than the knee does and ranks the variants differently, so knee/W_c(0.99) is looser (1.08-1.19). The earlier modulus-sweep statement "knee within ~10% of W_c" holds best for the test-0.5 definition.
- **A fixed logit scale does not set it.** At the knee, logit_std at minimum gradient ranges from 5.2 (D) to 8.6 (C), so no single logit-scale threshold marks it. The gradient-crossing quantity does predict the knee, but as a timing condition (gradient falls 10x under eps before the run has generalized, per the table above), not as a fixed logit or gradient magnitude. Since the free-run W_c(0.5) is itself the norm reached at the point where generalization is underway, the W_c match and the timing mechanism are consistent with each other. The clamped run is damaged by eps exactly when its norm is high enough to saturate (and collapse the gradient) before reaching the generalizing solution, and that threshold coincides with the norm the unclamped run has when it generalizes.
- Hidden width shifts the knee modestly and not monotonically (128: 82, 256: 88, 512: 91). The init norm is constant across these three, and W_c(0.5) follows the knee (80, 88, 89).

## Caveats
- Seed 0 only at most N; 2-3 seeds only around the knees. Seed spread at the knee is large (C N=90: 3050/5300/3200 for eps 1e-8). Knees are bracketed to 3-6 norm units, so ratios of 0.98-1.02 are within resolution, not evidence of exact equality.
- W_c comes from a single free run (seed 0, wd 1) per variant. The modulus sweep saw ~1-2 unit seed differences in W_c.
- The ratio criterion (1.5x) is somewhat arbitrary. Several N just below the knee show partial slowing (1.24-1.41x: ref 85, A 79 seed 1, C 90, D 84).
- Reference eps 1e-8 points at N <= 82 come from an older script (norm-clamp) with no gradient logging. The reference N=85 has one eps 1e-16 seed.
- grad_rms is an all-parameter RMS, not the per-coordinate distribution. Logged every 50 steps.
- Censored runs (12000-step cap) give lower-bound ratios. Only p = 97, lr 1e-3, train fraction 0.5, default PyTorch init.

## Full per-run and per-N tables
Generated by `width_summarize.py` (also written to `width_tables.md`):

### Variant ref: d_hid 256, d_emb 64, nparam 64161

Free run (wd 1, seed 0): init norm 79.6; test 0.5 at step 1750 norm 88.38; test 0.99 at step 2450 norm 78.71; max norm 90.6

| N | eps | seed | T (step test>=0.99) | min grad_rms | logit_std @ min grad | step frac_sat>=0.99 | step grad_rms<1e-9 | final max_logit | final test |
|---|---|---|---|---|---|---|---|---|---|
| 50 | 1e-8 | 0 | 1150 | 1.49e-04 | 0.0638 | - | - | 10.73 | 1.000 |
| 50 | 1e-8 | 1 | 1500 | 1.47e-04 | 0.0604 | - | - | 10.41 | 0.999 |
| 50 | 1e-8 | 2 | 2800 | - | - | - | - | - | 0.995 |
| 50 | 1e-16 | 0 | 1150 | 1.49e-04 | 0.0638 | - | - | 10.03 | 1.000 |
| 50 | 1e-16 | 1 | 1300 | 1.47e-04 | 0.0604 | - | - | 10.26 | 1.000 |
| 50 | 1e-16 | 2 | 2250 | 1.51e-04 | 0.0628 | - | - | 10.45 | 0.993 |
| 58 | 1e-8 | 0 | 1150 | - | - | - | - | - | 1.000 |
| 58 | 1e-8 | 1 | 1400 | - | - | - | - | - | 1.000 |
| 58 | 1e-8 | 2 | 1500 | - | - | - | - | - | 1.000 |
| 58 | 1e-16 | 0 | 1200 | 4.90e-05 | 2.55 | - | - | 14.22 | 1.000 |
| 58 | 1e-16 | 1 | 1400 | 7.69e-05 | 2.36 | - | - | 13.21 | 1.000 |
| 58 | 1e-16 | 2 | 1550 | 8.62e-05 | 2.37 | - | - | 12.98 | 1.000 |
| 65 | 1e-8 | 0 | 1350 | - | - | - | - | - | 1.000 |
| 65 | 1e-8 | 1 | 1500 | - | - | - | - | - | 1.000 |
| 65 | 1e-8 | 2 | 1550 | - | - | - | - | - | 1.000 |
| 65 | 1e-16 | 0 | 1350 | 5.47e-06 | 3.17 | - | - | 18.27 | 1.000 |
| 65 | 1e-16 | 1 | 1500 | 9.46e-06 | 2.59 | - | - | 16.38 | 1.000 |
| 65 | 1e-16 | 2 | 1550 | 1.20e-05 | 2.76 | - | - | 16.3 | 0.999 |
| 75 | 1e-8 | 0 | 1550 | - | - | - | - | - | 1.000 |
| 75 | 1e-8 | 1 | 1700 | - | - | - | - | - | 1.000 |
| 75 | 1e-8 | 2 | 1800 | - | - | - | - | - | 1.000 |
| 75 | 1e-16 | 0 | 1550 | 6.75e-08 | 4.44 | 1700 | - | 25.71 | 1.000 |
| 75 | 1e-16 | 1 | 1700 | 2.30e-07 | 3.57 | - | - | 22.71 | 1.000 |
| 75 | 1e-16 | 2 | 1800 | 1.31e-07 | 3.88 | - | - | 23.27 | 1.000 |
| 82 | 1e-8 | 0 | 1900 | 9.27e-10 | 5.61 | 1500 | 2150 | 29.4 | 0.999 |
| 82 | 1e-8 | 1 | 2050 | - | - | - | - | - | 0.999 |
| 82 | 1e-8 | 2 | 2150 | - | - | - | - | - | 0.999 |
| 82 | 1e-16 | 0 | 1850 | 1.76e-09 | 5.31 | 1550 | - | 26.67 | 1.000 |
| 82 | 1e-16 | 1 | 2000 | 5.31e-09 | 4.7 | 1750 | - | 28.39 | 1.000 |
| 82 | 1e-16 | 2 | 2100 | 8.97e-09 | 4.66 | 1800 | - | 27.17 | 0.999 |
| 85 | 1e-8 | 0 | 2650 | 4.45e-10 | 5.96 | 1450 | 2150 | 30.96 | 0.992 |
| 85 | 1e-8 | 1 | 2700 | 3.55e-10 | 5.63 | 1650 | 2400 | 26.0 | 0.994 |
| 85 | 1e-8 | 2 | 2500 | 3.94e-10 | 5.76 | 1550 | 2250 | 27.47 | 0.994 |
| 85 | 1e-16 | 0 | 1950 | 8.48e-10 | 5.7 | 1450 | 2050 | 43.45 | 1.000 |
| 88 | 1e-8 | 0 | 5200 | 4.57e-10 | 6.06 | 1400 | 2150 | 78.36 | 0.994 |
| 88 | 1e-8 | 1 | 6400 | 3.62e-10 | 6.01 | 1600 | 2400 | 71.75 | 0.993 |
| 88 | 1e-8 | 2 | 5900 | - | - | - | - | - | 0.993 |
| 88 | 1e-16 | 0 | 2100 | 7.44e-10 | 5.9 | 1400 | 1950 | 77.67 | 1.000 |
| 88 | 1e-16 | 1 | 2300 | 8.21e-10 | 5.54 | 1600 | 2250 | 52.45 | 1.000 |
| 88 | 1e-16 | 2 | 2200 | 7.76e-10 | 5.72 | 1550 | 2200 | 60.65 | 1.000 |
| 94 | 1e-8 | 0 | 8400 | - | - | - | - | - | 0.995 |
| 94 | 1e-16 | 0 | 2400 | 8.19e-10 | 6.64 | 1250 | 1950 | 214.07 | 1.000 |
| 94 | 1e-16 | 1 | 2550 | 8.27e-10 | 6.36 | 1300 | 2050 | 171.17 | 1.000 |
| 94 | 1e-16 | 2 | 2600 | 8.14e-10 | 6.27 | 1350 | 2150 | 199.64 | 1.000 |
| 100 | 1e-8 | 0 | 11300 | - | - | - | - | - | 0.994 |
| 100 | 1e-8 | 1 | 13650 | - | - | - | - | - | 0.995 |
| 100 | 1e-8 | 2 | 12450 | - | - | - | - | - | 0.994 |
| 100 | 1e-16 | 0 | 2700 | 8.02e-10 | 7.44 | 1150 | 1800 | 424.9 | 1.000 |
| 100 | 1e-16 | 1 | 2950 | 8.27e-10 | 7.31 | 1150 | 1900 | 391.44 | 1.000 |
| 100 | 1e-16 | 2 | 2750 | 8.13e-10 | 7.39 | 1200 | 1850 | 437.12 | 1.000 |
| 120 | 1e-8 | 0 | 14600 | - | - | - | - | - | 0.995 |
| 120 | 1e-16 | 0 | 3500 | 7.20e-10 | 10.9 | 1000 | 1550 | 1115.95 | 0.999 |
| 120 | 1e-16 | 1 | 3850 | 6.99e-10 | 10.4 | 1000 | 1500 | 933.51 | 0.998 |
| 120 | 1e-16 | 2 | 4000 | 7.40e-10 | 10.8 | 1050 | 1550 | 1001.68 | 0.999 |
| 150 | 1e-8 | 0 | >8000 | - | - | - | - | - | 0.010 |
| 150 | 1e-16 | 0 | >12000 | 8.26e-10 | 11.9 | 950 | 1450 | 1504.15 | 0.724 |
| 150 | 1e-16 | 1 | >12000 | 8.27e-10 | 13.1 | 950 | 1450 | 1650.17 | 0.509 |
| 150 | 1e-16 | 2 | >12000 | 8.30e-10 | 12.2 | 950 | 1450 | 1358.98 | 0.766 |

| N | mean T eps 1e-8 (n) | mean T eps 1e-16 (n) | ratio | mean min grad_rms eps 1e-8 | logit_std @ min grad, eps 1e-8 |
|---|---|---|---|---|---|
| 50 | 1817 (3) | 1567 (3) | 1.16 | 1.48e-04 | 0.0621 |
| 58 | 1350 (3) | 1383 (3) | 0.98 | - | - |
| 65 | 1467 (3) | 1467 (3) | 1.00 | - | - |
| 75 | 1683 (3) | 1683 (3) | 1.00 | - | - |
| 82 | 2033 (3) | 1983 (3) | 1.03 | 9.27e-10 | 5.61 |
| 85 | 2617 (3) | 1950 (1) | 1.34 | 3.96e-10 | 5.78 |
| 88 | 5833 (3) | 2200 (3) | 2.65 | 4.07e-10 | 6.03 |
| 94 | 8400 (1) | 2517 (3) | 3.34 | - | - |
| 100 | 12467 (3) | 2800 (3) | 4.45 | - | - |
| 120 | 14600 (1) | 3783 (3) | 3.86 | - | - |
| 150 | >8000 (1) | >12000 (3) | >=0.67 | - | - |

Knee (smallest N with ratio>=1.5): N=88; last N below: 85.0

### Variant A: d_hid 128, d_emb 64, nparam 35233

Free run (wd 1, seed 0): init norm 79.4; test 0.5 at step 1550 norm 80.1; test 0.99 at step 2350 norm 76.24; max norm 81.6

| N | eps | seed | T (step test>=0.99) | min grad_rms | logit_std @ min grad | step frac_sat>=0.99 | step grad_rms<1e-9 | final max_logit | final test |
|---|---|---|---|---|---|---|---|---|---|
| 60 | 1e-8 | 0 | 1900 | 2.44e-05 | 2.62 | - | - | 16.37 | 0.999 |
| 60 | 1e-16 | 0 | 1950 | 2.83e-05 | 2.68 | - | - | 17.23 | 0.997 |
| 70 | 1e-8 | 0 | 1900 | 1.93e-07 | 4.59 | - | - | 28.26 | 1.000 |
| 70 | 1e-16 | 0 | 1900 | 2.00e-07 | 4.47 | - | - | 27.83 | 1.000 |
| 76 | 1e-8 | 0 | 2200 | 4.49e-09 | 6 | 1900 | - | 32.62 | 0.997 |
| 76 | 1e-16 | 0 | 2100 | 5.85e-09 | 5.87 | 1850 | - | 32.58 | 0.999 |
| 79 | 1e-8 | 0 | 2300 | 1.31e-09 | 6.47 | 1800 | - | 34.44 | 0.997 |
| 79 | 1e-8 | 1 | 2800 | 7.28e-10 | 7.01 | 1950 | 2900 | 34.68 | 0.996 |
| 79 | 1e-16 | 0 | 2200 | 1.35e-09 | 6.34 | 1750 | - | 31.03 | 1.000 |
| 79 | 1e-16 | 1 | 2250 | 2.13e-09 | 6.22 | 1850 | - | 31.31 | 0.999 |
| 82 | 1e-8 | 0 | 3550 | 5.02e-10 | 7.02 | 1650 | 2450 | 41.05 | 0.992 |
| 82 | 1e-8 | 1 | 4900 | 4.58e-10 | 7.21 | 1800 | 2750 | 54.78 | 0.991 |
| 82 | 1e-16 | 0 | 2350 | 8.88e-10 | 7.03 | 1700 | 2400 | 54.23 | 0.999 |
| 82 | 1e-16 | 1 | 2700 | 9.65e-10 | 6.56 | 1850 | 2800 | 48.48 | 0.999 |
| 88 | 1e-8 | 0 | 7350 | 5.21e-10 | 8.13 | 1550 | 2400 | 111.76 | 0.992 |
| 88 | 1e-8 | 1 | 9300 | 4.49e-10 | 8.31 | 1700 | 2550 | 113.55 | 0.992 |
| 88 | 1e-16 | 0 | 2600 | 8.34e-10 | 7.9 | 1550 | 2200 | 141.2 | 1.000 |
| 88 | 1e-16 | 1 | 2750 | 6.89e-10 | 7.97 | 1700 | 2300 | 149.73 | 0.999 |
| 96 | 1e-8 | 0 | >12000 | 5.28e-10 | 9.71 | 1450 | 2250 | 280.51 | 0.985 |
| 96 | 1e-16 | 0 | 2850 | 7.03e-10 | 9.44 | 1450 | 2050 | 512.31 | 0.999 |
| 104 | 1e-8 | 0 | >12000 | 5.71e-10 | 11 | 1350 | 2050 | 481.34 | 0.939 |
| 104 | 1e-16 | 0 | 3200 | 6.87e-10 | 10.8 | 1350 | 1900 | 820.75 | 0.999 |

| N | mean T eps 1e-8 (n) | mean T eps 1e-16 (n) | ratio | mean min grad_rms eps 1e-8 | logit_std @ min grad, eps 1e-8 |
|---|---|---|---|---|---|
| 60 | 1900 (1) | 1950 (1) | 0.97 | 2.44e-05 | 2.62 |
| 70 | 1900 (1) | 1900 (1) | 1.00 | 1.93e-07 | 4.59 |
| 76 | 2200 (1) | 2100 (1) | 1.05 | 4.49e-09 | 6 |
| 79 | 2550 (2) | 2225 (2) | 1.15 | 9.77e-10 | 6.74 |
| 82 | 4225 (2) | 2525 (2) | 1.67 | 4.80e-10 | 7.11 |
| 88 | 8325 (2) | 2675 (2) | 3.11 | 4.83e-10 | 8.22 |
| 96 | >12000 (1) | 2850 (1) | >=4.21 | 5.28e-10 | 9.71 |
| 104 | >12000 (1) | 3200 (1) | >=3.75 | 5.71e-10 | 11 |

Knee (smallest N with ratio>=1.5): N=82; last N below: 79.0

### Variant B: d_hid 512, d_emb 64, nparam 122017

Free run (wd 1, seed 0): init norm 80.2; test 0.5 at step 1900 norm 89.08; test 0.99 at step 2500 norm 80.86; max norm 95.5

| N | eps | seed | T (step test>=0.99) | min grad_rms | logit_std @ min grad | step frac_sat>=0.99 | step grad_rms<1e-9 | final max_logit | final test |
|---|---|---|---|---|---|---|---|---|---|
| 66 | 1e-8 | 0 | 1150 | 2.48e-05 | 2.47 | - | - | 13.94 | 1.000 |
| 66 | 1e-16 | 0 | 1100 | 2.08e-05 | 2.55 | - | - | 14.36 | 1.000 |
| 76 | 1e-8 | 0 | 1250 | 7.66e-07 | 3.23 | - | - | 18.38 | 1.000 |
| 76 | 1e-16 | 0 | 1250 | 7.64e-07 | 3.2 | - | - | 18.33 | 1.000 |
| 82 | 1e-8 | 0 | 1450 | 4.78e-08 | 3.96 | 1600 | - | 22.78 | 1.000 |
| 82 | 1e-16 | 0 | 1450 | 5.53e-08 | 3.86 | 1600 | - | 22.84 | 1.000 |
| 88 | 1e-8 | 0 | 1800 | 6.45e-10 | 5.09 | 1450 | 2000 | 26.03 | 0.999 |
| 88 | 1e-8 | 1 | 1950 | 8.03e-10 | 4.68 | 1600 | 2200 | 25.91 | 0.999 |
| 88 | 1e-16 | 0 | 1650 | 4.97e-09 | 4.48 | 1450 | - | 24.83 | 1.000 |
| 88 | 1e-16 | 1 | 1750 | 1.19e-08 | 3.76 | 1600 | - | 23.87 | 1.000 |
| 91 | 1e-8 | 0 | 4350 | 3.53e-10 | 5.27 | 1200 | 1950 | 62.96 | 0.993 |
| 91 | 1e-16 | 0 | 1800 | 1.34e-09 | 4.63 | 1250 | - | 21.67 | 1.000 |
| 94 | 1e-8 | 0 | 7000 | 3.88e-10 | 5.53 | 1050 | 1950 | 139.3 | 0.993 |
| 94 | 1e-8 | 1 | 9150 | 3.30e-10 | 5.18 | 1100 | 2050 | 147.41 | 0.992 |
| 94 | 1e-16 | 0 | 1900 | 7.25e-10 | 5.09 | 1050 | 1950 | 42.12 | 1.000 |
| 94 | 1e-16 | 1 | 2050 | 9.62e-10 | 4.41 | 1100 | 2300 | 26.95 | 1.000 |
| 104 | 1e-8 | 0 | 11100 | 4.82e-10 | 6.53 | 900 | 1650 | 699.2 | 0.998 |
| 104 | 1e-16 | 0 | 2500 | 7.29e-10 | 6.25 | 900 | 1500 | 163.86 | 1.000 |

| N | mean T eps 1e-8 (n) | mean T eps 1e-16 (n) | ratio | mean min grad_rms eps 1e-8 | logit_std @ min grad, eps 1e-8 |
|---|---|---|---|---|---|
| 66 | 1150 (1) | 1100 (1) | 1.05 | 2.48e-05 | 2.47 |
| 76 | 1250 (1) | 1250 (1) | 1.00 | 7.66e-07 | 3.23 |
| 82 | 1450 (1) | 1450 (1) | 1.00 | 4.78e-08 | 3.96 |
| 88 | 1875 (2) | 1700 (2) | 1.10 | 7.20e-10 | 4.89 |
| 91 | 4350 (1) | 1800 (1) | 2.42 | 3.53e-10 | 5.27 |
| 94 | 8075 (2) | 1975 (2) | 4.09 | 3.58e-10 | 5.36 |
| 104 | 11100 (1) | 2500 (1) | 4.44 | 4.82e-10 | 6.53 |

Knee (smallest N with ratio>=1.5): N=91; last N below: 88.0

### Variant C: d_hid 256, d_emb 32, nparam 44673

Free run (wd 1, seed 0): init norm 56.9; test 0.5 at step 2450 norm 93.41; test 0.99 at step 3200 norm 79.76; max norm 98.5

| N | eps | seed | T (step test>=0.99) | min grad_rms | logit_std @ min grad | step frac_sat>=0.99 | step grad_rms<1e-9 | final max_logit | final test |
|---|---|---|---|---|---|---|---|---|---|
| 58 | 1e-8 | 0 | 1450 | 8.28e-05 | 3.45 | - | - | 17.0 | 1.000 |
| 58 | 1e-16 | 0 | 1400 | 9.54e-05 | 3.31 | - | - | 16.26 | 1.000 |
| 66 | 1e-8 | 0 | 1800 | 9.36e-06 | 3.88 | - | - | 20.55 | 1.000 |
| 66 | 1e-16 | 0 | 1800 | 1.10e-05 | 3.74 | - | - | 20.23 | 1.000 |
| 74 | 1e-8 | 0 | 2000 | 8.86e-07 | 5.27 | - | - | 27.19 | 1.000 |
| 74 | 1e-16 | 0 | 2050 | 7.76e-07 | 5.21 | - | - | 26.02 | 1.000 |
| 82 | 1e-8 | 0 | 2500 | 1.51e-08 | 7.05 | 2450 | - | 34.0 | 0.999 |
| 82 | 1e-8 | 1 | 2250 | 1.57e-08 | 7.91 | 2200 | - | 38.87 | 0.999 |
| 82 | 1e-16 | 0 | 2550 | 2.69e-08 | 6.41 | 2550 | - | 32.73 | 1.000 |
| 82 | 1e-16 | 1 | 2250 | 4.05e-08 | 7.39 | 2300 | - | 37.72 | 1.000 |
| 90 | 1e-8 | 0 | 3050 | 3.79e-10 | 8.15 | 2400 | 3000 | 37.22 | 0.995 |
| 90 | 1e-8 | 1 | 5300 | 3.97e-10 | 9.56 | 2150 | 2800 | 64.21 | 0.991 |
| 90 | 1e-8 | 2 | 3200 | 4.67e-10 | 7.01 | 2750 | 3300 | 34.49 | 0.995 |
| 90 | 1e-16 | 0 | 2800 | 6.80e-10 | 7.81 | 2450 | 3000 | 38.29 | 0.999 |
| 90 | 1e-16 | 1 | 2650 | 7.21e-10 | 8.96 | 2200 | 2800 | 48.11 | 1.000 |
| 90 | 1e-16 | 2 | 3000 | 1.04e-09 | 6.8 | 2700 | - | 31.37 | 1.000 |
| 95 | 1e-8 | 0 | 9150 | 3.75e-10 | 8.58 | 2600 | 3250 | 169.01 | 0.993 |
| 95 | 1e-16 | 0 | 3350 | 6.47e-10 | 8.08 | 2600 | 3200 | 110.97 | 0.999 |
| 100 | 1e-8 | 0 | >12000 | 4.18e-10 | 9.18 | 2600 | 3350 | 339.34 | 0.980 |
| 100 | 1e-8 | 1 | >12000 | 4.69e-10 | 10.1 | 2000 | 3050 | 253.26 | 0.854 |
| 100 | 1e-16 | 0 | 3700 | 7.01e-10 | 8.69 | 2500 | 3200 | 210.17 | 1.000 |
| 100 | 1e-16 | 1 | 3400 | 7.04e-10 | 9.08 | 2000 | 2950 | 245.88 | 1.000 |
| 110 | 1e-8 | 0 | >12000 | 6.10e-10 | 11 | 1500 | 4050 | 268.68 | 0.182 |
| 110 | 1e-16 | 0 | 4300 | 7.14e-10 | 9.89 | 1500 | 3400 | 633.9 | 1.000 |

| N | mean T eps 1e-8 (n) | mean T eps 1e-16 (n) | ratio | mean min grad_rms eps 1e-8 | logit_std @ min grad, eps 1e-8 |
|---|---|---|---|---|---|
| 58 | 1450 (1) | 1400 (1) | 1.04 | 8.28e-05 | 3.45 |
| 66 | 1800 (1) | 1800 (1) | 1.00 | 9.36e-06 | 3.88 |
| 74 | 2000 (1) | 2050 (1) | 0.98 | 8.86e-07 | 5.27 |
| 82 | 2375 (2) | 2400 (2) | 0.99 | 1.54e-08 | 7.48 |
| 90 | 3850 (3) | 2817 (3) | 1.37 | 4.13e-10 | 8.24 |
| 95 | 9150 (1) | 3350 (1) | 2.73 | 3.75e-10 | 8.58 |
| 100 | >12000 (2) | 3550 (2) | >=3.38 | 4.43e-10 | 9.66 |
| 110 | >12000 (1) | 4300 (1) | >=2.79 | 6.10e-10 | 11 |

Knee (smallest N with ratio>=1.5): N=95; last N below: 90.0

### Variant D: d_hid 256, d_emb 128, nparam 103137

Free run (wd 1, seed 0): init norm 112.4; test 0.5 at step 1500 norm 89.59; test 0.99 at step 2650 norm 78.64; max norm 112.8

| N | eps | seed | T (step test>=0.99) | min grad_rms | logit_std @ min grad | step frac_sat>=0.99 | step grad_rms<1e-9 | final max_logit | final test |
|---|---|---|---|---|---|---|---|---|---|
| 66 | 1e-8 | 0 | 1350 | 1.41e-05 | 2.01 | - | - | 13.88 | 0.998 |
| 66 | 1e-16 | 0 | 1450 | 1.62e-05 | 2.07 | - | - | 13.68 | 0.996 |
| 76 | 1e-8 | 0 | 1500 | 1.37e-07 | 2.98 | - | - | 19.62 | 0.999 |
| 76 | 1e-16 | 0 | 1500 | 2.45e-07 | 2.81 | - | - | 19.04 | 1.000 |
| 84 | 1e-8 | 0 | 2450 | 3.44e-10 | 4.67 | 1200 | 1950 | 24.36 | 0.993 |
| 84 | 1e-8 | 1 | 2100 | 3.80e-10 | 4.85 | 1150 | 1850 | 24.52 | 0.994 |
| 84 | 1e-8 | 2 | 2250 | 3.39e-10 | 4.87 | 1150 | 1800 | 24.64 | 0.995 |
| 84 | 1e-16 | 0 | 1750 | 3.30e-09 | 3.75 | 1200 | - | 23.2 | 1.000 |
| 84 | 1e-16 | 1 | 1700 | 1.94e-09 | 4.04 | 1150 | - | 24.09 | 0.999 |
| 84 | 1e-16 | 2 | 1600 | 1.20e-09 | 4.18 | 1150 | - | 21.12 | 1.000 |
| 88 | 1e-8 | 0 | 7000 | 3.45e-10 | 5.21 | 1100 | 1850 | 79.74 | 0.992 |
| 88 | 1e-8 | 1 | 6150 | 3.42e-10 | 5.26 | 1100 | 1750 | 73.18 | 0.992 |
| 88 | 1e-16 | 0 | 1950 | 8.32e-10 | 4.45 | 1100 | 1800 | 33.81 | 1.000 |
| 88 | 1e-16 | 1 | 1900 | 6.33e-10 | 4.65 | 1100 | 1700 | 33.77 | 1.000 |
| 92 | 1e-8 | 0 | 9350 | 3.35e-10 | 5.68 | 1050 | 1700 | 160.36 | 0.993 |
| 92 | 1e-8 | 1 | 8750 | 3.44e-10 | 5.72 | 1050 | 1650 | 136.98 | 0.992 |
| 92 | 1e-16 | 0 | 2100 | 6.23e-10 | 5.13 | 1000 | 1550 | 82.81 | 0.999 |
| 92 | 1e-16 | 1 | 2100 | 5.51e-10 | 5.26 | 1050 | 1550 | 72.7 | 0.999 |
| 100 | 1e-8 | 0 | 11400 | 3.59e-10 | 6.86 | 950 | 1550 | 732.14 | 0.995 |
| 100 | 1e-16 | 0 | 2300 | 4.94e-10 | 6.26 | 950 | 1450 | 314.31 | 1.000 |
| 112 | 1e-16 | 0 | 2600 | 4.76e-10 | 7.64 | 900 | 1350 | 444.15 | 0.998 |
| 120 | 1e-16 | 0 | 2750 | 4.99e-10 | 8.03 | 850 | 1350 | 568.27 | 0.997 |

| N | mean T eps 1e-8 (n) | mean T eps 1e-16 (n) | ratio | mean min grad_rms eps 1e-8 | logit_std @ min grad, eps 1e-8 |
|---|---|---|---|---|---|
| 66 | 1350 (1) | 1450 (1) | 0.93 | 1.41e-05 | 2.01 |
| 76 | 1500 (1) | 1500 (1) | 1.00 | 1.37e-07 | 2.98 |
| 84 | 2267 (3) | 1683 (3) | 1.35 | 3.54e-10 | 4.8 |
| 88 | 6575 (2) | 1925 (2) | 3.42 | 3.43e-10 | 5.24 |
| 92 | 9050 (2) | 2100 (2) | 4.31 | 3.39e-10 | 5.7 |
| 100 | 11400 (1) | 2300 (1) | 4.96 | 3.60e-10 | 6.86 |
| 112 | - (0) | 2600 (1) | - | - | - |
| 120 | - (0) | 2750 (1) | - | - | - |

Knee (smallest N with ratio>=1.5): N=88; last N below: 84.0

## Summary

| variant | d_hid | d_emb | nparam | init norm | W_c test 0.5 / 0.99 | knee N | last N below | knee/W_c(0.5) | knee/W_c(0.99) | knee/init | min grad_rms below / at knee (eps 1e-8) | logit_std below / at knee |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ref | 256 | 64 | 64161 | 79.6 | 88.38 / 78.71 | 88 | 85 | 1.00 | 1.12 | 1.10 | 3.96e-10 / 4.07e-10 | 5.78 / 6.03 |
| A | 128 | 64 | 35233 | 79.4 | 80.1 / 76.24 | 82 | 79 | 1.02 | 1.08 | 1.03 | 9.77e-10 / 4.80e-10 | 6.74 / 7.11 |
| B | 512 | 64 | 122017 | 80.2 | 89.08 / 80.86 | 91 | 88 | 1.02 | 1.13 | 1.13 | 7.20e-10 / 3.53e-10 | 4.89 / 5.27 |
| C | 256 | 32 | 44673 | 56.9 | 93.41 / 79.76 | 95 | 90 | 1.02 | 1.19 | 1.67 | 4.13e-10 / 3.75e-10 | 8.24 / 8.58 |
| D | 256 | 128 | 103137 | 112.4 | 89.59 / 78.64 | 88 | 84 | 0.98 | 1.12 | 0.78 | 3.54e-10 / 3.43e-10 | 4.8 / 5.24 |