SIMULATION · O0-CRP-025

Track 2 R1 · Discriminability Under Alternative Classifiers — δ*(0.7) is classifier-dependent (linear classifiers halve the RF value)

STATUSVerdict: CLASSIFIER_DEPENDENT at max/min δ*(0.7) ratio = 3.0. Substantive reading: linear classifiers (logistic, linear SVM) detect DUAL departures at δ*(0.7) = 0.10, half the RandomForest value of 0.20. The AUC-below-0.5 phenomenon documented in CRP-022 is largely tree-classifier-specific; logistic gives clean chance-level matched AUC = 0.505.
EVIDENCE TYPECOMPUTATIONAL · reanalysis of O0-CRP-022 raw features through five classifiers (RandomForest, logistic regression, linear SVM, MLP, gradient boosting).
REPLICATIONREGISTERED · joint 2D architecture × classifier grid, MLP without early stopping, permutation test for matched-null distinguishability.
PHYSICAL VALIDATIONNONE
VERSION1.0.0
DATE

O0-CRP-025 — Scientific Record

**Title:** Discriminability Under Alternative Classifiers

**Program:** Track 2 followup R1

**Version:** 1.0.0

**Status:** COMPLETED · Executed 2026-07-28

**Preregistration:** v1.0.0 frozen before execution

Claim status

CLAIM STATUS: **CLAIS_CLASSIFIER_DEPENDENT · Track 2 result is partially classifier-specific**

EVIDENCE TYPE: COMPUTATIONAL, reanalysis of O0-CRP-022/024 features

PHYSICAL VALIDATION: NONE

Ratio max/min δ*(0.7) = 3.0 across five classifiers with positive-control gate passed. Preregistered threshold for CLASSIFIER_DEPENDENT is ratio > 2.5; the result exceeds it. Verdict: CLASSIFIER_DEPENDENT.

Key finding

Linear classifiers (LogReg, LinearSVM) detect DUAL universe departures at **half the δ** that RandomForest requires. MLP is worst. This partially corrects the discriminability interpretation from CRP-022/024 — δ* is not purely a substrate property, it depends on the classifier.

| Classifier | δ*(0.7) | δ*(0.6) | matched-dual raw AUC | null-null raw AUC |

|---|---|---|---|---|

| logistic | **0.10** | 0.10 | 0.5054 | 0.4481 |

| linear_svm | **0.10** | 0.05 | 0.4650 | 0.4280 |

| random_forest | 0.20 | 0.10 | 0.3756 | 0.3952 |

| grad_boost | 0.20 | 0.10 | 0.4694 | 0.4311 |

| mlp | 0.30 | 0.20 | 0.4643 | 0.4606 |

**Ratio max/min δ*(0.7) = 3.0** → CLASSIFIER_DEPENDENT verdict.

Substantive reading

1. **The AUC-below-0.5 phenomenon in CRP-022/024 is largely tree-classifier-specific.** RandomForest matched AUC = 0.376; logistic regression matched AUC = 0.505 (right at chance). The systematic anticorrelation on matched trials is a property of tree-based classifiers on this problem, not evidence about the universes.

2. **Linear classifiers outperform ensembles on this linear-Gaussian problem.** Because the universe dynamics are linear-Gaussian, a linear discriminant is well-matched. Trees invest capacity in modeling nonlinear interactions that aren't there.

3. **The CRP-022/024 δ*(0.7) = 0.20 finding was pessimistic.** A better-matched classifier (logistic) reveals discriminability at δ = 0.10.

4. **δ* depends on both universe class AND classifier.** The CRP-024 "capacity-limited invariance" finding (5 architectures converge at 0.20 with RF) is really "5 architectures under RF converge at 0.20." Whether that invariance would hold with logistic regression as the classifier is a legitimate open question, addressable by combining CRP-024's architecture sweep with CRP-025's classifier sweep in a joint 2D grid.

Figures

![delta_sweep](figures/01_delta_sweep_by_classifier.png)

![matched_null](figures/02_matched_null_by_classifier.png)

![delta_star](figures/03_delta_star_by_classifier.png)

Adversarial interpretation

  • **Could MLP result be undertraining?** Possibly — MLP with early stopping may have exited before convergence. Followup: MLP without early stopping.
  • **Could linear-classifier advantage be a fluke on this data?** Repeat on fresh worlds (O0-CRP-027). If it survives, the finding is robust.
  • **Could logistic AUC = 0.505 on matched-dual be evidence of true indistinguishability?** More plausible than the RF AUC = 0.376 was. Followup: null-null vs matched-dual permutation test at n=300 per class.

Not established

  • That linear classifiers detect a "true" boundary. They may overfit differently at n = 300 on 146 features.
  • That the finding generalizes to nonlinear universes (registered R2 = CRP-030).
  • That any classifier can push δ* below 0.10 at this sample size.

Retrospective correction to O0-CRP-022 / CRP-024

The AUC = 0.376 vs 0.396 comparison in CRP-022 was reported as "matched-dual statistically indistinguishable from null-null." That reading remains correct *for RandomForest*. Under logistic regression, matched-dual AUC = 0.505 and null-null AUC = 0.448 — matched is closer to chance than null-null, which is different information. Neither classifier says matched-dual is *distinguishable*; both say something like "not distinguishable at this sample size." But the specific AUC values differ substantially.

Replication procedure


cd research/studies/O0-CRP-025/src
python run_study.py    # ~120s
python analyze.py       # regenerates figures

Requires O0-CRP-022 raw features and O0-CRP-024 null-null features.

Registered followups

  • R1: 2D grid — architecture × classifier — to see whether CRP-024's capacity-limited invariance holds across all classifiers.
  • R2: MLP without early stopping and with more hidden layers.
  • R3: Permutation test for matched-dual vs null-null distinguishability at each classifier.

Revision history

  • v1.0.0 (2026-07-28): initial record after preregistered execution.

Figures

Figure from O0-CRP-025: 01 delta sweep by classifier
Figure from O0-CRP-025: 01 delta sweep by classifier
Figure from O0-CRP-025: 02 matched null by classifier
Figure from O0-CRP-025: 02 matched null by classifier
Figure from O0-CRP-025: 03 delta star by classifier
Figure from O0-CRP-025: 03 delta star by classifier

Source proposition

“Followup R1 registered in O0-CRP-024 manifest. Tests whether the O0-CRP-024 finding of capacity-limited invariance holds across different classifiers or is RandomForest-specific.”

Conceptual provenance is not empirical support.

Related records

O0-CRP-022O0-CRP-024O0-CRP-026O0-CRP-027