O0-RP-007

Adversarial Standards Protocol

**Version:** 1.0

**Research status:** APPROVED

CLAIM STATUS: METHODOLOGICAL
EVIDENCE TYPE: GOVERNANCE STANDARD
PHYSICAL VALIDATION: NONE
INDEPENDENT REPLICATION: N/A
PHILOSOPHICAL PROVENANCE: O/0 ARCHIVE
ARCHIVE ENDORSEMENT: LIMITED TO REPORTED RESULT

Abstract

This document establishes the Adversarial Standards Protocol for the O/0 Research Program. Every study published under the program must undergo structured adversarial analysis designed to actively seek reasons why its conclusions might be wrong. The protocol mandates that researchers themselves generate devil's advocate positions, that independent adversarial reviewers attempt to replicate results using alternative explanations, and that all published records include a dedicated section documenting adversarial challenges and how they were addressed. The protocol is motivated by the program's heightened vulnerability to confirmation bias given its philosophical provenance: researchers investigating claims they find intuitively compelling must be held to a higher adversarial standard than typical science.

Source proposition

The O/0 archive emphasizes that "the map is not the territory" and that all models are approximations. The adversarial protocol operationalizes this principle by requiring that every model be actively challenged from within the research program itself, before external review.

Scientific audit

Adversarial methodology has precedent in red-teaming (military/cybersecurity), pre-mortem analysis (project management), and registered reports with adversarial reviewers (psychology). The protocol adapts these practices for computational philosophy-of-science research, where the primary risk is not fraud but sincere self-deception driven by prior commitment to a philosophical framework.

Research question

What structured adversarial process is minimally sufficient to prevent confirmation bias, scope creep, and inferential overreach in a research program with strong philosophical priors?

Operational definitions

1. **Adversarial review**: A structured process in which a designated reviewer actively attempts to invalidate, weaken, or provide alternative explanations for a study's conclusions.

2. **Devil's advocate position**: A formally stated argument against the study's conclusions, constructed in good faith to identify the strongest possible objection.

3. **Red team**: A designated group or individual whose role is to find flaws, not to confirm results.

4. **Pre-mortem**: An exercise conducted before publication in which the team imagines the study has been decisively refuted, then works backward to identify how that refutation occurred.

5. **Adversarial control**: A simulation run specifically designed to produce the target result through a mechanism other than the one hypothesized.

6. **Steel-man alternative**: The strongest possible version of a competing explanation, constructed to be maximally threatening to the preferred hypothesis.

Hypothesis

Not applicable. This is a governance protocol.

Null hypothesis

Not applicable.

Competing explanations

One might argue that standard peer review is sufficient adversarial scrutiny. However, peer review occurs post-hoc, is often superficial, and reviewers may share the same priors as the authors in a specialized program. The adversarial protocol is an internal, pre-publication safeguard that does not replace but supplements external review.

Formal model

The Adversarial Review Template

Every study submitted for publication must include a completed Adversarial Analysis section following this template:

---

**ADVERSARIAL ANALYSIS RECORD**

**Study ID:** [O0-SIM-XXX or O0-MATH-XXX]

**Adversarial Reviewer:** [Name/ID — must not be the primary author]

**Date of Review:** [Date]

**Section 1: Pre-Mortem Exercise**

*Imagine this study has been decisively refuted two years after publication. Describe the three most likely scenarios for how that refutation occurred.*

1. [Scenario 1: e.g., "The effect disappeared when the grid size was increased beyond 500×500, revealing it as a finite-size artifact."]

2. [Scenario 2: e.g., "An independent team showed the null model produces identical results when initialized with correlated noise rather than white noise."]

3. [Scenario 3: e.g., "The statistical test was shown to be inappropriate for the data's autocorrelation structure."]

**Section 2: Strongest Alternative Explanations**

*For each major conclusion, provide the strongest alternative explanation that does not invoke the proposed mechanism.*

| Conclusion | Alternative Explanation | How to Discriminate |

|-----------|------------------------|-------------------|

| [Conclusion 1] | [Alt explanation] | [Discriminating test] |

| [Conclusion 2] | [Alt explanation] | [Discriminating test] |

**Section 3: Assumption Vulnerability Analysis**

*List every assumption the study relies upon, ranked from most to least vulnerable.*

| Rank | Assumption | Vulnerability | If Wrong, Consequence |

|------|-----------|---------------|----------------------|

| 1 | [Most vulnerable assumption] | [Why it might be wrong] | [What happens to conclusions] |

| 2 | [Second most vulnerable] | [Why] | [Consequence] |

**Section 4: Adversarial Simulation Runs**

*Describe at least one simulation run designed to produce the target result via an alternative mechanism.*

• Alternative mechanism tested: [Description]
• Result: [Did the alternative mechanism produce similar results?]
• Implications: [What this means for the study's conclusions]

**Section 5: Prohibited Claims Check**

*Verify the study against RP-004. List any items that required revision.*

• [Item checked]: [PASS/REVISED — describe revision if applicable]

**Section 6: Scope Assessment**

*Does the study claim more than its evidence supports? Identify any scope creep.*

• Claimed scope: [What the study says it shows]
• Supported scope: [What the evidence actually supports]
• Gap: [Any over-claiming identified]

**Section 7: Final Adversarial Verdict**

• [ ] PASS: Adversarial challenges adequately addressed.
• [ ] CONDITIONAL PASS: Minor revisions required (listed below).
• [ ] FAIL: Fatal vulnerability identified (described below).

---

Methods

Mandatory Adversarial Controls in Simulation Studies

Every simulation study (O0-SIM series) must include at least the following adversarial controls:

1. **The Null Mechanism Control:** Run the null model under identical conditions. Report the full distribution of null-model outcomes, not just the mean.

2. **The Alternative Mechanism Control:** Identify at least one alternative mechanism that could produce the observed phenomenon and simulate it. Report whether the target model's results are distinguishable from the alternative.

3. **The Parameter Sensitivity Control:** Vary all free parameters across their plausible ranges. Report whether the result is robust or parameter-dependent.

4. **The Finite-Size Control:** Run the simulation at multiple scales. Report whether the result persists, strengthens, or disappears with increasing system size.

5. **The Initialization Control:** Run the simulation with multiple different initial conditions (random, structured, adversarial). Report initialization dependence.

6. **The Temporal Control:** Report whether the observed phenomenon is transient or persistent. If transient, report its lifetime distribution.

Required Devil's Advocate Positions

For every study claiming to demonstrate a property relevant to the O/0 framework (emergence, boundary formation, self-organization, observer-like behavior), the following devil's advocate positions must be explicitly addressed:

1. **"It's just a finite-size artifact."** — Demonstrate that the result survives scaling.

2. **"It's encoded in the initial conditions."** — Demonstrate that multiple initializations produce the result.

3. **"It's a trivial consequence of the update rule."** — Show that the update rule alone does not guarantee the result (via null model comparison).

4. **"Any sufficiently complex system would do this."** — Test with structurally different systems to assess generality vs. specificity.

5. **"The measurement creates the phenomenon."** — Vary the measurement protocol and show the result is measurement-independent.

6. **"It's confirmation bias in the analysis."** — Have an independent analyst re-analyze the raw data blind to the hypothesis.

Controls

The adversarial protocol is itself subject to meta-review: annually, the program conducts a red-team exercise in which a deliberately flawed study is submitted through the pipeline to test whether the adversarial review catches it.

Predictions

Implementation of this protocol will: (a) increase the time-to-publication by approximately 30-50%, (b) reduce the rate of post-publication corrections to near zero, (c) produce a documented record of adversarial reasoning that strengthens reader confidence in published results, and (d) identify approximately 1-2 fatal vulnerabilities per 10 studies submitted.

Falsification criteria

The protocol is inadequate if: (a) a published study is subsequently shown to have a fatal flaw that the adversarial review should have caught, (b) the protocol consistently fails to identify any vulnerabilities (suggesting rubber-stamping), or (c) independent researchers identify systematic biases in published results that the protocol was designed to prevent.

Results / Expected Outcomes

The Adversarial Standards Protocol has been fully specified, including a complete review template, six mandatory adversarial controls for simulation studies, and six required devil's advocate positions. The protocol is immediately enforceable for all new submissions.

Uncertainty

The appropriate balance between rigor and productivity is not known a priori. Excessively stringent adversarial review could paralyze the program; insufficiently stringent review could allow confirmation bias to persist. Calibration will occur over the first several review cycles.

Limitations

The protocol addresses honest intellectual errors (confirmation bias, scope creep, inferential overreach). It is not designed to detect deliberate fraud, data fabrication, or bad-faith manipulation. Additionally, adversarial review quality depends on the competence and independence of the reviewer, which cannot be fully formalized.

Replication status

Governance document. Retrospective application to the existing SIM series identified two cases where additional adversarial controls would have strengthened the published conclusions.

Data and code

The adversarial review template is available as a markdown template in the program repository for inclusion in all new research records.

Relationship to philosophical archive

The adversarial protocol embodies the archive's own principle that certainty is an epistemic trap. By institutionalizing doubt, the program operationalizes the contemplative insight that all models are incomplete. However, this structural alignment is noted as motivation only — the protocol's authority derives from its methodological soundness, not from its philosophical provenance. Conceptual provenance from the archive does not constitute empirical support for any claim.

References

• Lakatos, I. (1970). Falsification and the methodology of scientific research programmes.
• Klein, G. (2007). Performing a project premortem. Harvard Business Review.
• Chambers, C.D. (2013). Registered Reports: A new publishing initiative at Cortex.
• Mellers, B. et al. (2014). Psychological strategies for winning a geopolitical forecasting tournament.
• Tetlock, P. & Gardner, D. (2015). Superforecasting.
• Feynman, R. (1974). Cargo cult science. Caltech commencement address.

Revision history

• v1.0: Initial document generated.