O0-RP-008

Reproducibility and Code Standards

**Version:** 1.0

**Research status:** APPROVED

CLAIM STATUS: METHODOLOGICAL
EVIDENCE TYPE: GOVERNANCE STANDARD
PHYSICAL VALIDATION: NONE
INDEPENDENT REPLICATION: N/A
PHILOSOPHICAL PROVENANCE: O/0 ARCHIVE
ARCHIVE ENDORSEMENT: LIMITED TO REPORTED RESULT

Abstract

This document specifies the technical requirements for all code, data, random seeds, dependencies, computational environments, and archival formats used within the O/0 Research Program. Every simulation, analysis, and visualization produced by the program must be independently reproducible by any competent researcher given only the archived materials. The standards defined here are non-negotiable requirements for publication within the program. They exist because computational research that cannot be reproduced is not science — it is anecdote dressed in the language of mathematics.

Source proposition

The O/0 Research Program charter (RP-001) mandates that "any submitted simulation must be accompanied by reproducible code and raw statistical outputs." This document specifies exactly what reproducibility requires in technical terms.

Scientific audit

The reproducibility crisis in computational science is well-documented. Studies have shown that a majority of published computational results cannot be reproduced even by the original authors after a short period. Common failure modes include: undocumented dependencies, floating-point non-determinism, missing random seeds, hardware-specific behavior, and code rot. This standards document addresses each failure mode with specific countermeasures.

Research question

What is the minimum set of technical requirements that guarantees bit-for-bit or statistically-equivalent reproduction of all computational results in the program?

Operational definitions

1. **Bit-for-bit reproducibility**: Running the archived code with the archived parameters on a compatible system produces numerically identical output.

2. **Statistical reproducibility**: Running the archived code produces results within the pre-specified confidence intervals of the published results (applicable when hardware non-determinism prevents bit-for-bit reproduction).

3. **Computational artifact**: Any file produced by program research, including source code, data files, configuration files, environment specifications, and documentation.

4. **Archival format**: A file format chosen for long-term accessibility, readability, and independence from proprietary tools.

5. **Execution environment**: The complete specification of software and hardware required to run a simulation, including operating system, language runtime, library versions, and hardware constraints.

6. **Random seed**: The initial state of a pseudo-random number generator that determines the entire sequence of random numbers used in a simulation.

Hypothesis

Not applicable. This is a technical governance document.

Null hypothesis

Not applicable.

Competing explanations

One might argue that reproducibility requirements slow research and that "conceptual reproducibility" (independent implementation of the same idea) is sufficient. The program rejects this argument: in computational science, subtle implementation differences can produce qualitatively different results, and conceptual reproduction cannot detect implementation-specific artifacts.

Formal model

A research record R is reproducible if and only if there exists an archived package P(R) = {code, data, environment, seeds, parameters, instructions} such that for any competent researcher with access to P(R), executing the specified instructions produces outputs O' satisfying either:

• O' = O (bit-for-bit), or
• |O' - O| < ε for all measured quantities, where ε is the pre-specified tolerance documented in R.

Methods

Section A: Code Standards

**A.1 Language and Framework Requirements**

• Primary language: Python 3.10+ (for simulation and analysis).
• Acceptable secondary languages: Julia (for performance-critical simulations), R (for specialized statistical analyses).
• All code must be executable without proprietary software or paid licenses.
• No Jupyter notebooks as primary execution artifacts (notebooks may supplement but not replace standalone scripts).

**A.2 Code Structure Requirements**

• Every simulation must have a single entry point (e.g., `run_simulation.py`) that accepts all parameters via command-line arguments or a configuration file.
• Configuration must be separated from code. All parameters must be specified in a dedicated configuration file (YAML or JSON format).
• No hardcoded paths, magic numbers, or implicit defaults. Every parameter must be explicitly specified and documented.
• Code must include type hints for all function signatures.
• Functions must not exceed 50 lines. Files must not exceed 500 lines. Violations require documented justification.

**A.3 Documentation Requirements**

• Every public function must have a docstring specifying: purpose, parameters (with types and valid ranges), return values, and side effects.
• Every module must have a module-level docstring explaining its role in the simulation.
• A README.md must accompany every simulation package, containing: (1) what the simulation does, (2) how to install dependencies, (3) how to run it, (4) how to verify correct execution, and (5) expected runtime.

**A.4 Testing Requirements**

• Every simulation must include unit tests for core computational functions.
• A deterministic test case (fixed seed, small scale) must produce known outputs that are checked by the test suite.
• Tests must be runnable via a single command (e.g., `pytest`).
• Test coverage must exceed 60% for core modules.

Section B: Dependency Management

**B.1 Environment Specification**

• Every simulation package must include a complete environment specification sufficient for exact reproduction:
• `requirements.txt` with pinned versions (e.g., `numpy==1.24.3`, not `numpy>=1.24`).
• A `Dockerfile` or equivalent container specification for the complete execution environment.
• An `environment.yml` (conda) as an alternative to Docker for researchers who prefer conda environments.

**B.2 Dependency Constraints**

• Dependencies must be limited to well-maintained, open-source packages with stable APIs.
• GPU-dependent code must also include a CPU fallback that produces identical results (potentially slower).
• System-level dependencies (compilers, system libraries) must be documented in the Dockerfile.

**B.3 Version Pinning Protocol**

• All dependency versions are pinned at the time of publication and archived.
• A `dependency_audit.txt` file records the date each dependency was last verified to be compatible.
• If a dependency becomes unavailable, the archived Docker image serves as the canonical execution environment.

Section C: Random Seeds and Determinism

**C.1 Seed Management**

• Every simulation run must use an explicitly specified random seed.
• The seed must be recorded in the output metadata of every run.
• The seed generation process must be documented (e.g., "seeds drawn from range [0, 2³²) using Python's random.randint with meta-seed 42").
• Multiple runs use sequential or pre-generated seed lists, documented in the configuration.

**C.2 Determinism Verification**

• Before publication, each simulation must be run twice with identical seeds to verify deterministic output.
• If GPU non-determinism prevents bit-for-bit reproduction, this must be documented, and the tolerance ε must be reported with empirical justification.
• The determinism verification results must be included in the archived test suite.

**C.3 Seed Disclosure**

• All seeds used in published results must be disclosed in the research record.
• Selective seed reporting (reporting only seeds that produced favorable results) is prohibited and constitutes research misconduct under this program.

Section D: Data Standards

**D.1 Raw Data Archival**

• All raw simulation outputs must be archived in their complete form.
• Archival format for numerical data: HDF5 (primary) or NumPy `.npy`/`.npz` (acceptable for small datasets).
• Archival format for metadata: JSON (human-readable, tool-independent).
• No proprietary formats (no .mat, .sas7bdat, .xlsx for primary data storage).

**D.2 Data Organization**

• Each simulation run produces a self-contained output directory with the structure:

```

run_YYYYMMDD_HHMMSS_seed{SEED}/

├── config.yaml (exact parameters used)

├── raw_output.h5 (complete simulation state history)

├── summary_statistics.json (pre-computed summary statistics)

├── metadata.json (runtime, hardware, versions, seed)

└── checksums.sha256 (integrity verification)

```

**D.3 Data Integrity**

• SHA-256 checksums are computed for all archived data files at the time of archival.
• Checksums are verified before any analysis is performed on archived data.
• Any checksum mismatch invalidates the data and requires re-execution from source.

Section E: Version Control

**E.1 Repository Standards**

• All code is maintained in Git repositories.
• Every published simulation result is tagged with a Git tag matching the research record ID (e.g., `O0-SIM-001-v1.0`).
• The tagged commit must produce the published results when executed with the archived configuration.

**E.2 Commit Practices**

• Commits must be atomic (one logical change per commit).
• Commit messages must follow conventional format: `type(scope): description` (e.g., `feat(sim-001): add boundary detection algorithm`).
• No large binary files in Git. Data is stored in a separate archival system and referenced by checksum.

**E.3 Branching Model**

• `main` branch contains only publication-ready, tested code.
• Development occurs on feature branches.
• All merges to `main` require passing the full test suite and code review.

Section F: Archival Protocol

**F.1 Long-Term Accessibility**

• All published simulation packages are archived in at least two independent locations.
• Archives must be self-contained: an archived package must be executable without network access (all dependencies included or containerized).
• Archive format: A single compressed archive (`.tar.gz`) containing code, configuration, environment specification, and a subset of representative data sufficient to verify correct execution.

**F.2 Format Longevity**

• Preferred formats are chosen for maximum expected longevity: plain text (configuration), HDF5 (data), PNG/SVG (figures), Markdown (documentation).
• Formats requiring specific proprietary software for reading are prohibited.
• A format migration plan is reviewed annually to identify at-risk formats.

Controls

Reproducibility standards are verified by annual reproduction exercises in which an independent team member attempts to reproduce a randomly selected published result using only the archived materials.

Predictions

Strict adherence to these standards will ensure that all program results remain reproducible for a minimum of 10 years after publication, assuming standard hardware evolution. The initial overhead for compliance is estimated at 15-25% of total development time, decreasing to 5-10% as practices become habitual.

Falsification criteria

The standards are inadequate if: (a) a reproduction attempt fails despite correct execution of the archived materials, (b) the standards impose such overhead that research productivity drops below acceptable levels (fewer than 2 publications per year), or (c) the archival format becomes inaccessible within the 10-year target window.

Results / Expected Outcomes

Complete technical standards have been specified across six domains: code standards, dependency management, random seeds, data standards, version control, and archival protocol. These standards are immediately enforceable for all new simulation work.

Uncertainty

Hardware evolution may break Docker images over long timescales. Floating-point behavior may differ between CPU architectures in ways that are difficult to predict. The 10-year reproducibility target is aspirational and depends on the continued availability of x86-64 compatible execution environments.

Limitations

These standards do not address the reproducibility of theoretical (pen-and-paper) results, which rely on logical verification rather than computational reproduction. They also do not address the question of whether a reproducing a result on different hardware constitutes "the same" result when floating-point differences accumulate.

Replication status

Governance document. Standards have been retroactively applied to the existing SIM series. O0-SIM-001 through O0-SIM-003 were found to be fully compliant. O0-SIM-004 and O0-SIM-005 required minor archival corrections.

Data and code

A compliance checklist template and automated validation script are maintained in the program repository. The validation script checks for the presence of all required files, correct formatting of configuration, and verifiable checksums.

Relationship to philosophical archive

The O/0 archive's emphasis on transparency and honest inquiry motivates these technical standards, but their authority derives from established best practices in computational reproducibility, not from philosophical principles. Conceptual provenance from the archive does not constitute empirical support for any claim.

References

• Sandve, G.K. et al. (2013). Ten simple rules for reproducible computational research. PLoS Computational Biology.
• Stodden, V. et al. (2016). Enhancing reproducibility for computational methods. Science.
• Wilson, G. et al. (2017). Good enough practices in scientific computing. PLoS Computational Biology.
• Piccolo, S.R. & Frampton, M.B. (2016). Tools and techniques for computational reproducibility.
• National Academies of Sciences (2019). Reproducibility and Replicability in Science.
• Ivie, P. & Thain, D. (2018). Reproducibility in scientific computing. ACM Computing Surveys.

Revision history

• v1.0: Initial document generated.