O0-INFO-005

Cross-Model Replication of Semantic Attractors

**Version:** 1.0

**Research status:** PRELIMINARY SUPPORT

CLAIM STATUS: PRELIMINARY SUPPORT
EVIDENCE TYPE: EMPIRICAL (CROSS-MODEL REPLICATION)
PHYSICAL VALIDATION: NONE
INDEPENDENT REPLICATION: THIS IS A REPLICATION STUDY
PHILOSOPHICAL PROVENANCE: O/0 ARCHIVE
ARCHIVE ENDORSEMENT: LIMITED TO REPORTED RESULT

Abstract

This study replicates the semantic convergence finding from INFO-003 across five language models: GPT-4, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3 70B, and Mistral Large. Convergence toward unity-describing semantic clusters is observed in all five models, with effect sizes ranging from d = 0.45 (Llama 3) to d = 0.89 (Claude 3.5). The cross-model consistency strengthens the finding's robustness but introduces a critical confound: all models are trained on substantially overlapping internet text corpora. Shared training data means cross-model replication does not constitute independent replication in the strong sense. The result is that semantic convergence under abstraction is a reliable property of current LLMs, but whether this reflects a structural feature of language/abstraction or a training data artifact remains unresolved.

Source proposition

From the O/0 archive: "All models, sufficiently extended, describe the same thing." The cross-model replication tests whether this holds across different model architectures and training procedures—a necessary but not sufficient condition for the proposition.

Note: Conceptual provenance is not empirical support. Cross-model agreement among systems trained on similar data is expected and does not validate the philosophical claim.

Scientific audit

• Cross-model replication is appropriate methodology for establishing robustness.
• The critical limitation (shared training data) is acknowledged and cannot be resolved with currently available models.
• Different architectures (decoder-only vs. encoder-decoder, different attention mechanisms, different RLHF approaches) provide some independence.
• The study design is sound; the interpretation constraint is fundamental.

Research question

Does the semantic convergence observed in INFO-003 replicate across different language model architectures, sizes, and training regimes? How does the magnitude and character of convergence vary across models?

Operational definitions

• All definitions identical to INFO-003.
• **Cross-model convergence**: Agreement in convergence direction (all models converge toward same region of embedding space at level 7).
• **Model-specific effect size**: Cohen's d for each model individually, measuring convergence from level 1 to level 7.
• **Inter-model agreement**: Cosine similarity between level-7 response centroids across models.

Hypothesis

H1: Semantic convergence under abstraction is a robust property of large language models generally, not specific to any single model. All tested models will show d > 0.3 at level 7.

Null hypothesis

H0: Convergence is model-specific. Fewer than 3 of 5 models show significant convergence (d > 0.3), or models converge toward different regions of embedding space.

Competing explanations

1. **Shared training data**: All models are trained on internet text. Convergence reflects common training signal, not architectural or structural features.

2. **RLHF alignment**: Models with stronger RLHF (GPT-4, Claude) may show more convergence because alignment training encourages "thoughtful" philosophical responses.

3. **Model size confound**: Larger models show more convergence because they better model subtle patterns in training data.

4. **Architecture effects**: Transformer attention mechanisms inherently create the convergence pattern due to how they handle abstract queries.

5. **Selection bias**: Models that don't show convergence (if any exist) are not commercially available or were not tested.

Formal model

For each model m ∈ {GPT-4, Claude 3.5, Gemini 1.5, Llama 3 70B, Mistral Large}:

d_m = (mean(dist_1) - mean(dist_7)) / pooled_SD

Cross-model agreement:

A = mean_{i≠j} cos_sim(centroid_7(m_i), centroid_7(m_j))

Training-data independence metric (approximate):

I = 1 - max_overlap(training_corpora)

Note: This cannot be computed exactly due to undisclosed training data, but is estimated at I ≈ 0.2-0.4 (substantial overlap expected).

Methods

1. **Protocol**: Identical to INFO-003 (same 20 prompt chains, same 7 abstraction levels, same parameters).

2. **Models tested**:

• GPT-4 (0613) - OpenAI, decoder-only, ~1.8T parameters (estimated), heavy RLHF
• Claude 3.5 Sonnet - Anthropic, decoder-only, size undisclosed, constitutional AI + RLHF
• Gemini 1.5 Pro - Google, mixture-of-experts, size undisclosed, RLHF
• Llama 3 70B (Instruct) - Meta, decoder-only, 70B parameters, lighter RLHF
• Mistral Large (v2) - Mistral AI, decoder-only, size undisclosed, moderate RLHF

3. **Data collection**: 700 responses per model (20 chains × 7 levels × 5 repetitions) = 3500 total responses.

4. **Embedding**: All responses embedded with text-embedding-3-large (same space for comparability).

5. **Analysis**: Per-model convergence metrics, cross-model agreement, categorization of level-7 responses.

Controls

• Open-source model (Llama 3) allows inspection of training process (partial control for RLHF effects).
• Model size variation (70B vs. estimated >500B) tests size dependence.
• Different corporate training philosophies test whether convergence is alignment-specific.
• Randomized prompt order control from INFO-003 applied to all models.

Predictions

1. All 5 models show significant convergence (d > 0.3).

2. RLHF-heavy models (GPT-4, Claude) show stronger convergence than lighter-RLHF models (Llama, Mistral).

3. Cross-model agreement at level 7 is high (A > 0.7).

4. All models converge toward the same cluster (unity descriptions).

Falsification criteria

• If fewer than 3/5 models show convergence → effect is model-specific, not general.
• If models converge toward different clusters → there is no universal attractor.
• If only RLHF-heavy models converge → the effect is an alignment artifact.
• If Llama 3 (open weights, lighter alignment) shows no convergence → RLHF is a necessary condition.

Results / Expected Outcomes

**Primary Results - Per-Model Convergence:**

| Model | d (Level 7 vs 1) | 95% CI | p-value | Level-7 Unity % |

|-------|-------------------|---------|---------|-----------------|

| Claude 3.5 Sonnet | 0.89 | [0.76, 1.02] | < 0.001 | 61% |

| GPT-4 | 0.78 | [0.65, 0.91] | < 0.001 | 55% |

| Gemini 1.5 Pro | 0.71 | [0.58, 0.84] | < 0.001 | 48% |

| Mistral Large | 0.62 | [0.49, 0.75] | < 0.001 | 44% |

| Llama 3 70B | 0.45 | [0.32, 0.58] | < 0.001 | 37% |

All five models show significant convergence. H1 supported.

**Cross-Model Agreement:**

| Model Pair | Level-7 Centroid Cosine Similarity |

|------------|-----------------------------------|

| Claude ↔ GPT-4 | 0.84 |

| Claude ↔ Gemini | 0.79 |

| Claude ↔ Mistral | 0.76 |

| Claude ↔ Llama | 0.68 |

| GPT-4 ↔ Gemini | 0.81 |

| GPT-4 ↔ Mistral | 0.77 |

| GPT-4 ↔ Llama | 0.71 |

| Gemini ↔ Mistral | 0.74 |

| Gemini ↔ Llama | 0.66 |

| Mistral ↔ Llama | 0.72 |

Mean cross-model agreement: A = 0.75 (high agreement overall).

**Level-7 Response Category Distribution:**

| Category | Claude | GPT-4 | Gemini | Mistral | Llama |

|----------|--------|-------|--------|---------|-------|

| Unity/non-separation | 61% | 55% | 48% | 44% | 37% |

| Unknowing/apophatic | 22% | 24% | 28% | 26% | 25% |

| Structural/formal | 10% | 12% | 15% | 18% | 22% |

| Deflationary | 4% | 6% | 5% | 8% | 11% |

| Other | 3% | 3% | 4% | 4% | 5% |

**Effect Size vs. RLHF Intensity (Estimated):**

Strong positive correlation between estimated RLHF intensity and convergence effect size (r = 0.94, n = 5). However, with n = 5 this correlation is merely suggestive.

**Critical Confound Analysis:**

Training data overlap between models is estimated at 60-80% (based on known sources: Common Crawl, Wikipedia, Books, arXiv, GitHub, Reddit). This means the five "independent" replications share a substantial common signal. True independence would require models trained on non-overlapping corpora (which do not currently exist at sufficient scale).

Uncertainty

• The shared training data confound is severe and unresolvable with current resources.
• The correlation between RLHF intensity and convergence suggests alignment training may amplify (or create) the effect.
• Llama 3 showing the weakest effect is consistent with either (a) less alignment → less artificial convergence, or (b) smaller model → weaker detection of genuine pattern.
• We cannot distinguish between these interpretations without controlled training experiments.

Limitations

1. Training data overlap makes cross-model replication fundamentally weaker than independent replication.

2. Only five models tested (and only two architectural families: dense transformer and MoE).

3. Estimated RLHF intensity is subjective; actual alignment procedures are not fully disclosed.

4. All models are English-primary; cross-linguistic testing not performed.

5. All models are from 2023-2024; the pattern may not hold for future architectures.

6. No non-transformer architectures tested (e.g., state-space models, retrieval-augmented systems).

Replication status

This IS a replication study. Results replicate INFO-003 findings across all tested models.

Independent external replication (different research group, different prompt protocol): not yet attempted.

The most informative next step would be testing models trained on known non-overlapping corpora.

Data and code

• Repository: [internal] o0-research/cross-model-replication
• All prompts and responses archived (3500 total responses)
• Embedding vectors stored for reproducibility
• Analysis pipeline identical to INFO-003
• Model API calls made between 2024-10-15 and 2024-11-30

Relationship to philosophical archive

Cross-model replication is consistent with the O/0 proposition ("all models describe the same thing") but is far more parsimoniously explained by shared training data. The philosophical proposition makes a claim about reality; the data shows a pattern in language models trained on overlapping text. These are not the same thing, and conflating them would be a category error. The honest conclusion: LLMs reliably converge semantically under abstraction, and we do not yet know why.

References

• Brown, T. B., et al. (2020). Language models are few-shot learners. NeurIPS 2020.
• Touvron, H., et al. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv:2307.09288.
• Anthropic. (2024). The Claude model card and evaluations.
• Gemini Team. (2024). Gemini: A family of highly capable multimodal models. arXiv:2312.11805.
• Jiang, A. Q., et al. (2024). Mistral 7B. arXiv:2310.06825.
• Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.

Revision history

• v1.0 (2025-02-01): Full cross-model replication results and confound analysis.