DREAMLANDStudy design and controls
The original cohort used supermarket and colonyship premises, with three models and three scheduled independent repeats per premise/model cell: 18 saved worlds. The later offshore extension added nine worlds without rerunning or regrading the originals.
All generations began from the same blank world with the frozen Spiritguide system prompt, Responses API, high reasoning, and a 16,384-output-token allowance. The prompts contained 62 words for supermarket, 86 for colonyship, and 202 for offshore. Offshore was designed after the earlier findings had been inspected.
Compare quality within a scenario. The study cannot isolate prompt-length effects, establish statistical significance, or produce a universal model ranking.
Rubric and measured objects
Read the evaluation plan and exact grading anchors ↗
The primary object was saved profile.state.world_state, reviewed under world-state-static-v1@0.9.1. Eight categories remained separate: premise fidelity, size and breadth, detail, depth, complexity, coherence, realism, and openness. There was no weighted overall score.
Ten detail domains covered purpose/work, people/ownership, entry system, places, networks, services, records, history, access/security, and expansion guidance. Useful coverage counted grades at least 2/3; rich detail counted grade 3/3. Facts and concrete generative directions could support grades.
Size was canonical world-only UTF-8 JSON bytes / 1,024. Word counts used string values, excluding keys. Named entities were deduplicated and kept separate from materialized registry objects, population claims, and unspecified equipment families.
Review order and exceptions
One model-aware reviewer assigned semantic grades. The review was neither independent nor blinded. Saved-file grades were locked before auditing raw proposals and costs, with one recorded exception: the short completion message for offshore Astra repeat 2 was visible during timeout reconciliation before grading. Its full proposal and cost remained withheld.
The retry rule was adopted after the first Terra failure and applied to later calls. The first structurally accepted save was retained even when sparse. Failures have no invented quality grades; their generation costs and counts remain in the accounting.
Both original cohorts and their grading history were preserved in the combined report. Minor evidence-label corrections did not change the saved artifacts or grades.
Cost, transport, and source attestation
Costs count each unique generation once, including failed generations and observed cache effects, rather than adding parent observations. Provider-delivered service tier and SDK-internal transport retries were not attested. Historical charges should not be interpreted as current prices.
Two completed offshore Astra generations exceeded the 300-second MCP response wait. Saved profiles and matching completed telemetry supported reconciliation without retrying the successful generation.
The 124 local source files matched the recorded source freeze; the loaded MCP server Git revision was not attested. Controlled-source cohort identity therefore remains provisional. This qualification travels with comparisons that use the study.
What was not tested
No live walkthrough, simulated authentication, command execution, route traversal, or enrichment was used to grade these saved worlds. Authored access and network depth are declarations, not demonstrated functioning behavior.
No complete atomic claim-retention census was performed. A count of omitted fact objects is not a semantic-retention percentage. Selected technical checks and manufacturer references do not certify all versions or a real installation.
Three repeats per cell describe observed variation. Min–max ranges are not confidence intervals. Zero confirmed contradictions in the saved scope does not establish perfect coherence.