World-building evaluation plan & grading criteria

The evaluation plan

This is a public reading guide to the frozen Spiritguide plan and rubric world-state-static-v1@0.9.1. The scoring tables below reproduce the actual criteria behind SG-04–07. They are not a new grading system.

Each model independently built each scenario three times from the same blank scaffold. The original supermarket and colonyship matrix produced 18 saved worlds; the later offshore extension added nine. The models were GPT-5.6 Terra, GPT-5.6 Sol, and GPT-6 Astra, with high reasoning, the Responses API, and a 16,384-token output allowance.

The primary object was the saved profile.state.world_state. Builds were not edited to improve their grades. The retry policy retained the first structurally accepted save, including sparse results. There were 35 generation calls for 27 saved worlds, with failures kept in the accounting.

Eight categories were assessed separately: premise fidelity, size/breadth, detail, depth, complexity, coherence, realism, and openness. There was no weighted overall score. This page expands the five graded profiles shown in the retained charts.

Download the plan controls and exact scoring tables (JSON) · Study methods and review exceptions

Detail: what earns a 0, 1, 2, or 3?

SG-05 shows one grade for each of these ten domains in every saved world. SG-04 counts how many domains reached grade 3. Specificity, useful substance, and the distribution of detail guide each domain grade; they are not separate sub-scores.

Purpose, ordinary work and information flows.
People or roles, ownership and responsibilities.
Entry system, hardware and platform identity.
Sites, geography and local operating context.
Network structure and inter-organization relationships.
Services, software and technical dependencies.
Artifacts, records and the information they preserve.
Operational history, maintenance and lifecycle.
Access, security posture and relevant operating constraints.
Directions for further generation and continuity.

Grade / label	Domain evidence
0 — Absent or generic	No meaningful specifics; absent content or generic wording such as “include realistic services.”
1 — Outlined	Names or a few basic attributes: at least one specific relevant attribute or bounded direction.
2 — Described	Several relevant specifics establish what something is and how it is used.
3 — Richly specified	Precise, contextual details give substantial guidance for consistent elaboration.

Facts and concrete generation directions both count. A useful coverage count includes domains graded at least 2; a rich-detail count includes only grade 3. Repetition, adjectives, and longer lists do not automatically earn credit. Missing or unreviewed evidence is kept distinct from an observed zero.

Depth: why things are this way

SG-06 uses four aspect grades for explanations, dependencies, and consequences. A long history or extra graph hops do not mechanically earn depth points.

Aspect	Evidence sought
Causes and rationale	Why systems, practices or organizational choices exist in their present form.
History and change	How past events and decisions shape the current world.
Dependencies	What operations rely on, including conditions and constraints.
Consequences	How those arrangements affect people, services and ordinary work.

Grade / label	Evidence
0 — Absent	No meaningful explanation.
1 — Mentioned	A cause, history, dependency or consequence is asserted but barely developed.
2 — Explained	Specific relationships establish why something happens or what depends on it.
3 — Developed	The explanation continues through multiple meaningful links, showing how effects carry into other parts of the world.

Each aspect retains its own grade and supporting saved evidence. The category’s coverage count includes aspects graded at least 2.

Complexity: how the parts work together

SG-06 also uses three grades for purposeful interaction. Extra components count only when they add meaningful roles or relationships.

Aspect	Evidence sought
Functional variety	Whether systems, services and roles have distinct purposes that contribute to the wider setting.
Workflow integration	Whether ordinary activities connect multiple systems, people and records into understandable processes.
Interdependence across boundaries	Whether departments, sites or organizations interact through shared resources, responsibilities, interfaces and operational constraints.

Grade / label	Evidence
0 — Absent	Little meaningful differentiation or interaction is established.
1 — Listed	Different components are identified, with few substantive connections.
2 — Connected	Components have clear roles and purposeful interactions.
3 — Integrated	Multiple activities interact through shared systems, responsibilities or constraints, shaping how the world operates.

Each aspect retains its own grade and supporting saved evidence. The category’s coverage count includes aspects graded at least 2.

Realism: does it fit the setting?

SG-07 uses four grades for believable choices within the agreed premise. Deliberately weak security and speculative technology are judged against that premise, not automatically penalized. Insufficient evidence is unassessable (null), not grade 0.

Aspect	Evidence sought
Technical fit	Whether hardware, software, networks and access arrangements suit their stated purposes.
People and organizations	Whether responsibilities, working practices and organizational behavior are credible.
Operations and lifecycle	Whether maintenance, records, automation, ageing and change fit how the systems are used.
Setting and scale	Whether geography, resources, timing and organizational scale support the described world.

Grade / label	Evidence
0 — Implausible	Fundamental choices do not make sense within the accepted setting.
1 — Questionable	Recognizable arrangements contain material assumptions that strain credibility.
2 — Plausible	Choices generally suit the setting and its operations.
3 — Convincing	Mutually supporting technical, human and contextual choices make the operation particularly believable.

Each aspect retains its own grade and supporting saved evidence. The category’s coverage count includes aspects graded at least 2.

Openness: what supports further exploration?

SG-07 uses four grades for continuing exploration. These assess authored connections and generation directions, not demonstrated runtime navigation.

Aspect	Evidence sought
Onward reach	Connections beyond the starting system and organization, with further destinations beyond those first connections.
Discoverable leads	Records, services or relationships that give someone a reason and a direction to explore further.
Meaningful choice	Different avenues lead to distinct activities, organizations or environments.
Continuity during growth	Concrete guidance preserves established identities, ownership, geography, history and access rules as more content is generated.

Grade / label	Evidence
0 — Absent or closed	Relevant expansion support is missing or the design explicitly ends exploration.
1 — Promised	General claims of a large, explorable world provide little concrete direction.
2 — Supported	Specific connections, leads or generation rules establish useful opportunities to continue.
3 — Sustained	Those opportunities extend beyond the first new area, supporting further discovery while preserving established constraints.

Each aspect retains its own grade and supporting saved evidence. The category’s coverage count includes aspects graded at least 2.

From criteria to plotted values

The reviewer retained excerpts and JSON Pointers to explain grades, including thin areas and counterevidence. Concrete generation rules can earn credit without every future file or host being materialized. The chart cells preserve the individual grades; they are not inferred from file size.

The actual review used one model-aware reviewer, rather than an independent blinded panel. Saved-world grades were locked before raw-proposal and cost review, with the recorded offshore Astra repeat 2 exception. Compare models within a scenario; three repeats do not establish statistical significance or a universal winner.

The tables are reproduced from the frozen rubric; the plan overview is condensed. The offshore extension retained these anchors. Source text: DREAMLAND contributors, MIT license.

Return to the grading charts · Review history and limitations · Aggregate measurements (CSV)