Designing while the models keep changing

The assumptions change while the project is being built

Building DREAMLAND means designing around a platform that keeps changing. A workaround can take time to design, instrument, and evaluate, then become less valuable when a provider improves an output contract or releases a better model at a lower price. A faster serving tier can make an interaction feel different without changing the model at all.

That creates a maintenance problem as well as an opportunity: which complexity still earns its place? I want the project to preserve what an experiment actually showed while being willing to revisit the architecture when its constraints change.

JSON formatting has become a provider capability

Earlier experiments spent real effort distinguishing readable terminal output from a response that could be parsed and safely applied. The powerplant comparison retains thirteen explicit JSON rejections. Those results describe that historical configuration; they are not a timeless statement about what current models can reliably return.

Provider enforcement shifts this problem substantially. OpenAI’s JSON mode constrains valid JSON syntax; its strict Structured Outputs option additionally constrains supported schemas. With those controls enabled, basic formatting can largely move out of the foreground of the research. Incomplete responses and refusals still need handling, and the contents can still be wrong. Official Structured Outputs documentation.

DREAMLAND’s current boundary: the foreground OpenAI adapter requests json_object mode. It does not currently impose a strict schema on the dynamically keyed world state. Other providers and supporting-agent paths have their own output behavior. The runtime still parses the envelope and validates proposed state changes. Output and mutation contract ↗

A correctly shaped update can still invent a contradictory host, lose an established relationship, or conflict with a newer command. Provider format guarantees reduce one class of engineering work; deciding what the world may become remains part of the harness.

Lower model costs change the case for enrichment

As capable models become cheaper to run, more work can fit inside the foreground command budget. This changes the value of splitting work across agents. A background pipeline has its own calls, context, scheduling, freshness checks, and integration cost; a stronger foreground model may now supply detail that previously justified that machinery.

Both rows are a snapshot of rates checked on the same date, not launch prices or a before-and-after price history. In this example, the newer model’s listed input and output rates are half those of GPT-5.6 Sol, and its cache-read rate is one quarter. That is a concrete change in the cost of trying a newer foreground model. These are price points, not a matched-quality benchmark or a prediction that every provider and model will get cheaper. Official API pricing.

The design implication: enrichment’s marginal value may shrink as the foreground model improves within the same budget. That is a hypothesis to measure. The saved-world study did not run enrichment, and the serving-tier series disabled it; neither establishes a measured enrichment benefit or obsolescence.

Background work can still prepare detail before the user reaches it, deepen a setting during idle time, or separate a focused domain task from the active command. Those benefits need to show up in later exploration. Three successful background calls alone do not demonstrate a better world.

The useful comparison is a capable foreground model with enrichment off, the same model with enrichment on, and any proposed alternative allocation of models across roles. Compare detail the user actually encounters, continuity, command latency, and total session cost. Include background calls and proposals that never get used. Enrichment evaluation modes ↗

Speed tiers put a price on responsiveness

Is 2× the speed worth 3× the cost? New speed tiers make that a practical design question. A faster response may keep a human exploration flowing; a large automated test battery may value a different balance. A higher tier also does not, by itself, make the world more coherent.

Ultrafast’s listed per-token rates are three times Fast’s in this comparison. The premium is relative to Fast, not Standard. A rate multiplier does not guarantee the same multiplier in a session’s final bill: generated output and cache behavior also affect the total. Official API pricing.

What the three-session test observed

The October 21 series ran six commands in each of three stateful sessions: Fast → Ultrafast → Fast. Ultrafast used 25.879 seconds of command-agent time versus a 55.721-second mean for Fast: 2.15× faster completion. Estimated generation cost was $0.620765 versus $0.230049 per Fast run: 2.70× the cost.

That is about $0.39 more for roughly 30 seconds less aggregated agent time over the six-command run. It gives the tradeoff a useful scale, while leaving the value judgment dependent on the task. Agent spans are not measured end-to-end human waiting time.

The first Fast run wrote a cache prefix. Comparing the all-cache-read Ultrafast run with the final Fast run gives a 3.13× cost ratio, with different generated output volumes. Startup behavior also differs from completion: the final Fast run had a lower model time to first token on all six commands. First model token is not the first visible terminal character.

There was only one Ultrafast session. The observed speedup does not establish repeatability, tail latency, or a guaranteed future average. Cache state, fixed run order, provider conditions, output size, and evolving world state all matter.

Choose the tradeoff for the interaction

For a person exploring interactively, a shorter wait may justify a higher per-command cost. For unattended evaluation, bulk world preparation, or supporting detail the user may never see, the same premium needs a different justification. Tier choice should be tested against the experience and total budget, rather than becoming a permanent assumption that faster is always preferable.

Keep the design open to being simplified

The difficult part is deciding whether a component is solving today’s constraint or preserving yesterday’s workaround. New provider guarantees can reduce parsing machinery. Lower foreground costs can weaken the need for auxiliary agents. Faster tiers can improve completion while reopening the budget question.

For each comparison, retain the model identity, serving tier, reasoning settings, output contract, context and cache behavior, workload, output limits, and pricing date. Revisit the decision when those conditions change. Older results remain useful evidence of the original system; they need context when informing the next one.

I want DREAMLAND’s architecture to follow what improves the simulation, even when that means simplifying a part of the system I previously spent time building.

Return to the research figures · Explore the engineering decisions