API & MCP evaluation

Three complementary interfaces

The optional admin service exposes a shared capability model through REST and MCP. These bindings reuse runtime operations instead of scraping terminal output or reimplementing simulation behavior.

The API is optional, enabled at startup, local by default, and protected by bearer authentication. Mutating MCP tools require an explicit configuration gate. These interfaces support DREAMLAND admins and testing, not demo connection credentials or a promise of a publicly exposed admin service.

Create → exercise → inspect → close

1. Discover. Read the agent guide, status, and capabilities. Tool discovery alone does not prove a live provider call can succeed.
2. Create. Start a clearly labeled session and attach run/scenario tags. Retain the complete session_id.
3. Exercise. Send ordinary commands sequentially. Record stdout, stderr, exit_status, interruption state, and returned command_id.
4. Inspect. Correlate the live trace, direct command diagnostics, state deltas, and enrichment events. Do not confuse metadata with terminal output.
5. Close. Close the exact test session, including after a failed run. Avoid bulk cleanup of unrelated sessions.

dreamland_session_create
  → dreamland_session_run_command
  → dreamland_session_trace_read
  → dreamland_session_command_inspect
  → dreamland_session_state_delta
  → dreamland_sessions_close

This is a workflow sketch, not an executable script. MCP headless commands use the same backend state contract as SSH while bypassing the transport. That is useful for owning-layer simulation tests; use real SSH tests when transport behavior is the subject.

Inspect evidence through REST

REST exposes structured data for runtime status, session traces, current state, historical command inspection, and retained background jobs. Selected read endpoints are:

GET /api/v0/status
GET /api/v0/capabilities
GET /api/v0/config/effective
GET /api/v0/sessions/{session_id}/state
GET /api/v0/sessions/{session_id}/trace
GET /api/v0/sessions/{session_id}/commands/{command_id}
GET /api/v0/sessions/{session_id}/state-delta
GET /api/v0/sessions/{session_id}/enrichment-events

Requests require bearer authorization. Use bounded queries and retain identifiers. State deltas help distinguish a wrong terminal response from a missing artifact, incorrect working directory, or host-frame transition. Enrichment events connect a background outcome to its exact job rather than requiring inference from interleaved logs.

Live and historical evidence have different availability. Some historical inspection depends on optional Langfuse ingestion; flush telemetry when appropriate, and report unavailable or delayed evidence rather than treating it as an empty successful run. Retained enrichment events have bounded in-process retention.

Session tracing with Langfuse

With thanks to Langfuse for the observability tooling used in DREAMLAND’s research. Its tracing and session views help turn an interaction into evidence that can be reviewed after the terminal session ends.

A session contains many command traces

When tracing is enabled, DREAMLAND assigns a stable session ID to an SSH or MCP session and records a session-start event. Each LangGraph command invocation produces its own trace, grouped under that session. This preserves the relationship between individual commands and the longer exploration. Langfuse’s session view brings those traces together for chronological review.

Look inside the agent work

• Model calls: inspect recorded prompts, responses, timings, and model metadata when available.
• DREAMLAND admin guidance: examine Spiritguide instructions, parsed updates, and dry-run outcomes as separate DREAMLAND admin observations.
• Background enrichment: follow the Coordinator, Expert, and Integrator workflow, including validation and application outcomes, alongside the foreground session.
• Resource use: compare token usage, latency, and recorded or calculated costs. Cost interpretation depends on captured usage and the applicable pricing configuration.

Connect the record to a research question

If a later command contradicts an earlier observation, use the session record to locate the relevant model responses, then use DREAMLAND’s command inspection and state-delta tools to investigate what changed. The same evidence helps assess whether enrichment added useful detail or merely consumed additional calls.

Tracing is optional. A recorded interaction supports analysis; it does not by itself restore an executable simulation or capture every terminal byte. Saved state profiles provide the separate mechanism for carrying a world into another session.

Compare configurations without changing defaults

Admin MCP session creation accepts a selected state profile and per-session configuration overrides. With the normal runtime factory, sessions own isolated command agents, Spiritguide settings, enrichment graphs, event buses, workers, timers, and scheduling state.

A test can hold the saved starting world constant while varying the command model, reasoning/output controls, or enrichment settings. Spiritguide and the Coordinator, Ambient Coordinator, Expert, and Integrator can be configured independently.

{
  "state_profile": "evaluation-baseline",
  "label": "enrichment-shadow-run",
  "tags": ["experiment:continuity", "variant:shadow"],
  "config_overrides": {
    "enrichment": { "apply_mode": "shadow" }
  }
}

These are illustrative dreamland_session_create arguments; a DREAMLAND admin must first create the named profile. Overrides are fixed at creation. Credentials and arbitrary environment or prompt-path changes are not accepted through these controls. Inspect resolved settings and provider-reported usage rather than inferring delivered service from a requested setting.

Keep the evaluator outside the control room

The evaluator here is an unwitting agent: it is given an objective and access to a terminal, without being told that the environment is a simulation. It explores using the same clues available to an ordinary user, treating the systems it encounters as real. The aim is to let it pursue its objective naturally, rather than steer its behavior with knowledge of how DREAMLAND works.

That exploration produces test traces we can examine for consistency problems: a file whose contents change unexpectedly, conflicting information about a host, or a command result that contradicts an earlier observation. The agent focuses on its task while the trusted test harness records the evidence for review.

Repeat the progression, change the configuration

We give the unwitting agents a task brief describing the progression to attempt: identify the starting system, discover other hosts, follow available access, work inside another program, and return through the earlier shell contexts. They receive objectives and interaction rules, while the harness keeps the simulation’s hidden state and scoring criteria separate.

Each comparison run starts from the same saved scenario with the same task brief. The harness changes the variable under evaluation, such as the model answering terminal commands, while keeping the other configured controls fixed. The agent then attempts the progression again, choosing commands from what it discovers. This repeats the task rather than replaying an identical command transcript: the generated environment, access outcomes, and subsequent choices can differ between runs.

The resulting traces let us compare where the progression succeeds, stalls, or exposes a contradiction. Did the destination host stay consistent? Did values persist inside an interactive program? Did exiting restore the correct user and shell? A believable command response can still be followed by an incorrect state transition.

The powerplant test plan gives a concrete example of this progression and its fourteen evidence gates; PP-02 shows the outcomes across nine runs. That historical cohort used mixed executors and evolving briefs, so it was not uniformly unwitting or blinded. It illustrates the task structure, while the restricted client interface provides the separation described here.

The separate client MCP endpoint supports this separation by exposing only a client guide and three gated tools:

client_session_create
client_run_command
client_session_exit

It does not expose admin diagnostics, structured failures, command IDs, telemetry, or bulk cleanup. Command results include prompt-like context such as current user, host, working directory, and active program, so the agent can understand a pivot without privileged inspection.

The trusted harness can verify the selected profile label at session creation and record user-visible command events. Model/tier selection and nested role overrides belong on the admin surface, not the restricted client's creation call. This separation keeps the agent focused on what it can discover through the terminal, while researchers can inspect the underlying state and traces separately. It helps avoid both privileged shortcuts and behavior shaped by advance knowledge of the simulation; it does not guarantee that an agent will never notice something unusual.

Investigate failures without erasing them

Structured failure capsules preserve rejected model output and pre-command context for diagnosis. Parser replay can test how a captured response is interpreted without making another provider call; it does not replay the entire world or establish what a fresh model would produce.

Keep transport, provider, parser, simulation continuity, and cleanup outcomes separate. A successful health response is not a quality result, and a provider failure should not disappear from a cost or reliability comparison.

The optional Inspect AI harness supports trusted run specifications, command-event artifacts, manifests, preflight/postflight evidence, and scoring. Its generic client agent can explore while the harness records the run. Scenario conformance and command-artifact integrity answer different questions and should remain distinct.