Logs, events and provenance

Role: Run submitter or current model administrator. Prerequisites: Read access to the run. Files: Private analysis folder and its automatic log checkpoint; optional independent provenance export and archive download destinations.

Retain an analysis bundle

enumeray analysis collect DIR saves terminal inspection, effective inputs, scientific JSON outputs and provenance with checksum-verified files. Its bundle.json is indexed by declared output name and retains original source paths/checksums. Original input bytes remain external. Repeat collection to verify/reuse completed files and resume missing ones; existing conflicts are preserved. Scientific outcome and collection completeness are separate. See analysis directories for the staged workflow.

For an accepted analysis, use enumeray analysis logs DIR for retained pages or enumeray analysis logs DIR --follow for live logs. These commands use the saved receipt without requiring a run ID. If submission is uncertain and the directory has no receipt, explicitly repeat analysis submit DIR first to reconcile that operation. Log observation never submits work.

Inspect retained observations

enumeray run logs RUN_UUID
enumeray run events RUN_UUID
enumeray run provenance RUN_UUID --output provenance.json

Logs contain retained source text and gap/reset records. Events describe lifecycle observations; cancellation facts can be included independently of the connector-event cursor. The result is established by run inspection, not by a final-looking log line.

Log and event responses use opaque continuation cursors. When a response supplies next_cursor, pass its value with --cursor to continue. Do not derive a cursor from an offset or change its contents. Current log history is bounded to 1,024 entries and seven days, so older payloads may no longer be available.

For interactive reading, use enumeray run logs "$RUN_ID" --limit 50 and follow the printed Next page command. Human output retains gap/reset warnings and prints a separate Resume later command for polling after that page; continue through remaining pages before polling later. Events include cancellation facts that can repeat on each page. An expired cursor stops the command; review the reported retention gap before explicitly restarting without --cursor to read the earliest retained entries. HTTP clients can instead use the X-History-Recovery-Cursor response header; the CLI does not silently adopt it.

For scripts, with jq and the RUN_ID from your receipt, save and read retained history page by page:

enumeray --json run logs "$RUN_ID" > logs-1.json
jq -r '.items[] | "[\(.source)] \(.text)"' logs-1.json
CURSOR=$(jq -r '.next_cursor // empty' logs-1.json)
if [ -n "$CURSOR" ]; then
  enumeray --json run logs "$RUN_ID" --cursor "$CURSOR" > logs-2.json
fi

Repeat using the next page’s next_cursor until it is null. Each entry also includes offset, byte_length, epoch, gap and source_reset; keep the JSON if you need to preserve evidence of omitted data. Once the current history is drained, use .resume_cursor with --cursor on a later poll to read new entries without replaying the old page. Events use the same paging pattern, but their item structure differs and cancellation_events can repeat independently of the cursor.

Watch a live stream

Follow the run saved in an analysis directory:

enumeray analysis logs analysis --follow --interval 2 --deadline 3600
# Repeat the same command to resume after interruption or credential renewal.
enumeray --json analysis logs analysis --follow >> log-events.ndjson

The private checkpoint is automatically stored at analysis/state/logs.json, bound to the API origin, account and accepted run. One follower can own this checkpoint at a time; keep its lock file in place. Status, wait, cancellation and submission/collection transfers can continue concurrently. Retained-page commands such as analysis logs analysis --limit 50 remain independent. Use --cursor only for retained pages; analysis follow always resumes its checkpoint and has no --state option.

For a run addressed directly by ID, the existing commands remain available:

enumeray run logs "$RUN_ID" --follow --state watch.json --interval 2 --deadline 3600
# Repeat the same command to resume after interruption or credential renewal.
# For scripts, use a fresh output file or append to your existing event journal:
enumeray --json run logs "$RUN_ID" --follow --state watch.json >> log-events.ndjson

Both forms of follow check run state independently, stop the stream after a terminal observation, and drain retained final log pages. They exit 0 after successful observation of succeeded, failed or cancelled. Use analysis wait DIR --check-outcome or run wait --check-outcome when execution outcome should control a script. Stream silence/closure and final-looking log text do not establish completion. Follow does not wait for optional archives or connector delivery completion.

The two-second interval controls run inspection. The one-hour deadline includes requests, reconnection and final draining; expiry exits 5. Ctrl-C exits 130. Both stop observation only. Other errors exit 1. Transient disconnects and service errors reconnect from the last processed cursor with bounded backoff and respect Retry-After.

Both the automatic analysis checkpoint and an explicit watch.json are private and bound to the API origin, authenticated account and run. Each stores only a version, those identities, an opaque cursor and last displayed position. Atomic checkpoints follow successfully flushed output. A crash between output and checkpoint persistence can replay entries; consumers may deduplicate by run and log position. Each checkpoint permits one writer, enforced by a companion .lock file that should remain in place. Saved browser sessions renew before expiry, and live logs reconnect from the last processed cursor. Revoked access, unavailable refresh credentials or an uncertain renewal outcome stop observation without discarding its checkpoint. Follow the error guidance to renew or log in to the same account, or replace its scoped token, then repeat the folder or run command against the same API. Checkpoint creation/resumption also needs me:read to verify account identity.

For run logs without --state, supply an optional --cursor and retain the printed resume command or NDJSON event IDs. --state and --cursor are mutually exclusive. For either command family, --limit is for retained pages and cannot be combined with --follow; --interval, --deadline and --allow-gaps require --follow.

Retention gaps stop following after reporting the missing prefix. Review the gap, then repeat with --allow-gaps to explicitly acknowledge missing history and resume from the server’s recovery cursor. Gap/reset records stay visible. A terminal run whose retained logs were pruned follows the same rule.

Human mode writes source-labelled, terminal-safe logs to stdout and observation/recovery messages to stderr. --json --follow writes one {event,id,data} JSON record per log or stream control. Non-null log IDs are opaque resume cursors; final drained pages provide an ID at each page boundary, with null IDs for preceding entries. A final event: "run" record carries the unchanged terminal run response and has id: null. Errors remain JSON on stderr. Ordinary paginated --json run logs output is unchanged.

The underlying HTTP endpoint /api/v1/runs/{run_id}/logs/stream remains available. For authenticated curl, initial cursor selection and reconnection details, see HTTP log streaming.

Model-declared solver files appear under model.<name> alongside built-in sources. They use the same pages, live stream and checkpoints. A missing diagnostic file can be reported unavailable while the scientific result still succeeds. Sources have independent raw offsets and epochs; their order does not establish exact chronology across files. Named sources also appear in optional archive manifests.

Optional raw log archives

enumeray run log-archive RUN_UUID
enumeray artifact download ARCHIVE_UUID --output logs.tar.gz

Use the artifact UUID from a ready archive attachment. Archives are disabled unless the connector owner enables them in the connector’s log policy. They retain bounded source prefixes and recent tails in a .tar.gz containing fragments and manifest.json. Inspect that manifest for ranges, omitted data and completeness; a valid archive can be partial.

Archive publication status is capturing, pending, ready, unavailable or expired. Publication proceeds separately after scientific completion, including failed and cancelled runs. Missing storage or an archive error does not change scientific success or failure. Published attachments expire after 30 days; retrieve them while authorized and available.

After downloading, list the archive and read its manifest in a new directory:

tar -tzf logs.tar.gz
mkdir archived-logs
tar -xzf logs.tar.gz -C archived-logs
cat archived-logs/manifest.json

Use the recorded source ranges and omitted-byte information to judge completeness. Scientific results remain available through run get and ordinary output artifacts even if the archive is not ready.

Interpret provenance

The export retains the accepted publication, scientific definition, inputs, execution destination, installation snapshot and available execution/result evidence. Store it with outputs when recording a scientific analysis. Identity strings are author declarations; an export is not an independent proof of scientific correctness or a complete runtime/dataset attestation.

run provenance RUN_ID prints the structured view. Adding --output provenance.json downloads the export bytes to a new file and prints a receipt containing the local path and size. It preserves existing files. Save the terminal run inspection alongside it:

enumeray --json run get "$RUN_ID" > final-run.json
enumeray run provenance "$RUN_ID" --output provenance.json

Expected outcome: Readable observations with gaps understood, plus provenance and any available archive. Next: Investigate incomplete delivery or inspect execution state.