File inputs and outputs

Role: Model user. Prerequisites: Model access, configured service storage and appropriate upload/artifact token scopes. Files: Original source bytes and a private analysis directory; independent reusable uploads have their own private upload state.

Use the file and download chooser to distinguish scientific inputs, wrapper requests and collection manifests. The model example bundle supplies the scalar, file and collection projects, fixtures and local tests; its offline tests need Python 3.9+ and no account. The service commands below require an installed, accessible model.

Manage files in an analysis directory

The analysis workflow handles uploads and artifact references automatically:

Run in a new private working directory with the installed file starter available as csv-example:

umask 077
cat > dataset.csv <<'CSV'
index,value
0,1
1,2
CSV
enumeray analysis init csv-example --directory analysis \
  --example starter-example --file dataset=dataset.csv
# Review the saved destination/disclosure and local binding before submission.
enumeray analysis submit analysis
enumeray analysis wait analysis --check-outcome
enumeray analysis collect analysis

Use your model’s actual ID, example and declared slot. For these two file-starter rows, the result is sample_count: 2 and the processed_dataset CSV contains 0,2 and 1,4. Original local bytes are uploaded for remote staging; the directory retains source paths/checksums. Inputs remain editable until first submit freezes them; source bytes must remain unchanged while upload is unfinished. bundle.json maps declared output names to verified downloaded files. Repeat submit/collect with the same directory after interruption. The independent artifact operations below remain useful for reusable data and authorized investigation. Ordinary execution needs no artifact-ID extraction or separate upload-state management.

Upload reusable data independently

For an independent reusable upload, use the author’s file starter contract: its sole input dataset is a text/csv file with columns index,value. Confirm that your selected publication declares this slot before using this input. Install jq for the commands below. In a new private working directory:

umask 077
cat > dataset.csv <<'CSV'
index,value
0,2
1,3
CSV
cat > metadata.json <<'JSON'
{"description": "Two demonstration measurements"}
JSON

enumeray --json artifact upload dataset.csv --name dataset --media-type text/csv \
  --metadata metadata.json --state upload.json > upload-receipt.json
ARTIFACT_ID=$(jq -er '.artifact_id' upload-receipt.json)
enumeray artifact get "$ARTIFACT_ID"

The upload command saves retry state before contacting the API. Its operation reserves an upload, transfers the file, records the exact storage version and confirms completion. Read the resulting completed artifact UUID. resume can repeat reservation, transfer or completion using the same saved identity and verifies the original source checksum. Keep that source file unchanged and available.

Resumption retries the saved operation and retains confirmed multipart parts. An interrupted part transfers again from its beginning. HTTP multipart run attachments use a separate bounded request format.

Supply scientific metadata through --metadata metadata.json when appropriate. The model’s declared media types and byte limits must match the completed artifact.

--name is the artifact’s logical name, not a filename to stage on compute. --media-type declares the content type, and --metadata takes a JSON object describing the science. --state is a new local operation file; do not replace it on retry. A completion response contains artifact_id, which is the public reference to use. The upload ID identifies the transfer operation and cannot substitute for the artifact ID.

If uploading fails after the state was saved, recover the same operation:

enumeray --json artifact resume --state upload.json > upload-receipt.json
ARTIFACT_ID=$(jq -er '.artifact_id' upload-receipt.json)

The state includes the account/API, absolute source path, expected bytes, SHA-256, idempotency key, reservation and exact transferred version when known. Resume checks the original source if another transfer is needed. After a completed upload, resume returns the existing completion. Keep the source at its saved path until completion is confirmed.

Reuse a verified input artifact

For a model whose input schema declares dataset as a file:

{
  "dataset": { "artifact_id": "ACTUAL_COMPLETED_ARTIFACT_UUID" }
}

Use a real completed artifact UUID, not the label above. Preserve all other fields required by that model. Public inputs use artifact references, never local paths or storage URLs. The connector verifies the pinned storage version, size and checksum before staging a workspace-relative path for the wrapper. An artifact owned by someone else is not made readable by guessing its UUID. Current model owners/co-admins can inspect and download artifacts pinned by accepted runs or produced by their model; their reads are audited. An unrelated standalone upload remains private.

For the file starter, generate a complete inputs.json from the upload receipt:

jq -n --arg artifact "$ARTIFACT_ID" \
  '{dataset: {artifact_id: $artifact}}' > inputs.json
enumeray analysis init csv-example --inputs inputs.json --directory reused-data
# Review the publication and destination before submission.
enumeray analysis submit reused-data
enumeray --json analysis wait reused-data --check-outcome > final-run.json
enumeray analysis collect reused-data

Replace csv-example with the published model’s real ID. The same completed artifact can be referenced by subsequent authorized runs; each acceptance pins its verified content. You do not need to upload unchanged data again for each run.

The analysis workflow submits the saved JSON and retains exact file dependencies. For an original local source, use analysis init --file dataset=dataset.csv instead; the CLI handles upload and references for you.

Download verified outputs

analysis collect DIR downloads declared outputs and records their verified paths in bundle.json. The file starter maps processed_dataset to its downloaded CSV; inspect results/outputs.json for the ordinary sample_count value.

For an independent artifact download or investigation by a known accepted run ID:

enumeray run artifacts RUN_UUID
enumeray artifact get ARTIFACT_UUID
enumeray artifact download ARTIFACT_UUID --output result.csv

Obtain the UUID from run outputs or its artifact listing. Downloads verify byte counts and SHA-256 and do not replace existing destination files. Choose a new output filename when retaining another copy. A temporary download capability can expire; request fresh access instead of retaining capability URLs for reuse.

For input rows 0,2 and 1,3, the file starter’s output CSV contains 0,4 and 1,6, with ordinary JSON output sample_count: 2. Other models use their own output names and formats. run artifacts lists scientific artifacts produced by the run; input artifacts and the separate log archive are not in that list.

The API’s artifact-content endpoint returns a checked 303 See Other with a temporary storage capability in Location. Request that URL separately without API bearer headers. The service verifies the exact published version and rechecks credential, current model membership and availability before issuing it. The capability can be reused until its expiry, even after API access is revoked; never log or retain it as artifact identity. Verify the downloaded size and full-object SHA-256 against artifact metadata. The independent CLI performs these steps.

Common failure boundaries

An uploaded file is usable only after completion verification. A terminal model outcome and a log archive’s availability are independent. Missing storage, expired uploads, source changes, permission failures and output-schema violations have different remedies; see data and recovery troubleshooting and limits.

Expected outcome: Completed input artifacts and locally verified output files. Next: Retain provenance alongside your analysis results.

Upload large files and recover parts

New input, output and log-archive uploads select multipart object storage automatically above 64 MiB. Smaller uploads retry one bounded PUT. Keep the same source bytes and private operation state. Resume verifies the source, reconciles confirmed parts and sends only missing parts; it can also recover a lost storage or API acknowledgement. Refreshed storage authorization retains the original reservation and artifact identity. Existing reservations keep their original transport and deadlines.

After transfer, the storage worker verifies the entire completed object before publication. The CLI waits through this phase. An interruption remains recoverable with the same saved operation. A stopped worker or unavailable storage delays publication without launching another scientific execution.

The default per-file admission limit is 100 GiB. Aggregate upload or connector spool exhaustion can delay new transfers; preserve finalized outputs until normal delivery and cleanup settle. Connector owners can adjust limits through the documented storage policy and connector options. Higher limits need workload qualification; a declared file limit does not promise throughput.

Resume downloads

Repeat the same artifact download ID --output PATH or analysis collect DIR command after interruption. Downloads reuse a verified prefix of the exact published object version. The CLI prints the checkpoint location and resumed byte count; JSON completion includes resumed_bytes, transferred_bytes and verified_bytes.

A private .enumeray-download-… directory beside each destination retains the partial payload, lock and checksum checkpoint. Its compact receipt remains after completion to reconcile a lost reply. Keep it until the transfer is settled. Conflicting destinations and corrupt, truncated or identity-mismatched checkpoints are preserved and rejected; inspect the printed location and explicitly move conflicting state aside before restarting. The CLI never overwrites another destination file.

Storage authorization is refreshed through the same artifact endpoint. If API access expires, renew the same account with auth refresh or auth login and repeat the command. Checkpoints contain no tokens or temporary storage URLs. Every completed file is checked against the full published SHA-256 before publication.

Immutable file collections

Collection downloads retain collection.json with immutable collection metadata and manifest.json with every pinned member, even when selecting only a few files.

A collection is a bounded, immutable set of regular files with relative paths, media types and scientific metadata. Each completed member pins its artifact UUID, exact version, size and full-object SHA-256. Collections support up to 10,000 members, subject to the model’s smaller declarations, a 16 MiB manifest budget, existing byte quotas and the 10,000-file aggregate limit in each run direction. Every listed member is required; a collection becomes usable after all members verify. Its sealed identity remains recorded when individual bytes later become unavailable.

Use the complete two-member analysis example to create tiny source files and {} scientific inputs, bind them with --collection dataset=collection-inputs, review the destination/disclosure and submit the resulting folder. The CLI constructs verified collection references automatically; no UUID extraction is required. For selected paths and per-member media types, use --collection-manifest dataset=MANIFEST.json with the manifest in its source root and the analysis outside that tree.

For that example’s accepted terminal folder, collect manifest evidence, one selected member, then the complete outputs:

enumeray --json analysis collect collection-analysis --evidence-only > evidence.json
enumeray --json analysis collect collection-analysis \
  --member snapshots=mesh/grid.bin > partial.json
enumeray --json analysis collect collection-analysis > complete.json

The first two fresh steps report complete: false; the last reports complete: true and publishes bundle.json. Repeat the same folder command after interruption. Scientific success/failure/cancellation and collection completeness remain independent, and collection never launches the model again. Keep original input bytes separately when they are needed for reproducibility.

Manage reusable collections independently

For an independent reusable upload, place operation state outside the source tree:

enumeray collection upload --directory collection-inputs --name dataset \
  --media-type application/octet-stream --state dataset-upload
enumeray collection resume --state dataset-upload

For selected files or mixed metadata, write a manifest beside the source tree:

[
  {"path":"mesh/grid.bin","media_type":"application/octet-stream","scientific_metadata":{"units":"m"}},
  {"path":"snapshots/0001.bin","media_type":"application/octet-stream"}
]

enumeray collection upload --manifest members.json --root collection-inputs \
  --name dataset --state selected-dataset-upload

These commands are alternatives for independent reusable data; they do not submit an execution. Keep sources unchanged until completion. Directory traversal includes regular files, including hidden files; paths must use canonical Unicode NFC. Symlinks, traversal, duplicate paths, case aliases and the reserved .enumeray-download- checkpoint prefix are rejected. Membership is frozen before transfer and cannot be edited during resume. A changed set needs a deliberately new upload operation. Use {"dataset":{"collection_id":"COMPLETED_COLLECTION_UUID"}} to reuse a completed collection in that declared slot.

List outputs and download exactly the members needed:

enumeray run collections RUN_UUID
enumeray collection members COLLECTION_UUID
enumeray collection download COLLECTION_UUID --output selected \
  --member snapshots/0001.bin

collection members returns at most 100 members and 64 KiB per page; pass its next_cursor with --cursor. Downloads retain the full manifest, transfer only the selected members and resume verified bytes after interruption. Omit member selection to download all members. Existing conflicting files are preserved.

analysis collect --evidence-only saves terminal provenance and full input/output collection manifests without fetching scientific bytes. Partial selections stay incomplete; the final bundle.json is published only after every output is locally verified. Repeating the default collect command completes the same analysis and never launches the model again.