Cancellation and exact retries

Role: Run submitter. Prerequisites: Run ownership, cancellation scope when using a scoped token, and saved operation files for retries. Files: The private analysis directory; retain legacy prepared operations for import and independent upload state for data-transfer recovery.

Resume an analysis directory

Repeat enumeray analysis submit DIR after an upload or acceptance interruption. Saved upload keys, the frozen scientific inputs and the exact prepared run retain one operation identity. Once submission may have reached the API, recovery replays that request even if the current publication changed; it does not create another run. After acceptance, use analysis status, analysis wait, analysis logs, analysis cancel and analysis collect with the same directory. If no receipt was saved, explicitly reconcile with analysis submit DIR before logs, waiting or cancellation; those commands never submit work. Initialize a new directory for deliberately new work. Frozen input edits and changed pre-submission sources are rejected. See analysis directories and the noninteractive recipe, which preserves review and exact retries without extracting a run ID.

Candidate analysis directories also retain candidate_trial purpose, the exact publication, captured grant and any named delegation. Activation or replacement does not change the saved trial request. A retry after a lost acceptance reply can recover the original receipt after authority changes; fresh acceptance and pending launch still require current authorization. A replacement delegation cannot revive a trial pinned to a revoked handoff.

Request cancellation

enumeray analysis cancel analysis
enumeray analysis status analysis
enumeray analysis wait analysis

analysis cancel saves private local intent in state/cancellation.json before contacting the API. A lost reply leaves an uncertain local attempt; repeat the same folder command to retry cancellation of the same accepted run. Local intent is distinct from the service’s acknowledgement. analysis status shows current service cancellation and run state; status never automatically sends a pending cancellation request. Renew credentials for the same API and account after an authentication failure, then repeat analysis cancel analysis explicitly.

Cancellation can proceed while another terminal waits or follows logs, or while submission/collection transfers run. Concurrent cancellation attempts for one directory are serialized. Ctrl-C and observation deadlines stop local observation without requesting cancellation.

If you address a run directly, the existing commands remain available:

enumeray run cancel RUN_UUID
enumeray run get RUN_UUID
enumeray run wait RUN_UUID

Cancellation records durable intent. Its status can be requested, confirmed or completed when completion won the race. Queued cancellation can terminate before assignment. Assigned work remains publicly running until the connector confirms termination or establishes that another final outcome won. Repeating cancellation preserves the original request time.

Both bundled drivers support stopping pending, starting and running work. Process execution confirms its owned process group stopped. Slurm uses its retained cluster/job identity and terminal scheduler evidence. A lost response or missing accounting record leaves cancellation pending; neither grants permission to relaunch.

After a cancelled run, retained logs, provenance and available results remain readable. Cancelling your local CLI command or reaching a wait deadline does not request server cancellation.

Monitor cancellation through the same directory:

enumeray --json analysis cancel analysis > cancellation.json
enumeray --json analysis status analysis > status.json
jq '.run | {state, phase, cancellation, execution_control}' status.json
enumeray --json analysis wait analysis --deadline 600 > final-run.json
jq '{state, cancellation, result}' final-run.json

requested means the API recorded the intent; it does not mean the backend has stopped. confirmed means termination was established. completed means another terminal outcome won the race. The final public state determines which outcome to report to your analysis. A second cancellation request is safe and retains the original request time.

Retry the same operation

After an uncertain upload or submission response, explicitly resume the same directory:

enumeray --json analysis submit analysis > recovered-submission.json
enumeray analysis status analysis

The JSON is an analysis summary with a nested receipt; the raw acceptance object is retained at analysis/state/receipt.json. Its state remains the original queued acceptance state, including on replay. Current progress comes from status or wait. An empty redirected summary after interruption does not discard the directory’s durable request identity.

For a legacy file created by the former run prepare command, import it without contacting the service, then explicitly reconcile its original request:

enumeray analysis import --prepared prepared-run.json --directory imported-analysis
enumeray analysis submit imported-analysis

The removed run prepare and run submit syntax is rejected. Import preserves the original account/API, request and key; it leaves the source file unchanged and never invents a replacement execution. For an independent upload, use enumeray artifact resume --state upload.json with unchanged source bytes.

Observed failure	Action
Timeout, disconnect or lost submission response	Repeat submit with the same analysis directory.
Validation error before acceptance	Correct editable inputs; if already frozen, preserve the original and initialize a deliberately new directory.
Current publication changed during first-submit preflight, before submission was attempted	Rediscover the model, review the new disclosure and initialize a new directory.
HTTP 429 with retry_after	Respect the indicated delay; retain the original operation.
Idempotency conflict	Restore the original request; investigate why the saved operation changed.
Wait deadline	Resume waiting on the same directory; submit nothing.
Run failed or cancelled	Collect failure/provenance evidence; new scientific work requires an explicitly new directory.

An idempotency conflict means the key was reused with a different request. Preserve and investigate the original evidence. Do not edit saved keys or frozen inputs to recover a lost reply. If both directory and operation identity are lost, inspect authorized run history and attach accepted work; history cannot prove a lost request was never accepted. Renew credentials for the same API/account before retrying.

Distinguish administrative revocation

Revoking model or compute access blocks future dispatch and stops work before launch when observed. It requests pending-only Slurm cancellation, while already executing work drains and its results remain collectable. User cancellation separately requests stopping running work. An access change is not a substitute for an explicit analysis cancel DIR request.

Expected outcome: A single recovered operation or a cancellation whose confirmed outcome is visible. Next: Ask the connector owner to reconcile uncertain execution when backend evidence remains unavailable.