Queued, failed or uncertain execution

Role: Run submitter for initial inspection; model and connector owners for execution investigation. Prerequisites: Actual run UUID and original receipt. Files: Analysis directory and acceptance receipt; connector owners additionally retain journal and backend evidence.

enumeray run get RUN_UUID
enumeray run events RUN_UUID
enumeray run logs RUN_UUID

Inspect state, phase, freshness, blocked reason and cancellation status together. A wait deadline only ends client observation. A model may spend time preparing inputs, waiting in Slurm or delivering results after compute finishes.

With jq installed and your receipt available:

RUN_ID=$(jq -er '.run_id' receipt.json)
enumeray --json run get "$RUN_ID" > run.json
jq '{state, phase, observation_freshness, last_observed_at,
     blocked_reason, execution_control, cancellation, failure: .result.failure}' run.json

result can be null while work is outstanding. Preserve the run ID, publication ID, state and latest observed time when escalating. stale describes the age of an API-confirmed observation; it does not prove that the process or scheduler job stopped.

Symptom	Evidence to inspect / action
Queued with a blocked reason	Publication/installation revocation, readiness, model access, compute grant or connector lifecycle; see state reference
Queued without a blocked reason	Connector availability and global/user/context/grant capacity; connector or service owner
Candidate never ready	Exact configuration generation, installation cache/capacity, entrypoint/script digest, execution identity and backend access
Direct launch rejected	Restricted launcher and separate execution account; do not give the model account connector credentials
SSH validation/staging fails	Installed remote runner, trusted host keys, private credential reference, SFTP and account/root permissions
Slurm startup fails	Batch file, compute-node worker, shared work root, runtime on compute nodes and reported startup evidence
Slurm queue disappears	Exact retained cluster/job identity and accounting, plus worker ownership; absence alone proves no outcome
Backend finished, run still running	Result validation, collection, output transfer and exact-version publication
Cancellation remains requested	Connection and terminal backend evidence; a lost reply or accounting gap leaves confirmation pending

Preserve uncertainty

The connector owner can inspect operational evidence without model-data access:

enumeray connector runs lab-connector
enumeray connector get lab-connector
enumeray connector diagnostics lab-connector

On the connector host, under the connector account and using its real journal:

enumeray connector inspect --journal /var/lib/enumeray-connect/journal

Use connector recovery for the exact stopped-daemon, exclusive-ownership and reconciliation workflow. Contact Enumeray support for global queue pressure or unresolved service-side evidence.

Do not resubmit a scientific run, erase a journal or manufacture a failed result to clear operational attention. The connector writes launch intent before submission and uses exact backend evidence to reconcile lost acknowledgements. An ambiguous Slurm match or missing terminal receipt must remain uncertain.

User cancellation and administrative revocation differ: cancellation requests stopping running work, while revocation allows already executing work to drain. Inspect confirmed status rather than assuming revocation killed an allocation.

Expected outcome: A verified state or an explicit unresolved evidence gap. Next: The connector owner follows recovery and contacts Enumeray support when necessary.