Capacity, credentials and recovery

Use these procedures as the connector owner. Host commands assume the quickstart’s systemd service and paths. Keep the original journal, credentials, private SSH map and immutable releases. Contact Enumeray support for exceptional assignment resolution that cannot be recovered through the connector.

Inspect capacity and configuration

enumeray connector list
enumeray connector get lab-connector
enumeray connector diagnostics lab-connector
enumeray connector runs lab-connector
sudo -u enumeray-connect /usr/local/bin/enumeray connector inspect \
  --journal /var/lib/enumeray-connect/journal
sudo journalctl -u enumeray-connector -n 100 --no-pager

get shows the owner, ownership generation, configured connector generation, observed configuration generation, maintenance (draining), reversible authority withdrawal and capacity. diagnostics separates retained queued work, unfinished assignments by reported phase, unconfirmed work, terminal assignments awaiting durable delivery receipts, active uploads and pending log archives. It also reports configuration lag, failed publications and configuration capacity. inspect reads local journal identities and cleanup retention without showing scientific inputs or secrets. None of these commands launch or resolve work. Configuration pending after an update is normal until acknowledgement; persistent lag or failed publications needs investigation of the actual host configuration. configuration_capacity includes serialized desired bytes, the 2 MiB desired and 4 MiB cache limits, and historical, retained, requested and retired publication counts. Server retirement does not establish that a host’s cached files have already been reclaimed; inspect local cleanup as well. Local inspect also reports configuration_retention.release_protection_ready and release_protection_blocker. Investigate that blocker when cleanup is withheld even though scientific work continues; the protection inventory has a separate 1 MiB bound.

Limit	What it bounds	Where to change it
Connector dispatch capacity	Outstanding API assignments to that connector	connector configure
Context capacity	Outstanding assignments on that context	Context creation; replacement for execution-policy changes
Grant capacity	Queued runs and outstanding assignments across publications using the author grant	connector grant update
--max-workers	Assignments reserving compute, including pending Slurm allocations	Connector service argv
--max-transfer-workers	Concurrent bulk I/O subprocesses	Connector service argv
--max-pending-assignments	All retained unfinished assignments, including unpublished outputs	Connector service argv

The default runtime limits are one compute worker, two transfer workers and 64 pending assignments. Pending capacity must be at least compute capacity. Compute is released after allocation-release evidence, while pending output publication still counts against API connector/context/grant limits. Storage outages can fill the retained-assignment cap and apply backpressure; observation and cancellation continue independently of bulk I/O.

Read the current generation, then substitute it in a guarded change:

enumeray connector configure lab-connector --label 'Lab connector' \
  --max-outstanding-assignments 8 --generation 3
enumeray connector get lab-connector

Here 3 is illustrative. Lower capacity preserves accepted work and restricts new assignments. A conflict means someone changed the resource; reread it. A lost reply also requires rereading before retrying, because the mutation may have committed.

Update a compute grant’s limits

Inspect the exact grant before replacing both limits:

enumeray connector grant list lab-connector
enumeray connector grant get lab-connector GRANT_ID
enumeray connector grant update lab-connector GRANT_ID \
  --max-queued-runs 50 --max-outstanding-assignments 2 --generation 1
enumeray connector grant get lab-connector GRANT_ID

Replace GRANT_ID with the returned UUID and 1 with its observed grant generation. Owners/current co-admins can inspect and update their connector’s grants; a grant holder can inspect their own, including revoked grants. Inspection requires connectors:read; updating requires connectors:manage and connector administration. Updates preserve the grant ID, publication references and readiness. They take effect on the API without connector configuration acknowledgement.

Both limits are required: queued runs must be 1–10000 and outstanding assignments 1–1024. Counts cover all publications and models using the grant, including candidate trials. A decrease can be below current usage: accepted queued runs remain, and assigned work continues through preparation, scheduler waiting, execution and result collection/upload. At the new queued limit, fresh submissions return HTTP 429; at the assignment limit, fresh claims return HTTP 204. Capacity becomes available when the corresponding usage falls below its limit. Uncertain work retains capacity; terminal reporting frees assignment capacity, and pending durable delivery keeps its independent evidence and obligations. Context, connector and runtime worker limits still apply separately.

A changed update advances the grant generation and records previous/new limits in connector audit history. An unchanged update at the current generation does neither. Stale generations return 409, including an identical retry after a committed update whose response was lost. Inspect the grant and connector audit before deliberately sending another update; the CLI never refreshes the generation or retries automatically. Revocation also advances the generation and remains permanent. Revoked grants and archived connectors/contexts cannot be updated. Limits on a draining context can change while its cutover policy remains in force.

Replace and retire an execution context

Use this workflow when changing a host, execution account, backend or scratch policy, or decommissioning an unused context. The operational lifecycle is active → draining → retired. Read-only inspection and planning require connector management membership and connectors:read; changes additionally need connectors:manage. Connector authority exposes operational dependencies and responsible roles without granting access to scientific inputs, outputs or logs.

Start by inspecting the source. Once a proposed replacement is registered on the same connector, or on another connector you also administer, include its ID to inspect the handoff:

enumeray connector context get lab-connector lab-cpu
enumeray connector context replacement-plan lab-connector lab-cpu --replacement lab-cpu-next
enumeray connector context replacement-plan lab-connector lab-cpu --replacement lab-cpu-next \
  --section grants --limit 20
enumeray connector context replacement-plan lab-connector lab-cpu --replacement lab-cpu-next \
  --section runs --limit 20

The plan reports publications, including candidates and historical records, grants, affected work, delivery obligations and actions with responsible roles. Repeat each dependency section with its returned cursor using --cursor; the default section is publications. Planning writes no lifecycle or audit change. Omit --replacement before registering a successor or when decommissioning without one.

Prepare the host, account and software through normal site procedures. Register the replacement with connector context create using a fresh context ID and issue the intended grants with connector grant create. The replacement can belong to another connector you also administer. Each model author must update publication.json with the replacement context and initialize a fresh saved release, wait for installation validation, trial and review it, then activate explicitly through the saved release workflow. Readiness and grants never transfer from the source. Accepted runs retain their original context, publication and installation.

Cutover requires every live current publication to have moved off the source or been explicitly withdrawn by its model administrator. The service checks this again when committing the guarded operation. Inspect the source’s current generation and choose how to handle already accepted queued work:

enumeray connector context get lab-connector lab-cpu
# Replace this placeholder with the source context Generation just displayed:
GENERATION=OBSERVED_GENERATION
enumeray connector context cutover lab-connector lab-cpu --replacement lab-cpu-next \
  --accepted-work drain --generation "$GENERATION" --dry-run
enumeray connector context cutover lab-connector lab-cpu --replacement lab-cpu-next \
  --accepted-work drain --generation "$GENERATION"
enumeray connector context get lab-connector lab-cpu

drain continues eligible claims for previously accepted queued runs. cancel-queued cancels only runs still queued when cutover commits, recording the actual connector administrator and context-cutover reason. Assigned preparation, pending allocations, execution and delivery continue under either policy, subject to ordinary capacity, maintenance, authority, access and revocation controls. Cutover closes fresh grants, publication registration and activation, standard submissions and candidate trials on this context. Sibling contexts continue under their own controls.

Retain the same account, exact request arguments and generation after an interrupted reply. Retry that unchanged operation to recover its receipt; inspect connector audit and context status before deciding on a different operation. A stale changed request conflicts. The CLI never refreshes the supplied generation or retries a mutation automatically. --dry-run returns the observed state, proposed request and effects; --json retains the structured response for scripts.

After cutover, explicitly release obsolete publication configuration, including abandoned candidates. Read the new source generation first:

enumeray connector context get lab-connector lab-cpu
GENERATION=OBSERVED_GENERATION
enumeray connector context release-configurations lab-connector lab-cpu \
  --generation "$GENERATION" --dry-run
enumeray connector context release-configurations lab-connector lab-cpu --generation "$GENERATION"
enumeray connector context get lab-connector lab-cpu

This requests ordinary publication-configuration retirement. Accepted work and delivery keep their references until safe removal; shared installation references stay protected. Keep the original connector running so bounded maintenance and configuration acknowledgement can progress. Copy the returned context generation, then wait:

enumeray connector context wait-retirement lab-connector lab-cpu --generation CURRENT_GENERATION \
  --interval 2 --deadline 3600

Readiness requires no live current publication, queued work, unfinished or uncertain assignment, pending delivery, active upload or capturing/pending archive. All source publication configuration must be retired and the original connector must acknowledge the generation that removed its final reference. A disconnected connector or elapsed time supplies no such evidence. The wait observes only; generation changes, leaving draining or reaching the deadline stop it without changing the context.

When retirement_ready is true, preview and apply final retirement with that same observed context generation:

enumeray connector context retire lab-connector lab-cpu --generation CURRENT_GENERATION --dry-run
enumeray connector context retire lab-connector lab-cpu --generation CURRENT_GENERATION
enumeray connector context get lab-connector lab-cpu

Final retirement permanently archives the context while retaining its ID, definition, grants, publications, provenance, revocations and transition receipts. Before final retirement, connector context reopen lab-connector lab-cpu --generation CURRENT_GENERATION can restore future eligibility; preview with --dry-run first. Reopening does not undo queued cancellations, configuration-retirement requests, permanent revocations or author activation decisions.

Context retirement readiness is separate from whole-connector maintenance completion and local filesystem cleanup. Inspect connector_work, connector diagnostics and the original journal before stopping a connector shared with other contexts. Use local inspection and bounded cleanup for unresolved host evidence. Server retirement grants no permission to remove protected releases, journals, replay evidence or provenance.

Drain for maintenance

Read the connector generation, inspect the proposed effects, then enter maintenance:

enumeray connector get lab-connector
enumeray connector maintenance lab-connector --generation 3 --dry-run
enumeray connector maintenance lab-connector --generation 3
enumeray connector diagnostics lab-connector
enumeray connector runs lab-connector

Use 3 only when the read reports it. Both preview and apply inspect the same generation; a stale value stops the command. Apply prints the effects and observed counts before changing state. --json emits the preview or mutation receipt without progress messages. Counts can change as work proceeds; they are observations, not a promise that work has stopped. A lost reply requires inspecting diagnostics and audit before another change; the CLI never refreshes the guard or retries the write.

Maintenance sets only draining. It stops new acceptance and claims, retains queued runs for resume, and permits already assigned preparation, pending process/Slurm allocations, execution and result delivery to finish. User cancellation and every independent authority or permanent revocation still apply. Label/capacity changes use connector configure. Existing connector update --file scripts may use {"draining":true,"expected_generation":3} with the same maintenance semantics.

Copy the returned post-maintenance generation into the wait command:

enumeray connector wait-maintenance lab-connector --generation 4 \
  --interval 2 --deadline 3600

4 is illustrative. The wait completes when unfinished assigned work and delivery obligations have finished. Retained queued work can remain nonzero. A terminal scientific result alone does not finish delivery: the original connector records a durable delivery receipt after its required result and log/archive obligations finish. Active uploads and capturing/pending archives also block completion. Unconfirmed contact never proves a job ended; reported phases describe the last service evidence. Configuration acknowledgement lag alone does not block the wait.

A generation change or disabled maintenance stops the wait. Deadline expiry leaves maintenance enabled and work retained; continue investigating the same connector. If the original journal is unrecoverable, coordinate delivery recovery with Enumeray support instead of treating terminal science or elapsed time as delivered results. After a successful wait, stop the original service:

sudo systemctl stop enumeray-connector
sudo systemctl is-active enumeray-connector
sudo -u enumeray-connect /usr/local/bin/enumeray connector inspect \
  --journal /var/lib/enumeray-connect/journal

SIGINT/SIGTERM stop claims and allow a 30-second connector drain by default. A 40-second service-manager stop deadline allows that drain to complete. Expiry leaves evidence for restart; it is not proof that backend jobs stopped.

After maintenance, start the same service and inspect it. Read the current generation before previewing and applying resume:

sudo systemctl start enumeray-connector
enumeray connector diagnostics lab-connector
enumeray connector resume lab-connector --generation 4 --dry-run
enumeray connector resume lab-connector --generation 4
enumeray connector diagnostics lab-connector

Use the actual observed generation. Resume clears only maintenance; it does not restore withdrawn connector authority or independently revoked access.

Withdraw or restore connector authority

Use the separate reversible authority control when this connector must stop launching work, including assignments that have not launched yet:

enumeray connector diagnostics lab-connector
enumeray connector withdraw-authority lab-connector --generation 4 --dry-run
enumeray connector withdraw-authority lab-connector --generation 4

Authority withdrawal blocks new acceptance, claims and prelaunch execution and requests cancellation of pending allocations. Already running backend execution and result delivery can finish. The original backend identity and journal still govern cancellation and recovery; a pending cancellation race can reveal that the original job started. Maintenance remains unchanged.

To lift this particular restriction, read the new generation, then preview and apply connector restore-authority lab-connector --generation CURRENT_GENERATION. Use --dry-run for the preview. Restoration preserves maintenance and does not undo credential revocation, account suspension, model access blocks, compute-grant loss, or permanent publication/installation revocation. Cancellation already requested or confirmed remains in force. Contact Enumeray support to permanently withdraw an installation shared by several publications.

Rotate the connector credential

Retry a lost host-enrollment reply with the same enrollment file and durable identity directory first. A committed enrollment can replay its receipt even after the original enrollment deadline, provided its credential is still active. If an unused enrollment expires, request another into a new output file; an identity that never enrolled can use it without changing its directory. A creation replay never recovers an enrollment secret. See the creation recovery steps before requesting replacement after an uncertain creation or local save.

Use owner-issued enrollment when replacing credentials. Obtain a fresh enrollment immediately before use, stop the old daemon, and transfer the file privately to the connector host:

enumeray connector enrollment lab-connector --output rotated-enrollment.json
sudo systemctl stop enumeray-connector
sudo install -d -m 0700 -o enumeray-connect -g enumeray-connect \
  /var/lib/enumeray-connect/credentials-next
sudo install -m 0600 -o enumeray-connect -g enumeray-connect \
  rotated-enrollment.json /var/lib/enumeray-connect/rotated-enrollment.json
sudo -u enumeray-connect /usr/local/bin/enumeray connector enroll \
  --directory /var/lib/enumeray-connect/credentials-next \
  --api-base-url https://enumeray.example.org \
  --enrollment-file /var/lib/enumeray-connect/rotated-enrollment.json

Enrollment uses a new credential directory and successful enrollment revokes the previous credential. Retry an interrupted operation with this same new directory. Change the service’s --token-file to /var/lib/enumeray-connect/credentials-next/token; retain the original --journal /var/lib/enumeray-connect/journal. Then:

sudo systemctl daemon-reload
sudo systemctl start enumeray-connector
enumeray connector diagnostics lab-connector

Explicit credential revocation enables maintenance and withdraws connector authority. Replacement enrollment preserves both controls; inspect diagnostics and lift each restriction deliberately when the host is ready. Planned credential rotation alone preserves the existing controls.

Deleting a journal or reusing the old enrollment identity is not credential rotation. If operator credential intervention is needed, coordinate with the service administrator and save any one-time token privately.

Recover a stopped connector

A journal contains launch intent, the original backend identity, scientific completion evidence and pending delivery. Reuse that journal after a crash. With exclusive-local, the same unshared, uncloned volume and its process lock can prove that the old owner stopped and recover the API session automatically. That proves connector ownership, not scientific completion.

In manual mode, first stop and verify every prior connector process, including processes on another host or volume clone. With that evidence, reconcile ownership without claiming or launching:

sudo -u enumeray-connect /usr/local/bin/enumeray connector reconcile \
  --journal /var/lib/enumeray-connect/journal \
  --token-file /var/lib/enumeray-connect/credentials/token \
  --confirm-prior-process-stopped

This retrieves existing API work and reconciles the session. It does not claim or launch another job. Resume the daemon with the same API origin, connector ID, journal, private map and token; for an unclosed manual session, add the assertion to that recovery invocation only:

sudo -u enumeray-connect /usr/local/bin/enumeray connector run \
  --journal /var/lib/enumeray-connect/journal \
  --api-base-url https://enumeray.example.org --connector-id lab-connector \
  --token-file /var/lib/enumeray-connect/credentials/token \
  --max-workers 4 --recovery-mode manual --confirm-prior-process-stopped

Omit the assertion from ordinary unattended starts. After a possible launch, recovery observes/reconciles the original job and resumes delivery from retained completion. It never authorizes resubmission. Queue absence, lost SSH replies, missing accounting or a dead connector cannot establish that a job never started. For unresolved backend evidence, retain the journal and ask the site/service operator to investigate the original process or Slurm cluster/job identity.

Reclaim completed run data

The service checks completed run data every 30 seconds by default. Its completed retention default is zero seconds: eligible staging, workspaces, transfer copies, outputs and logs are reclaimed after confirmed execution/allocation release and final delivery. Use --completed-retention-seconds 604800 to retain eligible local copies for seven days, or change the cadence with --cleanup-interval-seconds. These options do not impose a scientific runtime deadline.

Cleanup resumes a durable queue of unfinished runs. Newly archived assignments are queued before leaving the active journal; blocked and partially cleaned runs stay queued for retry. The first applying passes after an upgrade discover legacy archives in bounded batches. Once discovery finishes, ordinary passes work from the pending queue as permanent completed history grows.

Cleanup preserves the original archived assignment, result, provenance and replay-prevention evidence. Published object versions remain in service storage. Active or uncertain execution, pending delivery, live transfer ownership and unavailable compute hosts remain blocked; old age or lost contact cannot make them eligible. Installed model releases and shared/site-owned files need separate connector management.

Permanent per-run cleanup checkpoints retain delivery authority and progress. The complete installation-protection inventory stays separate and is checked again for each deletion attempt. Older checkpoints containing a copied inventory are compacted when revisited. Preserve the entire private journal, including the cleanup queue, when moving or restoring connector state.

Deletion also needs a saved protection inventory and a service that reserves run directories against new installations. Older services without that capability retain cleanup candidates while scientific execution and delivery continue. Missing or uncertain protection evidence, including manual/older cache entries without known installation metadata, also blocks deletion. Known existing releases inside run directories and canonical aliases observed during validation stay permanently protected even when their publication configuration is retired. Unknown or unseeded protection on any compute host blocks payload deletion across the connector. Configuration reconciliation retains affected cache references; successfully protected bindings can still be pruned independently. Ordinary validation and accepted-work recovery continue. An overlapping protected release or executable/batch file blocks cleanup of the whole run. A broad ancestor release directory alone does not block unrelated run scratch.

New release directories and declared executable/batch-file paths must stay outside work_root/UUID and its descendants for every registered context. Registration rejects those paths with reserved_run_installation_path. This reserves only the UUID run namespace: broad installation/work roots may overlap, and a separate work_root/releases/r1 path can still satisfy the normal release rules.

inspect and cleanup --dry-run read a snapshot without taking the journal’s process lock or creating lock files, so they can run while the service is active. Relative journal paths are also supported. Inspect the dry-run report first, then stop the connector before applying a bounded manual pass:

sudo -u enumeray-connect /usr/local/bin/enumeray connector cleanup \
  --journal /var/lib/enumeray-connect/journal --dry-run --limit 8
sudo systemctl stop enumeray-connector
sudo -u enumeray-connect /usr/local/bin/enumeray connector cleanup \
  --journal /var/lib/enumeray-connect/journal --limit 8
sudo systemctl start enumeray-connector

The report identifies retained, eligible and blocked data. Repeat bounded passes as needed. Interrupted cleanup resumes from durable progress against the original launch identity; it never retries scientific execution. Preserve the journal and compute replay records when a path or identity conflicts.

Old publication configuration is retired explicitly by the model administrator using the retirement workflow, or by the connector owner through configuration release on a draining context. Current publications, retained candidates and accepted work continue to keep their installation references. Connector reconciliation drops an installation cache entry only after no required publication or local recoverable work uses it.

Retain unpublished outputs and optional raw logs

Finalized unpublished deliverables remain indefinitely by default. An explicit --unpublished-retention-seconds 604800 gives unpublished results a seven-day publication deadline. Expiry can become output_retention_expired while preserving scientific evidence; expiry alone does not authorize deleting the files. This publication deadline is independent of completed-run cleanup retention.

To enable bounded raw log archives, save log-policy.json:

{
  "segment_bytes":65536,
  "retained_segments":16,
  "archive_enabled":true,
  "archive_source_bytes":16777216,
  "archive_compressed_bytes":52428800,
  "archive_publication_seconds":86400
}

Add --log-policy-file /var/lib/enumeray-connect/log-policy.json to run and restart through the drain procedure. The policy is pinned per launch; recovered work keeps its original policy. Archives may be partial when retention fills. Enumeray never truncates active slurm.log; run-owned copies become eligible for completed-run cleanup after final delivery. Upstream/site-owned Slurm logs need site retention.

For a stopped one-off connector, publish retained archives without executing work:

sudo -u enumeray-connect /usr/local/bin/enumeray connector publish-log-archives \
  --journal /var/lib/enumeray-connect/journal \
  --token-file /var/lib/enumeray-connect/credentials/token
enumeray run log-archive RUN_UUID
enumeray artifact download ARCHIVE_UUID --output logs.tar.gz

The same prior-owner recovery assertion applies if a session is unclosed. Archive publication has a separate deadline and cannot change the scientific result. Contact Enumeray support about service-side archive retention.

Revoke a grant and hand off ownership

enumeray connector grant list lab-connector
enumeray connector grant revoke lab-connector GRANT_UUID
enumeray connector member list lab-connector
enumeray connector audit lab-connector

Replace GRANT_UUID with the grant’s returned ID. Revocation blocks future dispatch, stops prelaunch work when observed and requests pending-only Slurm cancellation. Jobs already executing drain and retain collectible results. User cancellation separately requests stopping running work.

To add or remove a co-admin, read connector get and supply the current ownership generation (ownership_generation, separate from connector configuration generation). Re-read between mutations:

enumeray connector member add lab-connector --user COADMIN_UUID --generation 1
enumeray connector member remove lab-connector --user COADMIN_UUID --generation 2
enumeray connector ownership transfer lab-connector --to NEW_OWNER_UUID --generation 3

The generations are illustrative. A transfer creates a pending handoff; the recipient accepts using its returned transfer ID under their own login:

enumeray connector ownership accept lab-connector TRANSFER_ID

Share the connector/context IDs and operational ownership instructions with the recipient. Co-admin membership applies only to this connector; repeat an explicit membership change for each additional connector a colleague should administer. Ownership transfer does not create, substitute or revoke context grants, and the previous owner loses connector administration unless still an explicit co-admin. CLI ownership transfer does not move host accounts, private keys, journals or service-manager configuration. Preserve those separately and inspect members and audit after acceptance.

Coordinate backups and restoration with Enumeray support. Preserve the original journal, installations, credentials and backend evidence before restoring a connector or reopening admission.

Permanently archive a connector

Use maintenance for a temporary pause. Archiving is an owner-only, permanent retirement of this connector’s authority; it does not stop its host process or delete contexts, grants, journals, publications or audit evidence. Inspect the current ownership_generation before applying it:

enumeray connector get lab-connector
enumeray connector archive lab-connector --generation OWNERSHIP_GENERATION
enumeray connector member list lab-connector
enumeray connector audit lab-connector

Substitute the observed ownership generation. Fresh registration, enrollment, grants, admission, claims and prelaunch execution stop. Queued work keeps its original destination and remains blocked. Assigned work that has not launched receives pending cancellation; already executing work can finish and deliver its results and logs. A separate user cancellation can request stopping running work. The original connector must remain available for reconciliation and delivery; service support can recover credentials for an already enrolled identity. Resume or authority restoration cannot undo the archive. Retain all host evidence until the usual delivery and cleanup conditions hold.