Execution contexts and backend setup

A connector owner creates the execution context. It fixes where releases may be installed, where jobs may write, which account executes them and which backend owns their lifecycle. Authors select a context and installed release when publishing; run inputs never select a host, key or executable. Complete connector enrollment first.

Driver / connection	Choose when	Complete manifest
process / direct	The wrapper runs in the foreground on the connector host	Download
process / ssh	A gateway reaches a separate execution host	Download
slurm / direct	A connector reaches the scheduler locally on a submission host	Download
slurm / ssh	A gateway reaches a Slurm submission host	Download

The driver tracks the scientific job; the connection reaches its runner. A process wrapper must retain the whole job in the foreground. Modules, Python environments and containers can be initialized by the wrapper. A wrapper that submits sbatch and exits needs the Slurm driver to track the allocation correctly.

For trusted same-user experiments, use the development manifest and development walkthrough. Managed contexts always require separate direct execution identities, including when the connector permits development contexts. Changing a mode requires a new context ID.

Read the context fields

All four downloads are version-1 execution-context manifests containing spec:

Field	What to put there
id	Unique context ID, such as lab-cpu; authors use this ID
connector_id	Enrolled connector selected by this manifest
definition.execution_mode	managed by default; development requires process/direct and explicit host startup consent
definition.driver	Process configuration with explicit scratch, or Slurm configuration shown below
definition.connection	Approved direct launcher or SSH target
definition.installation_roots	Absolute roots for readable immutable releases
definition.work_root	Absolute execution-account directory for private job state and staged files
definition.max_outstanding_assignments	Admission bound including completed work awaiting delivery
metadata.location	Location displayed to model consumers
metadata.disclosure	Connector owner’s explanation of who can access submitted data

Paths refer to the execution host, including for SSH. Create directories and install binaries before registration. The work root must be private to the approved execution account and distinct from the connector’s journal. Keep releases immutable and accessible for accepted work after publishing updates.

Process scratch policy

New process contexts require definition.driver.scratch. These limits are chosen by the connector owner and retained with execution records; authors and run inputs cannot override them. Context execution settings remain immutable. To change limits, register a replacement context and publish against it.

Field	Meaning and allowed values
max_bytes	Positive total logical file bytes in the workspace, including staged inputs, outputs, intermediates and caches
max_entries	1–1,000,000 files, directories and links; includes worker-created directories and request metadata
max_depth	1–64 directory levels below the workspace root
scan_interval_ms	100–60,000 ms between completed periodic scans
cache	per_run provisions .enumeray/cache and sets XDG_CACHE_HOME; none provides no managed cache
symlinks	forbid, or workspace_only for existing targets resolving within the same workspace

The example below permits 1 GiB, 10,000 entries and depth 32, with a one-second scan interval. These are example settings, not defaults. Include enough headroom for temporary files as well as input and result files. Scratch limits do not increase artifact upload/download limits or log retention.

Configured process jobs use .enumeray/tmp for TMPDIR, TMP and TEMP; these variables and XDG_CACHE_HOME cannot be supplied in configured process installation settings. HOME remains the workspace. A per-run cache is reclaimed with its run, never shared across jobs. cache: none still allows ordinary scratch writes and libraries using workspace HOME; it does not prevent every library cache.

Quota scans yield between bounded chunks so process observation and cancellation continue during large traversals. Checks occur before launch, during execution and after termination. Enforcement is sampled, so transient overshoot between checks is possible; use operator-managed filesystem quotas when hard limits are required. Individual filesystem operations can still block on slow storage. Regular-file sizes are counted conservatively per entry, including hard links; symlink targets are not scanned again. Missing, cyclic and external links fail. Published inputs and outputs retain their stricter no-symlink and checksum rules.

Run failures distinguish scratch_bytes, scratch_entries, scratch_depth, scratch_symlink, scratch_entry_type and scratch_scan_failed. Pre-launch violations appear in retained startup diagnostics with the exhausted limit and observed/configured values. Scanning checks workspace policy; it does not provide an operating-system sandbox. External dataset links and shared caches are outside this policy.

Existing legacy contexts and retained jobs without scratch keep their original limits and environment behavior. Exact registration retries remain valid. Update API, connector and remote runner binaries together before using new policies. Slurm retains scheduler/site-owned scratch and inherited file-size limits.

Direct access

The quickstart provides account creation, a sudoers rule and complete process/direct JSON. Its launcher is an argv array:

{
  "kind": "direct",
  "launcher": ["/usr/bin/sudo", "-n", "-u", "model-runner", "--",
    "/usr/local/bin/enumeray"]
}

The connector appends compute-runner ACTION --directory PATH and sends a bounded JSON request on stdin. The launcher must run it as the approved execution account, preserve its exit status and reserve stdout for the JSON response. Print launcher diagnostics on stderr. Validation rejects the connector’s own UID as the execution identity. The launcher must be site-controlled; models and run inputs cannot replace it with arbitrary shell commands.

SSH access: complete configuration

Save this as process-ssh-context.json, replacing the target and paths:

{
  "manifest_version": 1,
  "kind": "execution-context",
  "spec": {
    "id": "process-ssh",
    "connector_id": "lab-connector",
    "definition": {
      "driver": {
        "kind": "process",
        "scratch": {
          "max_bytes": 1073741824,
          "max_entries": 10000,
          "max_depth": 32,
          "scan_interval_ms": 1000,
          "cache": "per_run",
          "symlinks": "forbid"
        }
      },
      "installation_roots": ["/opt/enumeray/models"],
      "work_root": "/var/lib/enumeray-model-work",
      "max_outstanding_assignments": 4,
      "connection": {
        "kind": "ssh",
        "host": "login.cluster.example",
        "user": "model-runner",
        "port": 22,
        "credential_reference": "cluster-key",
        "runner_path": "/usr/local/bin/enumeray"
      }
    },
    "metadata": {
      "location": "Research cluster",
      "disclosure": "Cluster administrators and the model execution account can access submitted data."
    }
  }
}

On the connector host, save private credentials.json containing:

{"ssh":{"cluster-key":"/var/lib/enumeray-connect/.ssh/id_ed25519"}}

Add --credentials-file /var/lib/enumeray-connect/credentials.json to the service’s run command. The JSON and private key must be readable only by the connector account. Install the public key in the remote execution account’s authorized_keys, enable SFTP, and provision the remote host’s verified host key in the connector account’s ~/.ssh/known_hosts through your site’s trusted route. The connector uses batch authentication and strict host-key verification.

Host/account	Required SSH material
Connector host, connector OS account	Private credential map and key; verified remote host key in this account’s ~/.ssh/known_hosts
Remote host, execution account	Matching public key in ~/.ssh/authorized_keys; SSH shell, SFTP and installed runner

These are separate accounts and trust directions. Installing the host key only for the remote execution account does not establish the connector’s host trust.

From the connector account, test the same access policy before publication:

ssh -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes \
  -i /var/lib/enumeray-connect/.ssh/id_ed25519 -p 22 \
  model-runner@login.cluster.example \
  /usr/local/bin/enumeray compute-runner --help
sftp -o BatchMode=yes -o IdentitiesOnly=yes -o StrictHostKeyChecking=yes \
  -i /var/lib/enumeray-connect/.ssh/id_ed25519 -P 22 \
  model-runner@login.cluster.example

The first should print runner help and succeed. The second should open an SFTP session; enter bye to close it. A password prompt, unknown host key, missing runner or disabled SFTP must be fixed before registering usable publications. Private keys resolve on the connector host and never enter job JSON. The remote account needs an SSH-compatible login shell; unlike a direct-only execution account, /usr/sbin/nologin will prevent this transport.

Slurm access and the author’s batch script

For Slurm/direct, start with its full manifest. For Slurm/SSH, use the SSH manifest. Both replace the process driver and work root with:

{
  "driver": {
    "kind": "slurm",
    "worker_path": "/usr/local/bin/enumeray"
  },
  "work_root": "/shared/enumeray/jobs"
}

This fragment belongs inside spec.definition; retain the other fields from the complete manifest. The work root must be shared by submission and compute nodes. The binary at worker_path must be available and executable on every allocated node. The submission account needs sbatch, squeue, sacct and scancel, plus working accounting. Test those tools as that account:

command -v sbatch squeue sacct scancel
squeue --me
sacct --starttime today

Empty queue/accounting output can be normal; command errors are not readiness. Qualify the actual cluster and shared filesystem with a disposable model run. Neither a successful probe nor file validation proves scientific correctness.

The author supplies model.sbatch inside the immutable release and sets execution.slurm_script to "model.sbatch" in the editable publication:

#!/bin/bash
#SBATCH --account=research
#SBATCH --partition=compute
#SBATCH --cpus-per-task=1
#SBATCH --mem=256M
#SBATCH --time=00:01:00
set -eu
# Initialize modules, containers or the model environment here.
exec "$1" local-worker --directory "$2"

Replace scheduler directives with site-approved requests. The connector supplies the compute-node worker path as $1 and job directory as $2. Keep the worker in the foreground so it can supervise the wrapper and retain completion. The script owns account, partition, CPUs, memory, GPUs, cluster routing, runtime setup and time limits. Enumeray does not parse those directives or impose a scientific deadline. It sets lifecycle log paths, working directory and a unique recovery name, and retains Slurm’s actual cluster/job identity. The connector pins the readable script’s digest; changing it requires a new release and publication.

Register, inspect and hand off

enumeray connector context create lab-connector --file process-ssh-context.json
enumeray connector context list lab-connector
enumeray connector diagnostics lab-connector

Supply the matching file for your backend. Unsupported fields and combinations fail validation. Give authors the context ID and installation instructions, then grant compute. After publication registration, check its readiness and configuration acknowledgement before author activation.

Update descriptive metadata

Get the context’s current generation with connector context list, then substitute it:

enumeray connector context update lab-connector lab-cpu --location 'Research compute room' \
  --disclosure 'Research-owned machines process submitted model data.' --generation 1
enumeray connector context list lab-connector

The example generation is valid only if the read returned 1. A stale generation returns a conflict; reread after a lost reply before retrying. Location contains 1–200 UTF-8 bytes and disclosure 1–3800; surrounding whitespace and control characters are rejected. Publications snapshot this metadata at registration, alongside Enumeray’s standard model-admin data-access statement. Existing publications and accepted runs retain the original snapshot. Metadata changes grant no access and do not redirect execution. New execution settings require a new context.

For host, execution-account, backend or scratch-policy changes, register a fresh context ID. It may use the existing connector or another connector you also administer; both connectors must authorize the handoff. Follow context replacement and retirement to inspect dependencies, arrange author validation and activation, close new use of the old context and retire it after accepted work and delivery finish. Context definitions and accepted execution destinations remain immutable throughout.

An optional approved shared_workspace can map runner_root to a canonical connector_root mount for collection without an output copy. It must identify the same work root, with no symlink components. Use it after qualifying the shared mount; default direct/SSH transfer works without it. See the connector contract for exact validation rules.