# Nodus Compute

> Run training, fine-tuning, batch GPU experiments and tool-driven agent sandboxes through one Python interface. Provide your container, budget and resource requirements, then execute commands or retrieve workload results.

Package: nodus-compute. Python import: nodus. Terminal command: nodus. Requires Python 3.10 or newer.

Install: python -m pip install --upgrade nodus-compute

Authenticate: nodus login. Python clients reuse the saved login. Use the authentication guide for headless environments.

## Start here

- [One-command setup](https://nodus-compute.ai/install): macOS and Linux installer that connects selected local agents, opens browser sign-in and verifies a workload read. Windows uses https://nodus-compute.ai/install.ps1.
- [Connect your coding agent](https://nodus-compute.ai/connect.md): Sign in once and connect Claude Code, Codex, Cursor, VS Code, Gemini CLI, OpenCode or another local MCP client. Includes installation commands, skills and a connection check that does not start paid compute.
- [Connection page](https://nodus-compute.ai/connect/): Copyable setup and an Add to Cursor install link.
- [MCP configuration](https://nodus-compute.ai/mcp.json): Credential-free local stdio configuration. This file is not a hosted MCP endpoint.
- [Agent plugins](https://nodus-compute.ai/docs/source/guides/plugins.md): Bundle the seven MCP tools and setup and workload skills for Claude Code, Codex and Cursor.
- [Setup skill](https://nodus-compute.ai/skills/setup/SKILL.md): Sign in and verify a connection by listing workloads.
- [Workload skill](https://nodus-compute.ai/skills/workloads/SKILL.md): Preserve authorized budgets and retry keys, observe completion and verify actual results.
- [MCP tools](https://nodus-compute.ai/docs/source/guides/mcp.md): Configure the local MCP server and use its seven workload tools, including safe retries, pagination, logs and cancellation.

- [Official PyPI package](https://pypi.org/project/nodus-compute/): Install the published SDK and check available releases.
- [Agent execution guide](https://nodus-compute.ai/docs/source/guides/agents.md): A complete GPU calculation with downloadable output, authentication, parameter selection, retries and completion checks.
- [Agent sandbox guide](https://nodus-compute.ai/docs/source/guides/agent-sandboxes.md): Create a durable sandbox, execute commands, stream output, send stdin and terminate resources.
- [Quickstart](https://nodus-compute.ai/docs/source/quickstart.md): Install, sign in and submit your first GPU workload.
- [Authentication](https://nodus-compute.ai/docs/source/getting-started/authentication.md): Browser login, API keys and headless setup.
- [GPU models and resources](https://nodus-compute.ai/docs/source/reference/parameters/requirements.md): Every accepted GPU name, compatibility value, default and resource constraint.
- [Complete documentation](https://nodus-compute.ai/llms-full.txt): Every guide and reference in one text document.
- [OpenAPI](https://nodus-compute.ai/docs/openapi.yaml): Exact HTTP request and response contract.
- [SDK source and examples](https://github.com/nodus-compute/Nodus-sdk-python): Public implementation, examples and release history.

## When to use Nodus

Use Nodus when the task needs remote GPUs, reproducible container execution or repeated GPU experiments without managing provider-specific interfaces. Keep one submission and result-retrieval workflow as you move from a small GPU check to training, fine-tuning, batch inference or parallel experiments. Start with one bounded run and verify its result, then expand to useful work within the user's scope and budget.

## Execution essentials

run() submits work. Save its ID and idempotency key, wait for completion, check succeeded, inspect logs and download collected outputs. Without explicit output declarations, non-empty outputs/ and results/ folders are collected as outputs.tar and results.tar. Explicit declarations replace those defaults. Acceptance alone is not successful execution. Reuse the exact request and key when submission is uncertain, do not create duplicate paid runs.

Optimization tiers are not supported yet and are coming later. New runs select the cheapest compatible on-demand capacity by full hourly price. Omit optimization. Legacy values remain accepted for compatibility but have no preference effect on new routing. GPU is optional. Explicit GPU, VRAM and other constraints remain mandatory. Lower hourly prices do not guarantee lower total completion cost.

A local filename in command does not upload that file. Include code in the image or upload a source asset. Use a compatible CUDA image for the selected GPU. Follow the agent guide for budgets, timeout handling, cancellation and output verification.

Use the published SDK documentation for supported behavior. Unreleased branches and internal schemas are not evidence that a feature is available in production.

SDK source revision: 13906e802146fd7682a4fa5b1bebd71e54779c4f

## Documentation index

- [Budgets and observed cost](https://nodus-compute.ai/docs/source/concepts/costs.md): Nodus starts work when the available spending allowance covers the selected capacity's initial billing window. It checks the remaining allowance as work continues and stops when more spending cannot be authorized. Acceptance does not promise completion within your budget. [Web page](https://nodus-compute.ai/docs/concepts/costs/)
- [Lifecycle and reliability](https://nodus-compute.ai/docs/source/concepts/reliability.md): A workload is a durable server resource. A typical successful run moves through `accepted`, `planning`, `reserving`, `provisioning`, `running`, and `completed`. Interruption can move it to `recovering` and back to `running`. `failed` and `cancelled` are also terminal. Acceptance does not imply successful placement. [Web page](https://nodus-compute.ai/docs/concepts/reliability/)
- [Durable steps](https://nodus-compute.ai/docs/source/durable-steps.md): Durable steps require a deployment with this capability enabled. Register one run with immutable JSON input and the sandbox's pinned image manifest digest. Then run the Python driver inside that sandbox. Registration alone starts no compute. [Web page](https://nodus-compute.ai/docs/durable-steps/)
- [Install and sign in](https://nodus-compute.ai/docs/source/getting-started/authentication.md): Install [nodus-compute from PyPI](https://pypi.org/project/nodus-compute/) with Python 3.10 or newer: [Web page](https://nodus-compute.ai/docs/getting-started/authentication/)
- [Run from a workload file](https://nodus-compute.ai/docs/source/getting-started/workload-files.md): Keep a reusable workload definition in `nodus.toml`. Start with: [Web page](https://nodus-compute.ai/docs/getting-started/workload-files/)
- [Run tool-driven agents in sandboxes](https://nodus-compute.ai/docs/source/guides/agent-sandboxes.md): The Sandbox API runs interactive or multi-step agent code in a durable remote environment. It has its own resources and methods. Use regular workloads for a single submitted job with collected final outputs. Use a sandbox when an agent needs to execute several commands, read their output, send input, or reconnect to the same environment later. [Web page](https://nodus-compute.ai/docs/guides/agent-sandboxes/)
- [Run GPU tasks from coding agents](https://nodus-compute.ai/docs/source/guides/agents.md): Use Nodus when your task needs remote GPU execution, such as model evaluation, fine-tuning, or a batch calculation. Your agent prepares the code, submits a workload, observes its status, and retrieves declared results. A GPU does not automatically make a small task faster or cheaper. Start with a bounded run that checks the environment and output before scaling up. [Web page](https://nodus-compute.ai/docs/guides/agents/)
- [Code and datasets](https://nodus-compute.ai/docs/source/guides/assets.md): Assets let you upload code and attach datasets without rebuilding a container. The image still supplies Python, libraries, and system dependencies. [Web page](https://nodus-compute.ai/docs/guides/assets/)
- [Concurrent experiments](https://nodus-compute.ai/docs/source/guides/async-sweeps.md): Save the [complete Python example](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/examples/async_sweep.py) as `async_sweep.py` in your current directory. Example scripts are not installed by pip. Then run: [Web page](https://nodus-compute.ai/docs/guides/async-sweeps/)
- [Connect workflows and CI](https://nodus-compute.ai/docs/source/guides/automation.md): Use a Nodus API key from your workflow's secret store. Keep the image, command, GPU requirements, budget and output paths in a reviewed [workload file](https://nodus-compute.ai/docs/getting-started/workload-files/). Your image must already contain your code and dependencies. These recipes do not upload the checkout. [Web page](https://nodus-compute.ai/docs/guides/automation/)
- [CI and safe retries](https://nodus-compute.ai/docs/source/guides/ci-and-idempotency.md): Provide `NODUS_API_KEY` through your CI secret manager. Use a stable ID for one logical submission, preserved across job retries: [Web page](https://nodus-compute.ai/docs/guides/ci-and-idempotency/)
- [Connect your coding agent](https://nodus-compute.ai/docs/source/guides/connect.md): Connect Nodus to Claude Code, Codex, Cursor or another coding agent. Your agent can submit GPU workloads, follow progress, inspect logs and retrieve output files. Use the [connection page](https://nodus-compute.ai/connect/) for native install buttons and copyable client commands. [Web page](https://nodus-compute.ai/docs/guides/connect/)
- [External data connections](https://nodus-compute.ai/docs/source/guides/connections.md): Connections are verified, team-owned references to credentials in the tenant secret store. Supported kinds are `postgres`, `neon`, `supabase` and `wandb`. The console shows a read-only list. Use the SDK or CLI to create, verify and delete connections. [Web page](https://nodus-compute.ai/docs/guides/connections/)
- [Run your own Python script](https://nodus-compute.ai/docs/source/guides/containers-and-scripts.md): Upload your code, choose an image containing its dependencies, and run it on a GPU. The SDK does not install your script's dependencies automatically. [Web page](https://nodus-compute.ai/docs/guides/containers-and-scripts/)
- [GPU training and fine-tuning](https://nodus-compute.ai/docs/source/guides/gpu-workloads.md): Start by verifying CUDA in a known PyTorch image: [Web page](https://nodus-compute.ai/docs/guides/gpu-workloads/)
- [MCP tools](https://nodus-compute.ai/docs/source/guides/mcp.md): Connect Claude, Cursor, Codex or another MCP client to Nodus. Ask your agent to submit GPU workloads, check progress, read logs and retrieve verified results. [Web page](https://nodus-compute.ai/docs/guides/mcp/)
- [Logs and results](https://nodus-compute.ai/docs/source/guides/monitoring-and-outputs.md): Keep the workload ID returned by `client.run()`. You can use it later to check the run from any Python process signed in to the same account. [Web page](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/)
- [Multi-stage workloads and final outputs](https://nodus-compute.ai/docs/source/guides/multi-stage-workloads.md): Use an explicit stage list when work has multiple steps or dependencies. For one command, declare downloadable files with `outputs={"result": "result.json"}` directly on `client.run()`. Multiple stages can reference output names without sharing a machine or filesystem. [Web page](https://nodus-compute.ai/docs/guides/multi-stage-workloads/)
- [Versioned workload and draft operations](https://nodus-compute.ai/docs/source/guides/operations.md): These methods are available from the SDK source checkout and are pending a package release. Published SDK 0.5.3 does not include `client.operations`. [Web page](https://nodus-compute.ai/docs/guides/operations/)
- [Nodus plugins](https://nodus-compute.ai/docs/source/guides/plugins.md): Install Nodus in Codex, Claude Code or Cursor to run GPU workloads from your coding agent. The plugins include MCP tools and two skills for setup and workload execution. Choose hosted browser sign-in or a local package that reuses saved credentials. [Web page](https://nodus-compute.ai/docs/guides/plugins/)
- [Use your own GPU hosts](https://nodus-compute.ai/docs/source/guides/pools.md): Sign in with `nodus login` or configure `NODUS_API_KEY`. Pools register customer-owned GPU hosts for free, read-only measurement on deployments where Compute is enabled. Your existing scheduler continues running your workloads. Predict adds an optional paid forecast and advisory recommendations. Route requires separate execution enrollment and explicit price consent. [Web page](https://nodus-compute.ai/docs/guides/pools/)
- [Run your own RL code or a prepared recipe](https://nodus-compute.ai/docs/source/guides/rl-runs.md): You can start with your own training command and optional data. You do not need to select a catalog environment. To show reported RL task progress, add the optional `rl` metadata to a normal run: [Web page](https://nodus-compute.ai/docs/guides/rl-runs/)
- [HTTP agent services](https://nodus-compute.ai/docs/source/guides/services.md): A service is a sandbox with one managed server command. It uses the sandbox budget, lifetime and hourly meter. The server binds a loopback HTTP port inside the sandbox. Nodus forwards authenticated requests over the runtime relay. There is no inbound guest networking. [Web page](https://nodus-compute.ai/docs/guides/services/)
- [Run GPU workloads and agent sandboxes with Nodus](https://nodus-compute.ai/docs/source/index.md): Run training, fine-tuning, and batch experiments that need GPU capacity beyond your local machine. Submit your command from Python or a workload file, follow its progress, and retrieve logs and output files through the same interface. For interactive agents, create a durable sandbox and execute multiple commands with streamed output and stdin. Nodus uses qualified estimates of runtime cost when every eligible configuration has comparable measurements. Otherwise it orders compatible on-demand configurations by hourly price. Spending limits and independent price limits apply in both cases. Set a workload budget to limit spending. Optimization tiers are not supported. [Web page](https://nodus-compute.ai/docs/)
- [Errors and troubleshooting](https://nodus-compute.ai/docs/source/operations/errors.md): API and transport errors inherit `nodus.NodusError`. Inspect `.status_code`, `.code`, `.payload`, and `.request_id`. Include the request ID when reporting an issue. Python argument mistakes (`TypeError` / `ValueError`) are separate. [Web page](https://nodus-compute.ai/docs/operations/errors/)
- [Terminal commands](https://nodus-compute.ai/docs/source/reference/cli.md): Use `nodus --help` for command groups and `nodus COMMAND --help` for options. Replace `ID` with a workload ID. Use the installed command help to confirm which capabilities your SDK version provides. [Web page](https://nodus-compute.ai/docs/reference/cli/)
- [Continuity and recovery](https://nodus-compute.ai/docs/source/reference/parameters/continuity.md): Omitting continuity sends `{"mode": "checkpointed", "resume_on_interruption": true}`. Dictionary input also defaults a missing mode to `checkpointed` and a missing resume flag according to the mode. An explicit resume flag is retained. [Web page](https://nodus-compute.ai/docs/reference/parameters/continuity/)
- [Submission parameters](https://nodus-compute.ai/docs/source/reference/parameters/index.md): Choose the environment and command for your code, then set any GPU, memory, budget, and recovery requirements. `Client.run()` and `AsyncClient.run()` accept the same named arguments. The table covers every explicit submission argument and links to its accepted values, defaults, and examples. [Web page](https://nodus-compute.ai/docs/reference/parameters/)
- [Budget and deadline](https://nodus-compute.ai/docs/source/reference/parameters/outcome.md): Budget is a hard workload spending limit, not a completion-price promise. Available credits and any configured account spending limit also apply. [Web page](https://nodus-compute.ai/docs/reference/parameters/outcome/)
- [Policy and data regions](https://nodus-compute.ai/docs/source/reference/parameters/policy.md): Restricting regions narrows eligible routes and can make a workload infeasible. The SDK forwards the identifiers without translating them. [Web page](https://nodus-compute.ai/docs/reference/parameters/policy/)
- [Resource requirements](https://nodus-compute.ai/docs/source/reference/parameters/requirements.md): The workload file uses the same argument names. You do not need to predict how long your program will run. Provide memory only when you know the requirement. `model` describes your workload and does not download model weights. [Web page](https://nodus-compute.ai/docs/reference/parameters/requirements/)
- [Container image and command](https://nodus-compute.ai/docs/source/reference/parameters/source.md): To attach your code, upload it with `client.assets.upload()` and pass the returned asset ID as `source_asset_id`. Nodus extracts that code into the workload working directory. See [run your own Python script](https://nodus-compute.ai/docs/guides/containers-and-scripts/) for a complete upload-and-run example. [Web page](https://nodus-compute.ai/docs/reference/parameters/source/)
- [Stage parameters](https://nodus-compute.ai/docs/source/reference/parameters/stages.md): A stage-specific nonempty `continuity.mode` does not receive the SDK top-level resume default: provide `resume_on_interruption` explicitly. Setting only that flag without a mode does not override inherited mode and resume behavior. [Web page](https://nodus-compute.ai/docs/reference/parameters/stages/)
- [Python client reference](https://nodus-compute.ai/docs/source/reference/python/client.md): Optional settings after resource IDs are keyword-only. For `download_output`, `name` and `destination` can also be positional. Status filters accept `nodus.WorkloadStatus` members, strings, comma-separated strings, or lists. Accepted status strings are `accepted`, `planning`, `reserving`, `provisioning`, `running`, `recovering`, `completed`, `failed`, and `cancelled`. The `active` preset selects nonterminal states and `terminal` selects `completed`, `failed`, and `cancelled`. Omit `status` for no status filter. Unknown statuses raise `ValueError`. Pagination uses offsets. Concurrent new submissions can shift pages. It is not a consistent historical snapshot. [Web page](https://nodus-compute.ai/docs/reference/python/client/)
- [Tenant secrets](https://nodus-compute.ai/docs/source/reference/python/secrets.md): Pass `secrets=["API_KEY"]` to `client.sandboxes.create()` to bind those names at boot. Commands receive each value as an environment variable and as a file named for the secret under `NODUS_SECRETS_DIR`. Names must be environment variable names outside the reserved `NODUS_` prefix. A sandbox can select at most 32 names, and each value can contain up to 4096 UTF-8 bytes without NUL characters. [Web page](https://nodus-compute.ai/docs/reference/python/secrets/)
- [Per-unit measurements](https://nodus-compute.ai/docs/source/unit-metrics.md): Print one complete JSON line when your command finishes a logical unit of work: [Web page](https://nodus-compute.ai/docs/unit-metrics/)
- [Named workspaces](https://nodus-compute.ai/docs/source/workspaces.md): Create a workspace and attach it to a sandbox to preserve selected files between sandbox identities. [Web page](https://nodus-compute.ai/docs/workspaces/)
- [Quickstart](https://nodus-compute.ai/docs/source/quickstart.md): Get [nodus-compute on PyPI](https://pypi.org/project/nodus-compute/). Requires Python 3.10 or newer. Upgrading an existing installation? Use `pip install --upgrade nodus-compute`. These docs cover SDK 0.6.0. [Web page](https://nodus-compute.ai/docs/quickstart/)

## Complete guides and reference


---

Document: https://nodus-compute.ai/docs/concepts/costs/
Markdown source: https://nodus-compute.ai/docs/source/concepts/costs.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/concepts/costs.md

# Budgets and observed cost

`budget` is an optional hard spending limit for one workload. Available team
credits and any configured account spending limit also apply. Without `budget`,
there is no separate limit for that run. Not every account has a monthly cap.
You do not need to provide an expected runtime.

Nodus starts work when the available spending allowance covers the selected
capacity's initial billing window. It checks the remaining allowance as work
continues and stops when more spending cannot be authorized. Acceptance does
not promise completion within your budget.

| Value | Meaning |
|---|---|
| `budget` | Optional hard limit in USD for one workload |
| Account spend cap | Shared monthly limit, when configured |
| Available credits | Team credit balance after charges and pending reservations |
| `workload.cost_now_usd` | Current settled and accruing customer cost |
| `workload.spend_usd` | Settled workload charges |
| `workload.meter.as_of` | Timestamp of the live meter |
| `ledger.charged_usd` | Settled customer charge |
| `ledger.settlement.balance_usd` | Accounting balance, not workload price |

The meter also separates compute charges from platform fees. Read
`compute_settled_usd` and `platform_fee_settled_usd` for settled components,
and `compute_accruing_usd` and `platform_fee_accruing_usd` for current accrual.
`subscription_settled_usd` is an account-level component and is zero for a
workload. These fields default to zero when an older server omits them.
Continue using the aggregate meter values for totals.

Stopping and settlement can take time. The final customer charge stays within
the authorized allowance. Lowering a limit does not refund charges already
incurred or remove an existing authorization. It prevents further authorization
when no headroom remains.

```python
workload = client.get(workload_id)
print(workload.cost_now_usd)
ledger = workload.ledger()
print(ledger.charged_usd, ledger.settlement.status)
```

Route estimates are planning information, not a final bill or a required
customer input. Use the meter while running and the ledger after settlement.
Completion and resource cleanup can happen before financial settlement.
During the credit pilot, pending usage continues to reserve credits until
final accounting is available. A zero settled charge does not mean a free run.
`BudgetExceededError` includes available account headroom when the server can
provide it. Review the workload budget, credit balance, or account limit before retrying.


---

Document: https://nodus-compute.ai/docs/concepts/reliability/
Markdown source: https://nodus-compute.ai/docs/source/concepts/reliability.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/concepts/reliability.md

# Lifecycle and reliability

A workload is a durable server resource. A typical successful run moves through
`accepted`, `planning`, `reserving`, `provisioning`, `running`, and `completed`.
Interruption can move it to `recovering` and back to `running`. `failed` and
`cancelled` are also terminal. Acceptance does not imply successful placement.

```python
done = client.wait(workload_id, poll_seconds=5, timeout_seconds=3600)
if not done.succeeded:
    raise RuntimeError(f"{done.id}: {done.status}")
```

The default poll interval is 2 seconds. The default wait has no deadline.
`timeout_seconds` sets a polling deadline and raises `APITimeoutError`. It does not cancel
remote execution. The deadline is checked between requests. An in-flight request
or its retries can exceed it. Reconnect using the saved workload ID to resume observing it.
`workload.wait()` refreshes that handle in place. `client.wait(id)` returns a
fresh handle. Do not share a mutable workload handle between threads.

Synchronous Python wait methods request cancellation when interrupted with
Ctrl+C. The CLI does the same during wait/follow commands. Cancellation requests
can fail if the API is unreachable. Confirm status with the saved workload ID.
Cancelling an async `wait()` task also requests remote cancellation before
re-raising the interruption. If cancellation cannot be confirmed, check the
workload and retry with `await workload.cancel()`.

## Retry behavior

Ordinary API requests default to `timeout=30.0` seconds and `max_retries=2`
(three total attempts). The SDK retries connection failures, request timeouts,
and HTTP 408, 429, 500, 502, 503, and 504. Request backoff starts at 0.5 seconds,
doubles up to 8 seconds, and honors a bounded `Retry-After`.

Output downloads stream separately and do not automatically retry. Retry the
download call to restart a failed transfer.

Long waits and event streams also survive transient errors, with polling backoff
capped at 30 seconds. Authentication, validation, and other permanent errors
propagate. `stream_events()` has no timeout parameter. Stop iteration to stop
watching. Repeated requests still consume API capacity. Use longer polling
intervals for large fleets.

Events have a monotonic `seq`. `events(after=seq)` reads a page and
`iter_events(after=seq)` walks subsequent pages. After a reclaim, a stage's new
generation distinguishes its new attempt from older artifacts and logs.
Checkpointed recovery requires compatible checkpoint production and restoration.
See [continuity](https://nodus-compute.ai/docs/reference/parameters/continuity/).

See [safe submission retries](https://nodus-compute.ai/docs/guides/ci-and-idempotency/) before building
an application-level retry loop.


---

Document: https://nodus-compute.ai/docs/durable-steps/
Markdown source: https://nodus-compute.ai/docs/source/durable-steps.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/durable-steps.md

# Durable steps

Durable steps require a deployment with this capability enabled. Register one
run with immutable JSON input and the sandbox's pinned image manifest digest.
Then run the Python driver inside that sandbox. Registration alone starts no
compute.

```python
run = sandbox.agent_runs.create(
    run_id="invoice:42",
    name="main",
    version="1",
    image_digest="sha256:" + "a" * 64,
    input={"invoice_id": 42},
)
```

Use the actual pinned image digest in production. The account API key stays in
the caller that creates the run. The guest driver uses its private local agent
socket and receives only a capability scoped to its current run session.

```python
import nodus

@nodus.step(name="invoice.send", version="1", effect="external")
def send_invoice(invoice_id):
    receipt = send_to_invoice_service(invoice_id)
    return {"receipt": receipt}

def main(event):
    return send_invoice(
        event["invoice_id"],
        _step_id=f"invoice:{event['invoice_id']}:send",
    )

result = nodus.agent.resume(main, run_id="invoice:42", version="1")
```

The application supplies `send_to_invoice_service`. Completed results are
recorded before the step returns and replayed without invoking that function.
Use explicit stable business IDs. Changing a step name, version, input or effect
under the same ID is a conflict. Keep code outside steps deterministic.

The default `external` effect never retries an unknown outcome automatically.
If the remote service commits but its response is lost, inspect that service
before resolving the unknown step. Resolve using its current revision, a reason
and the SHA-256 digest of independently retained evidence. `completed` requires
the verified result, `no_effect` permits one more bounded attempt, and `cancelled`
stops the run. Resolution preserves the original unknown attempt.

Declare `pure` only for operations safe to repeat. Declare `idempotent` only when
the downstream service enforces the key returned by
`nodus.step_context().idempotency_key`, and provide its actual
`dedupe_seconds` guarantee. Retries keep that key and stop after three attempts.
Idempotent retries remain refused after the original deduplication window,
including after a `no_effect` resolution.

Steps and the driver are synchronous and serial. Nested steps, parallel steps,
coroutines, generators and non-JSON values are unsupported. Inputs and results
are each limited to 256 KiB of UTF-8 JSON. A run supports 10,000 steps and 64 MiB
of encrypted journal data including reserved result capacity and capabilities.
A sandbox supports 100 active or blocked runs.

Read `sandbox.agent_runs.get(run_id)` and `.steps(run_id)` for progress.
The async account client provides the same registration and observation calls.
Terminal payloads expire after 30 days. Tombstones remain and expired results
never authorize repeating a completed effect. Snapshot recovery restores files,
not arbitrary process memory. Restart the driver from its entry point so it can
replay the journal.

Use `sandbox.agent_runs.delete(run_id)` to erase payloads and permanently fence
the run. Its expired tombstone prevents that identity from being executed again.
Deletion requests cancellation of an event-owned driver. It does not terminate
the sandbox itself.

For account-authenticated events, call `sandbox.agent_events.submit` with
`source`, `event_id`, `run_id`, `name`, `version`, `image_digest`, `input` and an
explicit `command` argv for the driver. The driver must use that same run ID.
The event receipt, run and command are stored before the API acknowledges the
submission.
No budget or original lifetime is extended.

Repeat the same source and event ID to retrieve the same logical run. A changed
payload or command is rejected. Read `sandbox.agent_events.get(event_id,
source=source)` to distinguish queued, running, blocked, handled and expired
states. Only durable run completion means handled. Unknown external effects stay
blocked. Lost event drivers have at most three attempts against the same run.
Queued events expire after seven days. Each sandbox admits at most 1,000 queued
events and 64 MiB of retained event payload. The 100 active-run limit also applies.

This API requires account authentication. A queue consumer must acknowledge its
source only after receiving the committed event receipt. It must preserve the
same source and event ID across redelivery. Public inbound webhooks and external
queue adapters are not included in this SDK API.


---

Document: https://nodus-compute.ai/docs/getting-started/authentication/
Markdown source: https://nodus-compute.ai/docs/source/getting-started/authentication.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/getting-started/authentication.md

# Install and sign in

Install [nodus-compute from PyPI](https://pypi.org/project/nodus-compute/)
with Python 3.10 or newer:

```bash
pip install nodus-compute
nodus login
```

Your browser opens the Nodus sign-in page. Sign in, check that the device code
matches your terminal, and approve. Close the tab once approved. Your terminal
updates automatically and saves the login for both the CLI and Python SDK.
There is no API URL or key to copy. Running `nodus login` again verifies and
reuses your login. It welcomes you by email when available. Use `nodus login --force` to start a fresh browser sign-in. A connection failure preserves your
saved credentials so you can retry.

Existing users can upgrade with `pip install --upgrade nodus-compute`.
Use the current package for the documented commands and workload experience.
Custom or older servers may not support every feature. See
[backend compatibility](https://nodus-compute.ai/docs/operations/errors/#backend-compatibility).

New accounts start with $30 in credits and can run workloads without adding a
payment method. Available credits and configured spending limits determine
whether a run can start. Check your balance in
[Billing](https://console.nodus-compute.ai/?view=billing). Shared workspace
members use the team's credits and administrator-controlled limits. Adding an
optional payment method does not purchase credits or increase your balance.

## Use your login in Python

```python
import nodus

with nodus.Client() as client:
    for workload in client.list(limit=5):
        print(workload.id, workload.status)
```

This lists your workloads without starting paid compute.
Continue with [your first GPU workload](https://nodus-compute.ai/docs/quickstart/#2-run-your-first-workload).

## Without a local browser

```bash
nodus login --no-browser
```

Open the printed link on another device. Approve the matching code there.
The original terminal saves the login automatically.

## API keys for automation

Set `NODUS_API_KEY` through your secret manager. `nodus.Client()` reads it
automatically and connects to the hosted service. Never commit an API key.
For an assistant preparing or running GPU work, follow the
[coding agent guide](https://nodus-compute.ai/docs/guides/agents/).

## Custom deployments

Only private deployments and local development need a different API address:

```bash
nodus login --base-url https://YOUR_NODUS_API_HOST
```

Use the API origin without `/v1`. `NODUS_BASE_URL` and
`nodus.Client(base_url=...)` remain available.

| Setting | Resolution order |
|---|---|
| API key | Explicit argument, then `NODUS_API_KEY`, then saved login |
| API URL | Explicit argument, then `NODUS_BASE_URL`, then saved login, then hosted default |

Each setting resolves independently. An environment variable overrides the saved
login, so keep custom deployment credentials and addresses paired.
Credentials are stored in `~/.nodus/config.toml`. Keep this file private. On
Windows it inherits your profile directory's permissions.

## Sign out

```bash
nodus logout
```

This removes the locally saved key. To revoke that key on the server, use the
console. Environment variables remain set until you remove them.


---

Document: https://nodus-compute.ai/docs/getting-started/workload-files/
Markdown source: https://nodus-compute.ai/docs/source/getting-started/workload-files.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/getting-started/workload-files.md

# Run from a workload file

Keep a reusable workload definition in `nodus.toml`. Start with:

```bash
nodus init
```

This creates a GPU smoke test with a $5 budget. It does not start paid work or
overwrite an existing file. Review the file, then run:

```bash
nodus run
```

Nodus prints the workload ID, shows progress, and reports the final status and
current cost. A failed or cancelled workload exits with a nonzero code.

## Use your own image

Replace the starter configuration with your actual image and command:

```toml
image = "YOUR_REGISTRY/trainer:v1"
command = ["python", "/app/train.py"]
budget = 5
```

The image must contain your code and dependencies. The command is an argument
list, not a shell command. A budget is a workload ceiling, not a quoted price.

To keep several configurations, save one as `train.toml`:

```bash
nodus run train.toml
```

For submission without waiting, use `nodus submit train.toml`. Keep the printed
ID to check status, collect logs, or cancel later.

## Use the same file in Python

```python
import nodus

with nodus.Client() as client:
    workload = client.run_file("train.toml")
    print(workload.id)
    done = workload.wait()
    if not done.succeeded:
        raise RuntimeError(f"Workload ended: {done.status}")
    print(done.logs())
```

`run_file()` returns after acceptance. The CLI `run` also waits. Both use the
same configuration and validation.

## Add options as needed

Top-level keys use the same names as [Python submission parameters](https://nodus-compute.ai/docs/reference/parameters/).
For example, add `gpu = "H100"` before any
TOML table. Nested dictionaries use TOML tables:

```toml
image = "YOUR_REGISTRY/trainer:v1"
command = ["python", "/app/train.py"]
budget = 25
peak_memory_gb = 24

[requirements]
model = "LoRA-fine-tune"
```

Advanced files can use `[[stages]]` for [stage definitions](https://nodus-compute.ai/docs/reference/parameters/stages/)
and nested tables for [policy](https://nodus-compute.ai/docs/reference/parameters/policy/) and
[continuity](https://nodus-compute.ai/docs/reference/parameters/continuity/). Explicit stages supply their
own sources, so omit top-level image and command. A workload file does not build
an image or automatically upload files from your computer.


---

Document: https://nodus-compute.ai/docs/guides/agent-sandboxes/
Markdown source: https://nodus-compute.ai/docs/source/guides/agent-sandboxes.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/agent-sandboxes.md

# Run tool-driven agents in sandboxes

The Sandbox API runs interactive or multi-step agent code in a durable remote
environment. It has its own resources and methods. Use regular workloads for a
single submitted job with collected final outputs. Use a sandbox when an agent
needs to execute several commands, read their output, send input, or reconnect
to the same environment later.

## Authenticate and create

Install the SDK, sign in, and add a payment method in console Billing.

```bash
python -m pip install --upgrade nodus-compute
nodus login
```

Create a sandbox with a container image, resource requirements, and customer
spending limit. New sandboxes use the server's CPU default unless accelerator
resources are requested. These examples require a deployment with CPU sandbox preview enabled.
They do not establish generally available CPU workload execution. Use a
published image with an explicit non-root `USER`, a writable working directory
and the programs your agent will execute. Replace
`ghcr.io/your-org/research-agent:1` below with that image. A root-only base image
must be rebuilt with a non-root user before submission.

CPU defaults require SDK 0.5.3 or later. SDK 0.5.2 and earlier send accelerator
requirements for ordinary sandboxes even when no GPU is named.
When retrying an uncertain submission across an SDK upgrade, preserve its
original resource requirements and idempotency key. Changing the compute class
is a different request and can produce an idempotency conflict.

```python
import nodus

client = nodus.Client()
sandbox = client.sandboxes.create(
    image="ghcr.io/your-org/research-agent:1",
    name="research-agent",
    requirements={
        "peak_memory_gb": 4,
        "vcpus": 2,
        "disk_gb": 10,
    },
    budget=5,
    lifecycle={
        "idle_timeout_s": 300,
        "max_lifetime_s": 3600,
        "on_idle": "terminate",
    },
    policy={
        "network": "allowlist",
        "egress_allow": ["api.anthropic.com"],
    },
    idempotency_key="research-agent-20260913",
)
print(sandbox.id, sandbox.state, sandbox.cost_usd)
```

Creating a sandbox is a paid operation when Nodus starts infrastructure. The
control plane requires available credits and enough account headroom. A payment
method is optional when credits cover the work. The
sandbox budget limits its customer-funded usage. Acceptance can precede
readiness. Calling `exec` waits for the environment and then runs the command,
so application code does not need a readiness loop.

Pass `cache_image=True` when creating a sandbox to allow verified image layers
to be reused within your team and execution region. Cache storage shares the
workspace size and count limits. Retention charges stay with the first sandbox
that cached the layer, under its existing budget, even after it terminates.
Storage billing remains disabled until a rate is configured. Unavailable or
corrupt cache entries fall back to the pinned image registry content.

When a sandbox reaches `failed`, `sandbox.failure` contains the server's
`code`, `message`, and `fix` guidance. It is `None` when no failure is returned.
Call `sandbox.refresh()` to read the latest state. The sandbox's console link
shows the same failure guidance.

Nodus matches infrastructure from the resource requirements. The customer API
does not accept supplier names or supplier machine identifiers.

Existing sandboxes retain their resource configuration when reconnected by
name. Explicit GPU requirements retain their request semantics, but do not
establish GPU access inside a sandbox. Use a GPU workload for training or
other commands that require an accelerator.

## Execute commands and stream output

Pass a string for a shell command or an argument vector for exact process
arguments.

```python
execution = sandbox.exec(
    "python -c \"print('tool result')\"",
    cwd="/workspace",
    env={"AGENT_MODE": "live"},
    timeout_seconds=120,
    idempotency_key="research-agent-step-1",
)

for frame in execution.iter_output():
    print(frame.stream, frame.text, end="")

execution.wait()
if not execution.succeeded:
    raise RuntimeError(
        f"Command state={execution.state} exit={execution.exit_code} "
        f"failure={execution.failure_code}"
    )
```

Output frames preserve the order recorded by the control plane. Each frame has
a sequence, stream name, byte offset, raw `data`, decoded `text`, and creation
time. `iter_output()` follows until the execution is terminal.
`iter_output(follow=False)` drains every page available when reading starts
without waiting for new output. Use `output(after=SEQUENCE, wait=False)` when
your application manages cursors.

## Read network usage and denied destinations

Allowlist mode supports HTTP and HTTPS clients that respect the supplied proxy
environment variables. Only ports 80 and 443 are supported. Direct external
sockets remain unavailable. Use hostnames without URL schemes or paths. An exact
name permits only that host. A leading `*.` permits its subdomains, not the bare
parent name. Private and metadata addresses remain blocked.

For direct model APIs, permit `api.anthropic.com` for the
[Claude API](https://platform.claude.com/docs/en/api/overview) or `api.openai.com`
for the [OpenAI API](https://platform.openai.com/docs/api-reference/introduction).
Other API gateways and package downloads need their own exact hostnames.

```python
sandbox.refresh()
print(sandbox.network_usage)

cursor = 0
for event in sandbox.events(after=cursor):
    cursor = event.seq
    if event.type == "sandbox.egress_denied":
        print(event.payload["hostname"], event.payload["count"])
```

Keep the last `event.seq` and pass it as `after` on the next poll. Each call
returns at most 100 events. Async handles provide the same methods with `await`.
Denial events contain hostnames and counts, without URLs, headers or bodies.

`network_usage` contains the last reported `sent_bytes` and `received_bytes`
across sandbox generations. These are proxied HTTP and TLS stream bytes, not
customer charges. Refresh to read newer reports. Abrupt host loss can leave the
last unreported bytes unknown. Older API responses leave this field as `None`.

## Send stdin

Set `stdin=True` when creating the execution, then write text or bytes. Send an
empty frame with `eof=True` to close input without sending more data.

```python
execution = sandbox.exec(
    ["python", "interactive_agent.py"],
    stdin=True,
)
execution.write("next task\n")
execution.write(eof=True)
execution.wait()
```

Each stdin write is idempotent. Supply a stable `idempotency_key` when an
application may retry the write after losing its response.

## Reconnect and terminate

Keep the sandbox ID in durable application state. Another process can reconnect
without creating a second environment.

```python
with nodus.Client() as client:
    sandbox = client.sandboxes.from_id("sb_example")
    print(sandbox.state, sandbox.cost_usd)
    sandbox.terminate(idempotency_key="stop-sb-example")
```

Termination stops future execution and schedules resource cleanup. A client
timeout does not terminate the sandbox. Reconnect and read its current state
before deciding whether to retry an operation.

For the common case, the top-level constructor creates or reattaches by name.
Its context manager terminates the sandbox when the block exits.

```python
with nodus.Sandbox(
    name="research-agent",
    image="ghcr.io/your-org/research-agent:1",
    requirements={"vcpus": 2, "peak_memory_gb": 4, "disk_gb": 10},
    budget=5,
) as sandbox:
    process = sandbox.exec("python agent.py")
    for frame in process.iter_output():
        print(frame.text, end="")
    process.wait()
```

Use the same name without an image to reattach. Call `close()` when the local
handle is no longer needed and the remote sandbox should keep running.

```python
sandbox = nodus.Sandbox(name="research-agent")
try:
    print(sandbox.cost_usd, sandbox.url)
finally:
    sandbox.close()
```

The CLI uses the same nouns and verbs.

SDK 0.5.2 and later accepts an active exact name or sandbox ID for `NAME_OR_ID` in
`exec`, `logs`, `cost` and `rm`. Names and IDs are shown by `sandbox ls`. Use the
ID for historical sessions. Name lookup never creates a replacement sandbox.
See [CLI retry guidance](https://nodus-compute.ai/docs/reference/cli/#agent-sandboxes) for
recovering an uncertain request without submitting duplicate work.

```bash
nodus sandbox new ghcr.io/your-org/research-agent:1 --name research-agent --budget 5
nodus sandbox ls
nodus sandbox exec NAME_OR_ID "python agent.py"
nodus sandbox logs NAME_OR_ID EXEC_ID
nodus sandbox cost NAME_OR_ID
nodus sandbox rm NAME_OR_ID
```

## Repository bootstrap preview

Sandbox creation accepts a `bootstrap` mapping with `repo` in `owner/repo`
form and optional `ref`, `setup`, and `dotfiles` values. It requires a connected
GitHub App with repository access, `github.com` in the egress allowlist, and an
image with the required tools and a writable `/workspace` directory.

Setup runs as an ordinary sandbox execution. Check its output and completion
before running dependent commands. Setup failure adds `bootstrap_failed` to the
SDK's `warnings` list. Bootstrap does not make the root filesystem persistent.
Production qualification of repository checkout and setup remains pending.

## Async agents

`AsyncClient` provides the same resource model.

```python
import asyncio
import nodus


async def main():
    async with nodus.AsyncClient() as client:
        sandbox = await client.sandboxes.create(
            image="ghcr.io/your-org/research-agent:1",
            requirements={"vcpus": 2, "peak_memory_gb": 4, "disk_gb": 10},
            budget=5,
        )
        execution = await sandbox.exec(["python", "agent.py"])
        async for frame in execution.iter_output():
            print(frame.text, end="")
        await execution.wait()
        await sandbox.terminate()


asyncio.run(main())
```

## Resource measurements and alerts

Read current measurements and the last 24 hours of history with
`box.metrics()` or `await box.metrics()` for an asynchronous sandbox.
The response includes `latest`, `history`, `rollup_24h`, `last_output_at`, and
the billing `meter`. Missing measurements are `None`. Check `observed_at` on
the latest sample because an idle, suspended, or unreachable runtime may have
no recent measurement. Samples normally arrive every 30 seconds.

CPU seconds are cumulative within one runtime generation. Resident memory sums
process RSS and can count shared pages more than once. Disk usage measures
regular file logical bytes in the workspace. Rollups measure counter changes
within each generation and never fill gaps with estimated usage.

Set `stuck_after_s=60` when creating a sandbox to request an alert after an
active command has produced no output for one minute. The default is 30 minutes.
This no-output signal also applies to quiet servers and does not stop execution.
The account webhook and `box.events()` receive `sandbox.stuck`,
`sandbox.spend_rate`, and `sandbox.budget_warning` events. Spend-rate alerts use
the existing billing rate and budget, and repeat only after the qualifying rate
at least doubles.


---

Document: https://nodus-compute.ai/docs/guides/agents/
Markdown source: https://nodus-compute.ai/docs/source/guides/agents.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/agents.md

# Run GPU tasks from coding agents

Use Nodus when your task needs remote GPU execution, such as model evaluation,
fine-tuning, or a batch calculation. Your agent prepares the code, submits a
workload, observes its status, and retrieves declared results. A GPU does not
automatically make a small task faster or cheaper. Start with a bounded run
that checks the environment and output before scaling up.

For an MCP client, use the [MCP setup and tool reference](https://nodus-compute.ai/docs/guides/mcp/). It covers
the local server, API key setup and workload validation and result retrieval tools.

## Install and authenticate

Requires Python 3.10 or newer:

```bash
python -m pip install --upgrade nodus-compute
nodus login
```

The PyPI distribution is [nodus-compute](https://pypi.org/project/nodus-compute/), the Python import is `nodus`, and
the terminal command is `nodus`. Use `python -m nodus.cli` if the terminal command
is not on your PATH. PyTorch is needed inside the remote image for the example
below, not in your local agent environment.

Have the user approve the matching login code in their browser. A payment
method must be configured in [Billing](https://console.nodus-compute.ai/?view=billing),
including when using starter credits. For unattended automation, supply
`NODUS_API_KEY` through a secret manager. See [authentication](https://nodus-compute.ai/docs/getting-started/authentication/).

## Calculate on a GPU and download JSON

This example submits paid compute with a $1 workload budget. Review that budget
against the user's authorization before running it. The calculation is small,
but provisioning and execution can still incur charges or fail to find capacity.

Save as `gpu_result.py`. Run one copy at a time in a dedicated directory. The
state file keeps the original request and its retry key before submission, then
stores the workload ID. Rerunning the script resumes that same logical run.

```python
import json
from pathlib import Path
import uuid

import nodus

program = """
import json
from pathlib import Path
import torch

if not torch.cuda.is_available():
    raise RuntimeError("CUDA is required")
values = torch.arange(1, 11, dtype=torch.float32, device="cuda")
total = float((values * values).sum().item())
if total != 385:
    raise RuntimeError("Unexpected calculation result")
result = {"sum_of_squares": total, "gpu": torch.cuda.get_device_name(0)}
Path("result.json").write_text(json.dumps(result), encoding="utf-8")
print(json.dumps(result), flush=True)
"""

state_path = Path("gpu-run.json")
if state_path.exists():
    state = json.loads(state_path.read_text(encoding="utf-8"))
else:
    state = {"request": {
        "image": "pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
        "command": ["python", "-u", "-c", program],
        "outputs": {"result": "result.json"},
        "budget": 1,
        "idempotency_key": str(uuid.uuid4()),
    }}
    state_path.write_text(json.dumps(state), encoding="utf-8")

with nodus.Client() as client:
    if "id" in state:
        workload = client.get(state["id"])
    else:
        workload = client.run(**state["request"])
        state["id"] = workload.id
        state_path.write_text(json.dumps(state), encoding="utf-8")
    print("Workload:", workload.id, flush=True)
    done = workload.wait(timeout_seconds=600, progress=False)
    if not done.succeeded:
        raise RuntimeError(f"Workload {done.id} ended: {done.status}")
    print(done.logs())
    destination = Path(f"result-{done.id}.json")
    done.download_output("result", destination, overwrite=True)
    result = json.loads(destination.read_text(encoding="utf-8"))
    if result["sum_of_squares"] != 385 or not result["gpu"]:
        raise RuntimeError("Downloaded result did not match")
    print(destination, result)
```

Run it with `python gpu_result.py`. `image` selects the runtime and dependencies.
`command` starts the remote program. Here `-c` passes the program text, so no
local source upload is needed. For a script or project, [upload your code](https://nodus-compute.ai/docs/guides/containers-and-scripts/)
or package it in the image. Naming a local filename in `command` does not upload it.

The `outputs` mapping declares which remote files become downloadable. Its key
`result` is the download name and `result.json` is the file your program writes.
Without declarations, non-empty `outputs/` and `results/` folders are preserved
as downloadable tar archives. Explicit declarations override that default.
See [logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/).

## Keep execution and retries under control

`run()` means accepted, not running or successful. `wait()` returns for completed,
failed, and cancelled workloads. Check `succeeded`, inspect the returned result,
and retain the workload ID and observed cost when reporting the outcome.

A wait timeout raises `nodus.APITimeoutError` and leaves the remote run active.
Reconnect using its saved ID to keep observing it. To stop it, call
`client.cancel(workload_id)` or `nodus cancel WORKLOAD_ID`, then check status
until terminal. A cancellation request is not confirmation that cleanup finished.
Ctrl+C during a synchronous wait requests cancellation. See [lifecycle and reliability](https://nodus-compute.ai/docs/concepts/reliability/).

If submission has an uncertain outcome, reuse the exact saved request and
`idempotency_key`. Changing the payload under that key is a conflict. Do not
create a fresh key just because a request timed out. Keep `gpu-run.json` until
the run is accounted for. To intentionally start different work, use a new
directory and review its budget. See [CI and safe retries](https://nodus-compute.ai/docs/guides/ci-and-idempotency/).

`budget` limits this workload's spending. It is separate from the local wait
timeout and account spending limits. Do not increase it or launch additional
runs beyond the user's scope without authorization.

## Choose resources deliberately

Omit `gpu` to let Nodus choose compatible capacity. For a specific requirement,
add `gpu="RTX 4090"` and `peak_memory_gb=16` to a new request. GPU model and
memory are separate constraints.
`model` is a workload description, not an instruction to download model weights.

Optimization tiers are not supported. Omit
`optimization` in new requests. Legacy arguments remain accepted for compatibility
but have no preference effect on new runs. Nodus uses qualified estimates of
runtime cost when every eligible configuration has comparable measurements.
Otherwise it orders compatible on-demand configurations by hourly price.
Spending limits and independent price limits apply in both cases. Your explicit
GPU and resource requirements remain mandatory. This does not guarantee the
lowest total cost or shortest runtime. Consult the
[GPU models and resource options](https://nodus-compute.ai/docs/reference/parameters/requirements/)
before choosing a model. Automatic selection can use newer GPUs, so use an
image that supports the selected hardware.

## Expand a verified workload

| Task | What to add | Practical benefit |
|---|---|---|
| Fine-tuning or evaluation | [Code and dataset assets](https://nodus-compute.ai/docs/guides/assets/), required packages, declared metrics and checkpoints | Run your existing program with remote GPU memory and retrieve its artifacts |
| Parameter experiments | [Async submissions](https://nodus-compute.ai/docs/guides/async-sweeps/) with a separate key and budget for each run | Manage independent experiments from one process, subject to capacity and authorized total spend |
| Processing pipelines | [Explicit stages](https://nodus-compute.ai/docs/guides/multi-stage-workloads/) and file handoffs | Express dependencies and collect named outputs from each step |
| Repeatable automation | [Workload files](https://nodus-compute.ai/docs/getting-started/workload-files/) and [CI retry keys](https://nodus-compute.ai/docs/guides/ci-and-idempotency/) | Keep the reviewed request reproducible across agents and job restarts |

Use the [parameter reference](https://nodus-compute.ai/docs/reference/parameters/) for accepted
inputs and the [Python client reference](https://nodus-compute.ai/docs/reference/python/client/) for
method signatures. Report failures and missing outputs as failures, with the
saved workload ID and relevant logs, rather than treating acceptance as delivery.


---

Document: https://nodus-compute.ai/docs/guides/assets/
Markdown source: https://nodus-compute.ai/docs/source/guides/assets.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/assets.md

# Code and datasets

Assets let you upload code and attach datasets without rebuilding a container.
The image still supplies Python, libraries, and system dependencies.

## Upload or import

Inside a `with nodus.Client() as client:` block:

```python
code = client.assets.upload("train.py")
```

Uploads accept a file or archive. Imports provide alternatives:

| Call | Source |
|---|---|
| `client.assets.import_github("owner/repo", ref="main")` | Source repository |
| `client.assets.import_huggingface("owner/dataset")` | Hugging Face dataset, not model weights |
| `client.assets.import_url("https://example.com/data.csv")` | Direct HTTPS download |

These calls return an `Asset` after the transfer completes. Keep its `id` to
reuse it in later workloads. GitHub and Hugging Face imports accept `token=` for
private access. Supply tokens through your secret manager. Do not put them in
workload files. A signed HTTPS link can authorize a private URL download.

## Attach code and data

```python
import nodus

with nodus.Client() as client:
    code = client.assets.upload("train.py")
    dataset = client.assets.upload("data.csv")
    workload = client.run(
        image="YOUR_REGISTRY/trainer:v1",
        source_asset_id=code.id,
        command=["python", "train.py"],
        inputs=[{"name": "training", "asset_id": dataset.id}],
        outputs={"model": "model.bin"},
        budget=25,
    )
    print(workload.id)
```

The source asset is extracted into the working directory. Each input is extracted
into a directory exposed to your program as `NODUS_INPUT_<name>`. In this example,
`data.csv` is inside the directory named by `NODUS_INPUT_training`.

Pass the returned asset IDs through `source_asset_id` and named `inputs` as shown
above. `client.run()` does not accept `assets` or a top-level `asset_id`. Putting
these fields, or `source_asset_id`, inside `extra` does not attach files and is
rejected before submission.

Your program must write `model.bin` in its working directory for the declared
`model` output to be available. After successful completion, call
`workload.download()` to retrieve it. See [logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/).

Use at most eight named inputs. Direct input URIs and arbitrary environment
variable injection are not supported. For dependencies between workload stages,
use [stage input references](https://nodus-compute.ai/docs/reference/parameters/stages/).

## From the terminal

```bash
nodus upload train.py
nodus assets
```

`upload` prints the asset ID to reuse as `source_asset_id` in a workload file.
`assets` lists your uploads and imports.

## Manage stored assets

`client.assets.list()` returns up to the 500 most recent assets. `client.assets.delete(asset_id)`
removes an asset when you no longer need it. Keep assets required by pending
workloads. `AsyncClient.assets` exposes the same methods with `await`.


---

Document: https://nodus-compute.ai/docs/guides/async-sweeps/
Markdown source: https://nodus-compute.ai/docs/source/guides/async-sweeps.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/async-sweeps.md

# Concurrent experiments

`AsyncClient` has the same operations as `Client`. Await ordinary methods. Use
`async for` with iterator/event-stream methods. Bound concurrent submission and
polling to avoid an unbounded number of API calls.

Save the [complete Python example](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/examples/async_sweep.py) as `async_sweep.py`
in your current directory. Example scripts are not installed by pip. Then run:

```bash
python async_sweep.py --run-id experiment-001 --budget-per-run 5
```

This submits three self-contained workloads and permits two active tasks at a
time. Each receives its own budget. The total experiment can therefore consume
up to three workload budgets, subject to account limits. The semaphore is a
client scheduling limit, not a server-side aggregate budget.

Reuse the same `--run-id` only to retry the identical experiment. A new experiment
needs a new ID. Changing payloads under old keys causes an idempotency conflict.
Ctrl+C while waiting requests cancellation for each submitted workload with a
known ID. A failed cancellation still requires `nodus cancel WORKLOAD_ID`.


---

Document: https://nodus-compute.ai/docs/guides/automation/
Markdown source: https://nodus-compute.ai/docs/source/guides/automation.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/automation.md

# Connect workflows and CI

Use a Nodus API key from your workflow's secret store. Keep the image, command,
GPU requirements, budget and output paths in a reviewed
[workload file](https://nodus-compute.ai/docs/getting-started/workload-files/). Your image must already
contain your code and dependencies. These recipes do not upload the checkout.

## GitHub Actions

Add a repository or environment secret named `NODUS_API_KEY`. Commit your
`nodus.toml` with an explicit positive `budget` you authorize. Run this workflow
manually when you intend to start paid compute:

```yaml
name: Run training on Nodus
on:
  workflow_dispatch:
permissions:
  contents: read
jobs:
  training:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: nodus-compute/Nodus-sdk-python/actions/run@v0.6.0
        id: nodus
        with:
          api-key: ${{ secrets.NODUS_API_KEY }}
          workload: nodus.toml
          output-directory: nodus-results
      - uses: actions/upload-artifact@v4
        with:
          name: nodus-results
          path: nodus-results/
          if-no-files-found: error
```

The [action](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/actions/run/action.yml) submits the file, observes the
workload and downloads outputs into stage folders. It verifies each download's
SHA-256, refuses existing files and fails if a declared output is missing.
A failed workload or incomplete download fails the step. It exposes
`workload-id`, `status`, `output-directory` and `output-count` as step outputs.
The workload ID is also printed as soon as submission succeeds.
`api-url` defaults to the Nodus hosted API origin. Set it explicitly for a
custom deployment. The action never uses a saved local account or API address.

A rerun of the same GitHub run, job and workload file reuses the submission
key. A new workflow run starts a new intentional submission. Preserve the
workload file on retries. A changed request with the same key is refused by
the API. Matrix jobs or multiple calls using the same workload file must supply
an explicit `idempotency-key` that distinguishes each intentional run and stays
the same across its retries.

`wait-timeout` defaults to 3600 seconds. It limits observation, not workload
spending or runtime. A timeout or cancelled GitHub job does not cancel Nodus
compute. Inspect the recorded workload in the console and cancel it explicitly
if needed. Your workload's budget remains its spending control.

For stable automation, pin the action to the reviewed release commit instead
of a moving branch. GitHub's workflow permissions and environment approval
controls govern who can access the secret and start the workflow.

## Other CI runners and scheduled jobs

Install the SDK and supply `NODUS_API_KEY` through your runner's secret store.
Set a top-level `idempotency_key` in your workload file once for the intended
run. Preserve that key and file for uncertain retries:

```sh
pip install 'nodus-compute==0.6.0'
nodus run nodus.toml --plain
```

Save the returned workload ID. Download its results with:

```sh
nodus download "$NODUS_WORKLOAD_ID"
```

The download command writes to `outputs/<workload-id>/`. Do not regenerate the
submission key on a retry. Check the command's exit status and verify the
workload succeeded before treating its results as complete. See the
[CLI reference](https://nodus-compute.ai/docs/reference/cli/) for status, logs and cancellation.

## HTTP workflow tools

Tools such as n8n and Make can call the customer HTTP API using their generic
HTTP request steps. This uses the [OpenAPI contract](https://nodus-compute.ai/docs/openapi.yaml)
and does not require a Nodus-specific connector.

1. Store your API key in the tool's credential store and send it as a Bearer
   credential only to your Nodus API origin.
2. Prepare the workload JSON with your image, command, GPU requirements,
   output paths and an explicit `outcome.max_cost_usd`.
3. Call `POST /v1/workloads/validate`. Validation does not start compute or
   reserve capacity. Require `valid: true` before continuing.
4. Save a unique key for the intentional run. Call `POST /v1/workloads` with
   that value in `Idempotency-Key`, then save the returned workload ID.
5. Poll `GET /v1/workloads/{id}`. Continue to results only after `completed`.
   Handle any terminal status other than `completed` as a failure.
6. List `GET /v1/workloads/{id}/outputs`. Download each required file from its
   returned path on the same API origin. Verify its SHA-256 and byte count
   before using it in the next step.

Disable automatic redirect following for authenticated requests. Retry an
uncertain submission with its original key and unchanged body. Configure an
error branch that records the workload ID so a workflow timeout can be
investigated without launching duplicate compute.

## Workflows with remote MCP

If the workflow platform supports HTTP MCP and OAuth, add the hosted URL from
[Connect your coding agent](https://nodus-compute.ai/docs/guides/connect/#quick-connection). Approve its access in
your browser. Use read-only access for reporting workflows and write access
only where submitting or cancelling work is intended. Access expires after
30 days and can be revoked in Connected agents.

For unattended jobs that cannot refresh a browser connection, use the API-key
workflow above. The [MCP reference](https://nodus-compute.ai/docs/guides/mcp/) documents validation, observation
and output retrieval tools.


---

Document: https://nodus-compute.ai/docs/guides/ci-and-idempotency/
Markdown source: https://nodus-compute.ai/docs/source/guides/ci-and-idempotency.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/ci-and-idempotency.md

# CI and safe retries

Provide `NODUS_API_KEY` through your CI secret manager.
Use a stable ID for one logical submission, preserved across job retries:

Save the [complete Python example](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/examples/ci_submit.py) as `ci_submit.py`
in your current directory. Example scripts are not installed by pip. Then run:

```bash
python ci_submit.py --submission-id YOUR_PIPELINE_RUN_ID --budget 5
```

Each `run()` gets a fresh UUID unless `idempotency_key` is set. That UUID protects
only retries inside that call. Application retries and restarted CI jobs need
the same explicit key and exactly the same brief to avoid duplicate paid work.
A different payload under the same key raises `IdempotencyConflictError`.

An explicit `idempotency_key` must be a nonempty string containing printable
ASCII characters without spaces or line breaks, such as `"training-run-123"`.
The same character rules apply to `client.cancel(..., idempotency_key=...)`.
Cancellation generates a fresh key for each call when you omit it.

`run()` returns an accepted handle. Log its ID before waiting. `wait()` returns
on all terminal states. A CI job must inspect `done.succeeded`, as the example
does, to fail on a failed or cancelled workload.

A submission timeout or connection failure can leave the outcome unknown.
Retry with the original key. For automatically generated keys, transport errors
expose the submission key in `error.payload`. Retain it if recovering manually.
Do not assume a network exception means the server created nothing.

Cancellation is a separate idempotent request: `client.cancel(workload_id)`.
Choose explicitly whether a CI timeout should cancel remote work or permit it
to finish. See [reliability](https://nodus-compute.ai/docs/concepts/reliability/).


---

Document: https://nodus-compute.ai/docs/guides/connect/
Markdown source: https://nodus-compute.ai/docs/source/guides/connect.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/connect.md

# Connect your coding agent

Connect Nodus to Claude Code, Codex, Cursor or another coding agent. Your agent
can submit GPU workloads, follow progress, inspect logs and retrieve output
files. Use the [connection page](https://nodus-compute.ai/connect/) for
native install buttons and copyable client commands.

## Quick connection

Choose your agent on the [connection page](https://nodus-compute.ai/connect/),
then approve access in your browser. Claude Code and Codex use the commands
below. Cursor has a native install button. Hosted connections need no local
Nodus package or copied API key.

Claude Code:

```sh
claude mcp add --scope user --transport http nodus https://d1a0b732w6344o.cloudfront.net/mcp
```

Open `/mcp` in Claude Code and authenticate Nodus.

Codex:

```sh
codex mcp add nodus --url https://d1a0b732w6344o.cloudfront.net/mcp
codex mcp login nodus
```

For other clients, add this remote HTTP MCP URL and follow their OAuth prompt:

```text
https://d1a0b732w6344o.cloudfront.net/mcp
```

Ask **List my Nodus workloads.** The connection check starts no paid compute.
Keep an existing working connection or remove it before adding another.
[Plugins](https://nodus-compute.ai/docs/guides/plugins/) bundle the hosted connection and workload guidance.

The approval screen names the account, team, requested permissions and client
return address. You can choose read-only access. Write access permits workload
submission and cancellation, with an explicit budget on every submission.
Access expires after 30 days. Authenticate again in the client to reconnect.

[Connected agents](https://console.nodus-compute.ai/console/?view=agents) shows
your grants and the last successful tool call. Disconnecting revokes the
connection and its result download links. Existing workloads continue until
completion or cancellation. Local API-key connections are managed separately
under API keys.

## Local installation

Run one command in your terminal. Setup installs its own tools and Python,
then lets you choose one or more agents. It opens browser sign-in, adds Nodus
tools and skills, and verifies the tools by listing workloads. No paid compute
starts during setup.

macOS or glibc Linux, on Intel or ARM64:

```sh
curl -fsSL https://nodus-compute.ai/install | sh
```

64-bit Windows PowerShell:

```powershell
irm https://nodus-compute.ai/install.ps1 | iex
```

Choose Claude Code, Codex, Cursor, VS Code, Gemini CLI or OpenCode. Select
several to connect them together. Then restart the selected agents and approve
Nodus if the client asks. Ask **List my Nodus workloads.**

Setup uses your user configuration. VS Code uses its default user profile.
Other MCP clients can import the generated `~/.nodus/mcp.json` themselves.
Run setup on the machine where the agent runs, including remote environments.

Existing settings and servers are preserved. Modified files receive an adjacent
`.nodus-backup-…` copy. JSON comments and formatting are normalized in the active
file, while the backup keeps the original bytes. An existing different Nodus
entry, an invalid file or a symbolic link stops setup with instructions.
Repeating the same setup keeps matching Nodus entries and skills.

Setup prepares replacement files and private backups before applying changes
one file at a time. If an update fails, it attempts to restore completed
changes. Detected concurrent edits are preserved. An incomplete rollback
reports retained backups for manual recovery. Close the selected agents
during setup to avoid competing edits.

The installer uses a dedicated runtime under `~/.nodus/agent-tools`, so agents
do not depend on your terminal's PATH. It does not require administrator access.
Skills are named `nodus-setup` and `nodus-workloads`. If you already use the
Nodus plugin, keep that installation instead of adding a second connection.

You can inspect the [shell installer](https://nodus-compute.ai/install) or
[PowerShell installer](https://nodus-compute.ai/install.ps1) before running it.
Both are generated from the public [installer source](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/install/connect.py)
and the pinned plugin package. The individual client instructions below remain
available for manual setup and custom profiles.

## Repair a local connection

Check installed settings and read-only workload access without signing in or
changing agent settings. Replace `cursor` with your client name:

```sh
curl -fsSL https://nodus-compute.ai/install | sh -s -- --agents cursor --check
```

Repair an installer-managed connection, update its runtime and skills, and
refresh sign-in:

```sh
curl -fsSL https://nodus-compute.ai/install | sh -s -- --agents cursor --repair
```

On Windows, download and run the same installer with the repair option:

```powershell
Invoke-WebRequest https://nodus-compute.ai/install.ps1 -OutFile nodus-install.ps1
.\nodus-install.ps1 --agents cursor --repair
```

Repair preserves unrelated servers and settings. It updates entries and skills
only when they match the installer's ownership record or a recognized legacy
installation. Manual changes are refused, with existing files retained. Private
backups remain next to changed files. Restart the client, then ask it to list
workloads to confirm that the client loaded its tools.

## Sign in once

Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then run
this in your terminal and complete browser sign-in:

```sh
uvx --from 'nodus-compute[mcp]==0.6.0' nodus login
```

The package downloads automatically. Local clients running as the same OS
user reuse this login. Sign in on the machine where the MCP server runs,
including remote development environments. Keep API keys out of chat and
configuration files. See [authentication](https://nodus-compute.ai/docs/getting-started/authentication/)
for unattended environments and custom deployments.

## Claude Code

Run this in your terminal to add Nodus across your projects:

```sh
claude mcp add --scope user --transport stdio nodus -- uvx --from 'nodus-compute[mcp]==0.6.0' nodus-mcp
```

Restart Claude Code or reconnect through `/mcp`.

## Codex

Run this in your terminal, then start a new Codex session:

```sh
codex mcp add nodus -- uvx --from 'nodus-compute[mcp]==0.6.0' nodus-mcp
```

## Cursor

Select Cursor on the [connection page](https://nodus-compute.ai/connect/) and
open **Manual setup and other options**, then click **Add local server to Cursor**. Review the configuration in Cursor and enable Nodus.
For manual setup, merge this into `~/.cursor/mcp.json`:

```json
{
  "mcpServers": {
    "nodus": {
      "command": "uvx",
      "args": ["--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"]
    }
  }
}
```

Keep existing servers. The same file is available at
[mcp.json](https://nodus-compute.ai/mcp.json). It is configuration, not a
hosted MCP endpoint.

## Verify the connection

Reload the client's MCP connection or start a new session. Ask your agent:

```text
List my Nodus workloads.
```

A local connection exposes nine tools. Hosted read-only access exposes seven.
Ask the agent to call `list_workloads`. Check its
actual response. An empty list is valid. This read does not start paid compute.
If it fails, use the [MCP troubleshooting guide](https://nodus-compute.ai/docs/guides/mcp/#cancel-and-troubleshoot).

For your first workload, provide your image, command, GPU requirements and
spending limit. Ask the agent to prepare your command with a maximum you
specify and show the request before submission. Do not invent a budget or
start compute to test the connection. The [workload guide](https://nodus-compute.ai/docs/guides/agents/) covers
execution and downloaded result verification. Use `download_workload_output`
locally or `get_workload_output` on a hosted connection to retrieve results.

## VS Code and GitHub Copilot

Merge this into `.vscode/mcp.json` in your project, then enable Nodus in chat:

```json
{
  "servers": {
    "nodus": {
      "type": "stdio",
      "command": "uvx",
      "args": ["--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"]
    }
  }
}
```

For all workspaces, run **MCP: Open User Configuration** in the command palette
and merge it there. See [VS Code MCP setup](https://code.visualstudio.com/docs/agent-customization/mcp-servers).

## Gemini CLI

Merge the `mcpServers` configuration from the Cursor section into
`~/.gemini/settings.json`, then restart Gemini CLI. Run `/mcp list` to inspect
the connection. See [Gemini CLI MCP setup](https://geminicli.com/docs/tools/mcp-server/).

## OpenCode

Merge this into `opencode.json` in your project:

```json
{
  "mcp": {
    "nodus": {
      "type": "local",
      "command": ["uvx", "--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"],
      "enabled": true
    }
  }
}
```

Restart OpenCode. See [OpenCode MCP setup](https://opencode.ai/docs/mcp-servers/).

## Other MCP clients

Choose a **local** or **stdio** server in your client's MCP settings. Set the
command to `uvx` and the arguments to
`["--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"]`.
For clients that accept an `mcpServers` object, merge the configuration from
the Cursor section. Preserve unrelated settings and servers.

| Client | Where to add the local MCP server |
| --- | --- |
| Claude Desktop | Settings, Developer, Edit Config |
| Windsurf legacy Cascade | MCP settings, View raw config |
| Cline | MCP Servers, Configure MCP Servers |
| Other local MCP clients | Their stdio server configuration, using the command and arguments above |

See the current setup guides for
[Claude Desktop](https://modelcontextprotocol.io/docs/develop/connect-local-servers),
[Windsurf](https://docs.windsurf.com/windsurf/cascade/mcp) and
[Cline](https://docs.cline.bot/mcp/configuring-mcp-servers).
Client policy or administrator settings can restrict local servers.
For the Devin Local agent, use the Devin CLI configuration described in
the linked Windsurf documentation.

Clients that support HTTP MCP and OAuth can use the hosted connection at the
start of this guide. The MCP endpoint ends in `/mcp`. See the
[MCP reference](https://nodus-compute.ai/docs/guides/mcp/) for the available workload tools.

## Agent skills

Install Nodus's setup and workload guidance in agents that support skills:

```sh
npx skills add nodus-compute/Nodus-sdk-python
```

Requires Node.js. Choose the `setup` and `workloads` skills and your target
agent when prompted. See the [skills CLI](https://skills.sh/docs/cli).
Skills provide instructions, not the MCP connection. Complete setup above
separately. The [Nodus plugins](https://nodus-compute.ai/docs/guides/plugins/) bundle both skills and MCP for
Claude Code, Codex and Cursor. Choose one MCP installation method per client.

The same skill files are readable without installation:

- [Setup skill](https://nodus-compute.ai/skills/setup/SKILL.md)
- [Workload skill](https://nodus-compute.ai/skills/workloads/SKILL.md)

## Let your agent help

Paste this into an agent that can read URLs and configure local tools:

```text
Read https://nodus-compute.ai/connect.md and help me connect Nodus to this agent. Reuse any existing Nodus connection. Verify setup by listing my workloads. Do not start paid compute.
```

The agent can prepare configuration and explain the steps. You complete
browser sign-in. Preserve existing settings and use the client's supported
configuration interface.

## Documentation and custom agents

- [llms.txt](https://nodus-compute.ai/llms.txt) indexes the public documentation.
- [llms-full.txt](https://nodus-compute.ai/llms-full.txt) combines every guide
  and reference into one text document.
- [connect.md](https://nodus-compute.ai/connect.md) provides this setup guide
  as Markdown. Every documentation page also links to its Markdown source.
- [OpenAPI](https://nodus-compute.ai/docs/openapi.yaml) defines the customer HTTP contract.
- [Python SDK](https://nodus-compute.ai/docs/reference/python/client/) and [CLI](https://nodus-compute.ai/docs/reference/cli/)
  support custom agents and automation with execution environments.
- [Agent sandboxes](https://nodus-compute.ai/docs/guides/agent-sandboxes/) support interactive commands and
  streamed output through the SDK. The workload MCP tools do not
  expose sandbox operations.


---

Document: https://nodus-compute.ai/docs/guides/connections/
Markdown source: https://nodus-compute.ai/docs/source/guides/connections.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/connections.md

# External data connections

Connections are verified, team-owned references to credentials in the tenant
secret store. Supported kinds are `postgres`, `neon`, `supabase` and `wandb`.
The console shows a read-only list. Use the SDK or CLI to create, verify and
delete connections.

## Store a credential

Store a secret from a private UTF-8 file. The value is never returned by the
secret or connection read endpoints. Files contain the exact value, so omit a
trailing newline for API keys.

```sh
nodus secret set LAB_DB --from-file ./database-secret.txt
nodus connection add neon --name lab-db --secret LAB_DB --scope read --region us-east-1
nodus connection ls
nodus connection verify lab-db
```

`--secret` is an existing secret name or ID, never a credential value. Connection
names start with a letter and contain up to 128 letters, digits, underscores or
hyphens. The name `github` and prefix `conn_` are reserved.

| Kind | Secret file content | Additional connection fields |
| --- | --- | --- |
| `postgres` | A `postgres://` or `postgresql://` URL with user, password, hostname and database | None |
| `neon` | A Postgres URL, or JSON with `postgres_url` and optional `api_key` | Optional `branch` |
| `supabase` | JSON with `project_url`, `service_key` and `postgres_url` | None |
| `wandb` | The API key | Required `entity` and `project` |

Database JSON may include `ca_cert` containing a PEM certificate authority for
certificate verification. For Supabase, use the CA downloaded from the project's
database SSL settings. Its session pooler is supported when the direct database
address is unreachable over IPv4. The complete secret must fit the tenant secret
limit of 4096 UTF-8 bytes.

Database URLs support only `sslmode` and `channel_binding` query parameters.
Accepted SSL modes are `require`, `verify-ca` and `verify-full`. Every accepted
mode verifies the server certificate and hostname. TLS is also required when the
parameter is omitted. Arbitrary certificate file paths, plaintext connections,
private addresses and internal hostnames are refused.

Creating a database connection executes `SELECT 1` inside a read-only transaction.
Creating a wandb connection sends a GraphQL viewer query to `api.wandb.ai` with
the key. Verification has a five-second limit. Failure or timeout saves no
connection. Verification confirms authentication, not table permissions or
access to the selected wandb project.

Connection creation is submitted once and is not automatically retried, including
on transport errors or transient HTTP responses. If the response is lost, the
connection may already exist. Look it up with `client.connections.get("lab-db")`
or `nodus connection ls` before submitting another create request. The name is
unique within your team.

## Python

```python
from nodus import Client

with Client() as client:
    connection = client.connections.create(
        "lab-db",
        "neon",
        secret="LAB_DB",
        scope="read",
        region="us-east-1",
    )
    connections = client.connections.list()
    metadata = client.connections.get(connection["id"])
    verified = client.connections.verify(connection["id"])
    client.connections.delete(connection["id"])
```

`AsyncClient.connections` exposes the same methods with `await`. Scope is `read`,
`write` or `readwrite`. Omitting it selects `read` for database kinds and `write`
for wandb. Region is optional. Connections with no region have no location
restriction.

A connection pins one immutable secret ID and version. Rotating or revoking that
version prevents new uses. Create a new connection to adopt a new version.
Deleting a connection removes its metadata without revoking the underlying
secret. A connection referenced by an export or load cannot be deleted.

## wandb live opt-in

```sh
nodus secret set WANDB_KEY --from-file ./wandb-key.txt
nodus connection add wandb --name lab-wandb --secret WANDB_KEY --entity lab-team --project training --live
nodus connection verify lab-wandb
nodus connection rm lab-wandb
```

Only a current team administrator can enable `live_mode`, using a console session
or an API key owned by that administrator. Member keys and ownerless machine
keys cannot enable it. The action is recorded in team activity.

Only wandb supports the live flag, with `write` or `readwrite` scope. Its declared
hosts are `api.wandb.ai`, `files.wandb.ai` and `storage.googleapis.com`. Database
connections keep credentials on the control plane. Database query imports are
described below.

## Import a database query

Export a query to a normal input asset. The query runs on the control plane and
the database credential never reaches the workload.

```python
from nodus import Client

with Client() as client:
    dataset = client.assets.import_query(
        "lab-db", "SELECT id, uri, label FROM clips WHERE split='train'"
    )
    metadata = client.assets.get(dataset.id)
    print(metadata.export["row_count"])
```

Pass `dataset.id` in your workload's `inputs`, for example
`inputs={"clips": dataset.id}`. The input is a directory containing
`data.parquet`. A training program can read it with
`pandas.read_parquet(os.path.join(os.environ["NODUS_INPUT_clips"], "data.parquet"))`.
Install the appropriate Parquet reader in your training environment.

```sh
nodus asset import-query lab-db "SELECT id, uri, label FROM clips" --format parquet
nodus asset import-query lab-db "SELECT id, uri, label FROM clips" --format csv --reuse
```

`AsyncClient.assets.import_query` and `get` provide the same interface with
`await`. `format` defaults to `parquet`. CSV exports contain a header and use an
empty field for SQL nulls. Parquet preserves nullable integer, floating point,
boolean, UTF-8 text, UTC microsecond timestamp and JSON logical types. Other
Postgres types become UTF-8 strings. Use unique nonempty column aliases.

Queries must begin with `SELECT` or `WITH`. Each export uses a read-only
transaction, a ten-minute timeout and 10000-row cursor batches. Modifying CTEs
and multiple statements are refused. The maximum is 5 GB or 50 million rows,
with no request override. Existing storage quota can impose a smaller byte
limit. A limit error includes the row count reached.

`reuse=True` may return an existing ready asset from the last 24 hours for the
same team, connection, SQL, branch and format. Only outer SQL whitespace is
ignored. Rotated or revoked credentials cannot create or reuse an export.
`reuse=False` creates a fresh asset. An optional `branch` must match the Neon
branch already configured and verified on the connection. Create a separate
connection to use another branch.

An admitted export keeps its original credential version even if the secret is
rotated during execution. Delete the export asset before removing its connection.
The HTTP API returns 202 immediately after durable admission. Poll
`GET /v1/assets/{id}` until `state` is `ready` or `failed`. Failed assets expose
an `error` message, including the reached row count for size or row limits.
They remain visible after cleanup and have zero stored bytes once their
reservation is released. Delete a failed asset when you no longer need its error.

The SDK and CLI poll automatically with short HTTP requests for up to twelve
minutes, returning the ready asset or raising an error with its asset ID.
Execution and queue time together have a ten-minute limit. If observation times
out or you interrupt the client, inspect that asset before repeating the import.
Disconnecting stops observation and leaves the admitted export owned by the
server. Server shutdown cancels active queries and attempts cleanup. After a
restart, pending work resumes while expired active work is failed and cleaned.
Temporary storage failures retain the reservation until cleanup succeeds.

Query assets retain their connection's declared region when used as workload
inputs, workload or stage source assets, or sandbox source assets. If you set
`policy.data_regions`, include each input source's declared region. A connection
without a region and a run without a region constraint remain unrestricted.

After query admission, SDK observation errors expose the admitted ID in
`error.asset_id`. Async cancellation remains `asyncio.CancelledError` and should
be re-raised after your cleanup. Some Python versions wrap that exception at a
task boundary, so use `nodus.asset_id_from_error(error)` to recover the ID from
the preserved exception chain. This returns `None` if no admission ID is
available, including when another layer discards the original exception chain.

The CLI preserves the ID on failure or interruption. Inspect the exact asset
before repeating the import:

```sh
nodus asset get asset_ID
```

This displays the current state, stored bytes, export format and row count when
available, and the safe export error for a failed asset. The Python equivalent
is `client.assets.get(asset_id)` or `await client.assets.get(asset_id)`.

## Attach live wandb to a run

Use an existing administrator-enabled wandb connection with write scope. A run
accepts one connection, by name or ID. Admission pins its metadata and exact
secret version. Rotation and connection deletion do not change an admitted run.
They prevent new admission using that connection. A connection region must match
`policy.data_regions` when that policy is supplied.

```python
from nodus import Client

with Client() as client:
    workload = client.run(
        command=["python", "train.py"],
        connections=["lab-wandb"],
        sweep_id="experiment-42",
        budget=5,
    )
    workload.refresh()
    for link in workload.links:
        print(link.url)
```

The workload image must already contain wandb and your training dependencies.
Your script calls `wandb.init()` normally. Nodus supplies `WANDB_API_KEY`,
`WANDB_ENTITY`, `WANDB_PROJECT`, `WANDB_RUN_GROUP` and `WANDB_NAME` in the process
environment. `NODUS_CONN_LAB_WANDB_KIND` is `wandb`. The default group is the Nodus
workload ID and the name includes the workload and stage IDs. Set the same scalar
`sweep_id` on several runs to group them. This does not schedule a sweep.

The key is delivered to the authenticated execution in memory. It is absent from
payloads, image layers, Docker environment files and checkpoint artifacts.
Captured logs redact the credential, including fragments split between writes.
Managed `WANDB_*`, `NODUS_CONN_*` and `NODUS_SECRET_*` names cannot be overridden
through submit or sandbox exec environment fields.

Live runs use an isolated network namespace. HTTPS can reach the union of the
connection's declared hosts and `policy.egress_allow`. HTTP, unlisted hosts,
private addresses and direct sockets are blocked. Proxy denials appear in
`workload.egress_denied` events. Additional tenant secret references can be
specified with `policy.secret_refs` and are pinned at admission. Deployments
without an enabled isolated execution provider refuse admission before acquiring
capacity. Explicit private pool placement is not supported for live runs.

Run links printed by wandb are validated against the pinned entity and project.
They appear in `workload.links`, workload detail and list responses, and the
console run row. `nodus run` prints a newly captured URL while waiting, including
with `--plain`. `nodus workload get` also displays captured links. Python callers
can use `workload.wait(on_update=callback)` to observe new links on each poll.
The sync and async clients support the same live fields.

A CLI workload file uses the same fields:

```toml
command = ["python", "train.py"]
connections = ["lab-wandb"]
sweep_id = "experiment-42"
budget = 5

[policy]
egress_allow = ["metrics.example.com"]
```

Sandboxes accept `connections=["lab-wandb"]` in `client.sandboxes.create` and
`Sandbox(...)`. Credentials follow the existing authenticated guest boot and
recovery channel. Reconnect preserves the admitted connection references and
secret versions. Sandbox live connections enforce the same HTTPS host union.

## Load results into a database

Declare a CSV, JSONL or Parquet result file with a database sink. Use an active
Postgres, Neon or Supabase connection with `write` or `readwrite` scope. Its
region must be allowed by the workload's `data_regions`, when supplied.

```python
from nodus import Client

with Client() as client:
    workload = client.run(
        command=["python", "train.py"],
        budget=2,
        outputs={
            "results": {
                "path": "results.jsonl",
                "sink": {"connection": "lab-db", "table": "eval_results"},
            }
        },
    )
    workload.wait()
    for output in workload.outputs():
        print(output.name, output.sink_state, output.sink_rows, output.sink_error)
```

Your command writes the declared file. After upload, Nodus loads it on the
control plane using the credential version pinned when the workload was
admitted. Credentials never enter the workload. Secret rotation or revocation
prevents new admissions but preserves already admitted loads. A connection
referenced by saved sink outputs cannot be deleted because reload needs it.

Load state is `pending`, `loading`, `loaded` or `failed`. Workload completion
and output downloads remain available if a database load fails. Inspect the
value-free `sink_error`, correct the target schema or permissions, and retry:

```python
from nodus import Client

with Client() as client:
    workload = client.get("YOUR_WORKLOAD_ID")
    workload.reload_output("results", stage="main")
```

```bash
nodus workload outputs wl_example
nodus workload outputs wl_example --reload results --stage main
```

Table names are single PostgreSQL identifiers without a schema prefix.
Uppercase letters fold to lowercase. Names beginning with `nodus_` are reserved.
Two outputs in the same stage must use different connection and table targets.
Each file column must be a distinct identifier and cannot begin with `nodus_`.

CSV requires a header. Its columns load as `text` and empty cells become SQL
NULL. JSONL requires one object per line. Strings, booleans and numbers become
`text`, `boolean` and `numeric`. Nested objects and arrays become `jsonb`.
Missing keys and JSON null become SQL NULL. A column must keep one non-null
type across the file. Parquet supports flat nullable boolean, integer, float,
text, binary, date, timestamp, decimal and JSON columns. Unsigned integers up to
32 bits load as `bigint`, and unsigned 64-bit integers load as exact `numeric`.

Files are limited to 5 GB, 50 million rows and 256 columns. CSV records, JSONL
lines, Parquet pages and Parquet footers are limited to 8 MiB. Parquet row groups
must fit the reader's 128 MiB decoded-data allowance. A header-only CSV, an empty
JSONL file or a zero-row Parquet file loads zero rows. A CSV without a header
fails. Empty JSONL has no inferred file columns.

Nodus creates a missing table in `public`, or checks that the existing table
contains all file columns with compatible types. Every row also carries
`nodus_workload_id`, `nodus_generation`, `nodus_stage` and `nodus_loaded_at`.
Those four columns must have types `text`, `integer`, `text` and `timestamptz`.
One transaction replaces earlier rows for the same workload and stage. A failed
replacement preserves the prior successful rows. Successful newer generations
fence older loads for the same target table, even when the newer output has zero
rows. Metadata updates do not advance this fence. Reloading does not duplicate rows.

The `public.nodus_runs` table holds workload, stage and generation metadata,
including status, GPU, GPU count, region, customer charge, start and end times,
optional sweep ID, and load time. Final status and charges are updated after
completion. Supplier details are excluded. Join results to metadata with:

```sql
SELECT e.*, r.cost_usd, r.gpu, r.region
FROM eval_results e
JOIN nodus_runs r
  ON e.nodus_workload_id = r.workload_id
 AND e.nodus_stage = r.stage
 AND e.nodus_generation = r.generation
```


---

Document: https://nodus-compute.ai/docs/guides/containers-and-scripts/
Markdown source: https://nodus-compute.ai/docs/source/guides/containers-and-scripts.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/containers-and-scripts.md

# Run your own Python script

Upload your code, choose an image containing its dependencies, and run it on a
GPU. The SDK does not install your script's dependencies automatically.

For example, save this as `hello.py`:

```python
import torch

print(torch.cuda.get_device_name(0))
```

Then submit it from Python:

```python
import nodus

with nodus.Client() as client:
    code = client.assets.upload("hello.py")
    workload = client.run(
        image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
        source_asset_id=code.id,
        command=["python", "hello.py"],
        budget=5,
    )
    print(workload.id)
    done = workload.wait()
    if not done.succeeded:
        raise RuntimeError(f"Workload ended: {done.status}")
    print(done.logs())
```

`assets.upload()` transfers the file explicitly. `run()` uses the uploaded asset
as the source working directory. A filename in `command` alone never uploads it.
For several source files, upload an archive or import a GitHub repository. See
[code and datasets](https://nodus-compute.ai/docs/guides/assets/).

## Use a custom container

When you need additional dependencies, package them with your code in an image.
For example, put this `Dockerfile` beside `hello.py`:

```dockerfile
FROM pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime
WORKDIR /app
COPY hello.py /app/hello.py
```

Build and push to a registry Nodus can pull from. Replace the namespace below:

```bash
docker build -t YOUR_REGISTRY/hello:v1 .
docker push YOUR_REGISTRY/hello:v1
```

Submit the image without a source asset:

```python
import nodus

with nodus.Client() as client:
    workload = client.run(
        image="YOUR_REGISTRY/hello:v1",
        command=["python", "/app/hello.py"],
        budget=5,
    )
    print(workload.id)
```

Use absolute paths for code baked into the image. Pin versions or digests for
repeatability. The SDK has no registry-credential argument, so confirm access
before using a private image. Do not bake credentials into images.


---

Document: https://nodus-compute.ai/docs/guides/gpu-workloads/
Markdown source: https://nodus-compute.ai/docs/source/guides/gpu-workloads.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/gpu-workloads.md

# GPU training and fine-tuning

Start by verifying CUDA in a known PyTorch image:

```python
import nodus

with nodus.Client() as client:
    workload = client.run(
        image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
        command=[
            "python", "-c",
            "import torch\n"
            "assert torch.cuda.is_available()\n"
            "print(torch.cuda.get_device_name(0))",
        ],
        peak_memory_gb=24,
        budget=5,
    )
    print(workload.id)
    done = workload.wait()
    if not done.succeeded:
        raise RuntimeError("GPU smoke test did not complete successfully")
```

The budget is illustrative, not a price guarantee. A feasible route still needs
to fit your account, region policy, and available capacity.

For training, [build an image](https://nodus-compute.ai/docs/guides/containers-and-scripts/) containing your code,
framework dependencies, and data-access logic. Then run its actual command:

```python
import nodus

with nodus.Client() as client:
    workload = client.run(
        image="YOUR_REGISTRY/trainer:v1",
        command=["python", "/app/train.py", "--epochs", "3"],
        model="LoRA-fine-tune",
        peak_memory_gb=24,
        budget=25,
    )
    print(workload.id)
    done = workload.wait()
    if not done.succeeded:
        raise RuntimeError(f"Training ended: {done.status}")
    print(done.logs())
```

This is a template: `/app/train.py` and `--epochs` belong to your application.
`model` is a sizing hint, not a model download. Choose enough GPU memory for your program.

Declare final model files with `outputs` when you need SDK downloads. See
[multi-stage workloads](https://nodus-compute.ai/docs/guides/multi-stage-workloads/). Nodus handles placement and
execution. Advanced application integrations are documented separately in the
[parameter reference](https://nodus-compute.ai/docs/reference/parameters/).

## Eight H100s for single-node pretraining

Use an image containing your training code and its distributed dependencies.
Keep the application's training arguments in your command:

```python
with nodus.Client() as client:
    workload = client.run(
        image="YOUR_REGISTRY/trainer:v1",
        command=[
            "torchrun", "--nnodes=1", "--nproc_per_node=8",
            "/app/pretrain.py", "--config", "/app/pretrain.yaml",
        ],
        gpu="H100",
        gpu_count=8,
        peak_memory_gb=80,
        budget=100,
    )
    print(workload.id)
```

This example requires eight H100s on one machine with at least 80 GB per GPU.
The budget is illustrative and applies to the whole run. Capacity is not
guaranteed. Eight devices do not imply NVLink, NVSwitch or pooled memory.
Nodus preserves the distributed command and does not rewrite your training
arguments. See [resource requirements](https://nodus-compute.ai/docs/reference/parameters/requirements/)
for count and topology validation.


---

Document: https://nodus-compute.ai/docs/guides/mcp/
Markdown source: https://nodus-compute.ai/docs/source/guides/mcp.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/mcp.md

# MCP tools

Connect Claude, Cursor, Codex or another MCP client to Nodus. Ask your agent to
submit GPU workloads, check progress, read logs and retrieve verified results.

For the shortest setup, [choose your coding agent](https://nodus-compute.ai/docs/guides/connect/). It includes
copyable commands, a Cursor install link and configurations for other clients.

For Codex, Claude Code or Cursor, use the [Nodus plugin](https://nodus-compute.ai/docs/guides/plugins/) to install
the tools and setup guidance together. The manual configuration below works
with other MCP clients too.

## Hosted connection

Use the [connection page](https://nodus-compute.ai/docs/guides/connect/#quick-connection) for hosted HTTP MCP with
browser authorization. Tools use a revocable grant bound to your account and
team. Read-only grants omit submission and cancellation tools.

Hosted `get_workload_output` returns a download URL valid for ten minutes,
plus the file's SHA-256 and byte count. Download with your agent's own file
tools, without an Authorization header and without following redirects.
Verify the checksum before reporting delivery. Treat the URL as a secret.
The hosted server cannot write to your local filesystem. Revoking the
connection invalidates its download links.

## Local connection in two steps

You need [uv](https://docs.astral.sh/uv/getting-started/installation/) installed.
`uvx` downloads the public Nodus package and starts the server for your client.
It manages the Python runtime and package dependencies for you.

**1. Sign in once.** Run this in your terminal and complete browser sign-in:

```sh
uvx --from 'nodus-compute[mcp]==0.6.0' nodus login
```

**2. Add Nodus to your MCP client.** In Claude Desktop or Cursor, add this to
your MCP server configuration and reload the connection:

```json
{
  "mcpServers": {
    "nodus": {
      "command": "uvx",
      "args": ["--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"]
    }
  }
}
```

For Codex, run this instead of editing JSON:

```sh
codex mcp add nodus -- uvx --from 'nodus-compute[mcp]==0.6.0' nodus-mcp
```

The server uses your saved sign-in. There is no API key to paste into the
configuration, private repository to clone or executable to compile.

Ask your agent: **"List my Nodus workloads."** This checks the connection
without starting paid compute. Your local client should discover nine tools.

## Already using pip?

Install the MCP extra and reuse your existing Nodus sign-in:

```sh
pip install --upgrade 'nodus-compute[mcp]==0.6.0'
nodus login
```

Set your client's command to `nodus` and its arguments to `["mcp"]`.
The `nodus-mcp` executable is also installed by the package. Both commands
start the same local server over standard input and output.

## Authentication and custom deployments

The server reads the same saved credentials as the Python SDK. For automation,
set `NODUS_API_KEY` in the server process environment. For a custom deployment,
set `NODUS_BASE_URL` to its API origin without `/v1`, or sign in with
`nodus login --base-url https://your-api.example`.

Environment settings take priority over saved settings. Keep keys out of
prompts, tool arguments and committed files. See
[authentication](https://nodus-compute.ai/docs/getting-started/authentication/) for more details.

## Tool reference

Arguments below are the JSON object passed to the named tool. Hosted connections
use `get_workload_output` instead of `download_workload_output`. Local downloads
write on the machine running MCP and require an existing destination directory.

| Tool | Required arguments | Optional arguments | Result |
| --- | --- | --- | --- |
| `validate_workload` | `workload` | None | Validation without admission, spending or capacity reservation |
| `download_workload_output` | `workload_id`, `name`, `destination` | `stage` | Local verified download, existing files refused |
| `submit_workload` | `idempotency_key`, `workload` | None | The API's submission response, including the workload ID |
| `list_workloads` | None | `scope`, `limit`, `offset` | Workloads and `next_offset` when another page exists |
| `get_workload` | `workload_id` | None | Workload details, status and current meter |
| `cancel_workload` | `workload_id` | None | Cancellation requested, followed by asynchronous cleanup |
| `get_workload_events` | `workload_id` | `after` | Up to 100 lifecycle events after the supplied event ID |
| `get_workload_logs` | `workload_id` | None | Retained workload log text |
| `list_workload_outputs` | `workload_id` | None | Output metadata and download paths |

Workload IDs contain only letters, digits, underscores and hyphens. The
idempotency key is a nonempty printable ASCII string without spaces or line
breaks. `workload` is the HTTP
workload request object, not the keyword arguments to Python `client.run()`.
For example, the HTTP budget field is `outcome.max_cost_usd`, not `budget`.
Use the [OpenAPI contract](https://nodus-compute.ai/docs/openapi.yaml) for the full request
schema and the [parameter reference](https://nodus-compute.ai/docs/reference/parameters/) for
field descriptions.

`scope` accepts `team` or `mine`. Omitted scope uses the team's workloads.
`mine` requires a credential associated with a team member. `limit` accepts
integers from 1 to 100. `offset` accepts integers from 0 to 2147483647.
`after` accepts integer event IDs from 0 to 9007199254740991.

The tools return MCP text content containing API JSON or log text. API failures
return a tool result with `isError: true` and the API error details. These are
workload tools. Use the [sandbox guide](https://nodus-compute.ai/docs/guides/agent-sandboxes/) for interactive
sandbox commands and streaming output.

## Submit and monitor a workload

This example checks the remote GPU and allows up to $1 in workload spending.
Choose a budget within your authorization before submitting. An accepted
request does not guarantee completion within that limit.

Call `submit_workload` with:

```json
{
  "idempotency_key": "gpu-check-unique-run-id",
  "workload": {
    "source": {
      "image": "pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
      "command": [
        "python",
        "-c",
        "import torch\nassert torch.cuda.is_available()\nprint(torch.cuda.get_device_name(0))"
      ]
    },
    "requirements": {
      "gpu_count": 1
    },
    "outcome": {
      "max_cost_usd": 1
    }
  }
}
```

Choose a new unique key for each intentional run. If a submission times out,
retry the exact same request with its original key. Do not create a second
paid run by changing the key during an uncertain retry.

Save the returned workload ID. Pass it to `get_workload`,
`get_workload_logs` or `list_workload_outputs`:

```json
{
  "workload_id": "wl_your_workload_id"
}
```

Check workload status until terminal and verify it succeeded before reporting
success. The example writes its result to the log. For downloadable files,
your program must write the output files described in
[logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/).

`list_workload_outputs` lists metadata. It does not download files to your
machine. Call `download_workload_output` locally or `get_workload_output`
on a hosted connection to retrieve them.

## Read subsequent pages

For `list_workloads`, pass the response's `next_offset` in the next call.
For example, when `next_offset` is 20:

```json
{
  "limit": 20,
  "offset": 20
}
```

For `get_workload_events`, pass the last event's `id` as `after`. For example,
when the last event ID is 100:

```json
{
  "workload_id": "wl_your_workload_id",
  "after": 100
}
```

Keep the cursor to read later events without requesting the first page again.
An empty page means there are no later events at that moment.

## Cancel and troubleshoot

To stop a workload, call `cancel_workload` with only `workload_id`.
Cancellation does not take an idempotency key or workload body and can be
requested again for the same workload. Continue checking `get_workload`
until the workload reaches a terminal state. A cancellation acknowledgement
does not mean resource cleanup has finished.

If your client cannot find `uvx`, restart the client after installing uv or
set `command` to the full path printed by `command -v uvx` on macOS and Linux,
or `where.exe uvx` on Windows.

If a tool asks you to sign in, run the sign-in command above from the same
computer and user account as the MCP client. For expired or rejected sign-in,
run it with `--force`. For `scope: mine`, use a credential associated with a
team member.

If `nodus mcp` asks for MCP support, install the `[mcp]` extra using the pip
command above. The plain Python SDK install keeps MCP dependencies optional.

Custom API origins must use HTTPS, except for local loopback development.
HTTP redirects are refused. Set the final API origin directly. The server
reuses HTTP connections between tool calls and closes them on shutdown.
Saved sign-in changes apply to the next call without restarting the server.
API requests have a 30 second timeout. A timeout does not cancel a remote workload.

Log calls support the API's 8 MiB log payload plus truncation notices.
Responses larger than the server's 16 MiB bound return a tool error.


---

Document: https://nodus-compute.ai/docs/guides/monitoring-and-outputs/
Markdown source: https://nodus-compute.ai/docs/source/guides/monitoring-and-outputs.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/monitoring-and-outputs.md

# Logs and results

Keep the workload ID returned by `client.run()`. You can use it later to check
the run from any Python process signed in to the same account.

## Wait for the result

```python
import nodus

with nodus.Client() as client:
    workload = client.get("YOUR_WORKLOAD_ID")
    done = workload.wait()
    print(done.status, done.cost_now_usd)
    if not done.succeeded:
        raise RuntimeError(f"Workload ended: {done.status}")
    print(done.logs())
```

`wait()` returns when the workload finishes, fails, or is cancelled. Check
`succeeded` before using its results. Interactive waits show elapsed time, lifecycle events, live program output,
and available training progress. Retrieve recorded output with `logs()`.
If cancellation stops the run before a log artifact is committed, this call
can return the retained live snapshot for up to 24 hours after termination.
That snapshot is limited to 8 MiB per attempt and may omit output that had
not reached Nodus before cancellation. Download it promptly if you need to
keep it. A committed log artifact retains its normal retention.

## Download files

When a stage omits output declarations, Nodus preserves non-empty `outputs/`
and `results/` folders as `outputs.tar` and `results.tar`. Save the complete model
bundle there, including weights, configuration and tokenizer files. Directories
named `.venv`, `venv`, `node_modules`, `.git`, `.nodus`, `__pycache__` and `.cache`
are excluded recursively. Symbolic links and special files are skipped, so save
model files directly into the folder rather than linking to a cache.
Other locations require explicit output declarations, which replace automatic
folder collection for that stage.

Download available results inside the client context after the workload completes:

```python
for path in done.download():
    print(path)
```

Files go into `outputs/WORKLOAD_ID/STAGE/NAME` by default, using the published output name. Pass a directory to
`done.download("results")` to choose another location. For one file, use
`done.download_output("result", "result.json")` with your declared output name. See
[output declarations](https://nodus-compute.ai/docs/reference/parameters/source/#input-and-output-files) when your program writes
files outside the default folders. Default folder archives are downloaded as tar
files and are not automatically extracted. Empty or absent default folders produce
no archive. A script must actually save its model to disk for Nodus to preserve it.

## Progress and cancellation

`workload.status` gives the last fetched status. Use `workload.refresh()`
to update that handle or `client.get(workload.id)` to get a new one.
For lifecycle updates, iterate over `workload.stream_events()`.
These events describe execution progress, not your program's stdout.

Ctrl+C during a synchronous wait requests cancellation and remote resource cleanup.
Cancelling an async `wait()` task also requests remote cancellation before re-raising
the interruption. If cancellation cannot be confirmed, check the run and retry with
`await workload.cancel()`.
To cancel explicitly, call `client.cancel(workload.id)`. A wait timeout ends
local observation without cancelling the run.

## From the terminal

```bash
nodus wait WORKLOAD_ID
nodus logs WORKLOAD_ID
nodus download WORKLOAD_ID
```

Check a workload without waiting with `nodus status WORKLOAD_ID`. Stop it with
`nodus cancel WORKLOAD_ID`. Downloads include declared output files, not the
entire container filesystem.

## Live display

`wait(progress=None)` automatically enables a Rich display in an interactive
terminal on Windows, macOS, and Linux. Use `progress=True` to enable output
explicitly or `progress=False` for silent waiting. Progress goes to stderr and
does not capture your program's stdout. Redirected output has no animations.

Elapsed time refreshes every second. Server updates are polled every two
seconds by default. Recognized training output can show steps, epochs, loss,
and throughput. Percentages appear only when a matching total is reported.
For other programs, stage updates, elapsed time, and logs remain visible.

For a custom display, call `client.live_logs(workload.id, after=cursor)`. It
returns `chunks`, `next_cursor`, and `truncated`. Each chunk has a numeric ID,
stage ID, generation, and text. Each response contains at most 16 chunks.
Pass the returned cursor on the next request and keep reading until a page is
empty. An empty page means there is no new output yet, not that the run has finished.
Live capture is limited to 8 MiB per attempt, with explicit truncation. The live
view is retained for 24 hours after termination. Saved logs retain their normal
retention. Older runners show saved logs as they become available.

Live output combines stdout and stderr. Buffered programs may delay their own
output. Python output is unbuffered unless explicitly overridden.

## Existing local files

`download()` creates its destination directories and refuses to overwrite files.
Choose a fresh directory if a previous download already exists. The lower-level
`download_output()` requires an existing parent directory and replaces its target
only after the complete download passes its integrity checks.


---

Document: https://nodus-compute.ai/docs/guides/multi-stage-workloads/
Markdown source: https://nodus-compute.ai/docs/source/guides/multi-stage-workloads.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/multi-stage-workloads.md

# Multi-stage workloads and final outputs

Use an explicit stage list when work has multiple steps or dependencies.
For one command, declare downloadable files with `outputs={"result": "result.json"}`
directly on `client.run()`. Multiple stages can reference output names without
sharing a machine or filesystem.

Save the [complete Python example](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/examples/multi_stage.py) as `multi_stage.py`
in your current directory. Example scripts are not installed by pip. Then run:

```bash
python multi_stage.py --submission-id pipeline-001 --budget 10
```

The first stage writes `numbers.json`. The second reads the resolved upstream
input from `NODUS_INPUT_numbers`, writes `result.json`, and declares it as
`result`. After successful completion, the example downloads that output to
`results/result.json`. The example uses GPU capacity and small inputs to demonstrate file transfer.
Replace the stage commands with your training and evaluation code for real work.
Stages execute serially, including stages without dependencies.

This example requires the deployment's runner to support declared outputs and
resolved input environment variables. It illustrates the runtime contract. It
is not a promise that every deployed runner revision supports every feature.

See [`examples/multi_stage.py`](https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/examples/multi_stage.py) for the code and
[every stage field](https://nodus-compute.ai/docs/reference/parameters/stages/) for inheritance and rules.
No top-level `image` or `command` is passed when stages provide their own sources.


---

Document: https://nodus-compute.ai/docs/guides/operations/
Markdown source: https://nodus-compute.ai/docs/source/guides/operations.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/operations.md

# Versioned workload and draft operations

These methods are available from the SDK source checkout and are pending a
package release. Published SDK 0.5.3 does not include `client.operations`.

Sign in with `nodus login` or configure `NODUS_API_KEY`, then use
`client.operations` on a `nodus.Client`. `nodus.AsyncClient` provides the same
methods with `await`. This interface requires a server that exposes
`/v1/operations/v1`.

Prepare a customer workload request with your image, command, resources and an
explicit `outcome.max_cost_usd`. Call `operations.validate(workload)` to check
the request without starting compute. Submit it with
`operations.submit(workload, idempotency_key="your-stable-run-key")` when you
intend to start paid execution.

Keep the request and key for recovery. Transport retries preserve a snapshot of
the supplied workload. After an uncertain result, retry with the same request
and key. A new intentional run requires a new key. Validation does not reserve
capacity or guarantee admission.

The returned `Workload` supports observing progress and retrieving results.
Check `succeeded` after waiting. List final files with `operations.outputs(id)`
and use `client.download_output(...)` to download and verify an output.
Asynchronous submissions return `AsyncWorkload`.

## Available methods

| Method | Result |
|---|---|
| `catalog()` | `OperationCatalog` containing server definitions and argument schemas |
| `get_run_draft()` | `RunDraft` with saved form values and their revision |
| `update_run_draft(patch, expected_revision=...)` | `RunDraft` after applying the partial edit |
| `list(scope=None, limit=None, offset=None)` | `WorkloadPage` with `workloads` and `next_offset` |
| `get(workload_id)` | `Workload` with current status and meter |
| `events(workload_id, after=None)` | One page of `Event` objects |
| `logs(workload_id)` | Retained log text |
| `outputs(workload_id)` | Final `Output` metadata and download paths |
| `validate(workload)` | `WorkloadValidation` confirming validation without submission |
| `submit(workload, idempotency_key=...)` | Accepted `Workload` |
| `cancel(workload_id)` | `None` after cancellation is requested |

Omitted list arguments keep the server defaults, including team scope. Pass a
returned `next_offset` to read the next workload page. For events, pass the last
event's `seq` as `after`. Cancellation acknowledges the request while cleanup
continues asynchronously.

## Shared run forms

Use a key associated with your team membership to read and edit your saved
**New run** form. The console, personal agent and SDK share that form within
your active team. A key without a member identity receives HTTP 403.

`get_run_draft()` returns `revision`, partial `values` and `updated_at`.
`RunDraftValues` describes the saved fields. An empty form has revision zero,
empty values and no update time. The server does not invent a budget.

Pass a `RunDraftPatch` and the revision you read to `update_run_draft()`.
Supported fields are `name`, `command`, `image`, `gpu`, `gpu_count`, `memory_gb`,
`max_cost_usd`, `checkpoint_paths` and `result_paths`. Omit a field to leave it
unchanged. Set it to `None` to remove its saved value. The command is a form
text value. Updating the form does not submit work or authorize spending.

An identical immediate retry can return its saved revision. A conflicting edit
raises `RunDraftConflictError`. Read the latest form and review your changes
before writing again. The SDK does not apply a stale patch to a newer revision.
Validate a complete customer workload request separately before submission.

## Contract discovery

`operations.catalog()` reads definitions from the server. Each definition
includes its canonical ID, version, tool alias, argument schema, permission
scope and supported transports. These describe capabilities. Authorization is
enforced when an operation runs.

The typed methods call canonical IDs such as `workloads.submit` through the
version 1 interface. The server owns argument and budget validation. The SDK
adds response types, retry handling and checks for invalid submission receipts.
An uncertain submission error retains the supplied key in
`error.payload["idempotency_key"]`.

See the [Python client reference](https://nodus-compute.ai/docs/reference/python/client/) for workload
handles and downloads, and [HTTP request parameters](https://nodus-compute.ai/docs/reference/parameters/)
for preparing a workload body.


---

Document: https://nodus-compute.ai/docs/guides/plugins/
Markdown source: https://nodus-compute.ai/docs/source/guides/plugins.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/plugins.md

# Nodus plugins

Install Nodus in Codex, Claude Code or Cursor to run GPU workloads from your
coding agent. The plugins include MCP tools and two skills for setup and workload execution.
Choose hosted browser sign-in or a local package that reuses saved credentials.

For a direct MCP connection or an Add to Cursor install link, use
[Connect your coding agent](https://nodus-compute.ai/docs/guides/connect/). Choose one MCP installation method
per client to avoid duplicate tools.

## Hosted plugin with browser sign-in

The `nodus-hosted` plugin bundles remote MCP and skills. It requires no local
Python or uv installation. Enable one Nodus plugin or manual connection at a
time to avoid duplicate tools.

Claude Code:

```sh
claude plugin marketplace add nodus-compute/Nodus-sdk-python
claude plugin install nodus-hosted@nodus
```

Codex:

```sh
codex plugin marketplace add nodus-compute/Nodus-sdk-python
codex plugin add nodus-hosted@nodus
```

Authenticate Nodus through your client's MCP controls, then ask it to list
workloads. In Cursor, use the hosted install button on the
[connection page](https://nodus-compute.ai/connect/) or import the repository's
`plugins/nodus-hosted` package into your team marketplace.

These packages are distributed through the Nodus repository marketplace.
Public vendor directory listings require separate vendor review. No public
listing is required to use the direct install buttons or repository commands.


## Sign in once

Install [uv](https://docs.astral.sh/uv/getting-started/installation/), then run:

```sh
uvx --from 'nodus-compute[mcp]==0.6.0' nodus login
```

Complete browser sign-in on the same machine where your client runs. The
plugin reuses this saved login. No API key belongs in your plugin settings.

## Codex

Run in your terminal:

```sh
codex plugin marketplace add nodus-compute/Nodus-sdk-python
codex plugin add nodus@nodus
```

Start a new Codex task, then ask **"List my Nodus workloads."** You can also
ask **"Help me connect Nodus"** to use the setup skill.

## Claude Code

Run in your terminal:

```sh
claude plugin marketplace add nodus-compute/Nodus-sdk-python
claude plugin install nodus@nodus
```

Restart Claude Code. Ask **"List my Nodus workloads"** or run `/nodus:setup`
for connection help. `/nodus:workloads` loads the workload guide.

## Cursor

The plugin can be installed locally without a marketplace listing. Download
the [public repository ZIP](https://github.com/nodus-compute/Nodus-sdk-python/archive/refs/heads/main.zip)
and extract it. Copy its `plugins/nodus` folder to:

- macOS/Linux: `~/.cursor/plugins/local/nodus`
- Windows: `%USERPROFILE%\.cursor\plugins\local\nodus`

Copy the whole folder, including its hidden manifest directories and
`.mcp.json`. Do not nest an extra `nodus` folder inside the destination.
Restart Cursor or run **Developer: Reload Window**, then open **Customize**
and check that Nodus's skills and MCP server appear. Ask
**"List my Nodus workloads."**

If the local plugin does not appear on a managed account, ask your Cursor
admin to check **Allow Local Plugin Imports**. This is off by default on
Enterprise. An installed marketplace plugin with the same name takes
precedence over a local copy.

For teams, import `https://github.com/nodus-compute/Nodus-sdk-python` from
**Dashboard > Plugins & MCPs > Team Marketplaces > Add Marketplace > Import
from Repo**. Team marketplaces require a compatible Cursor plan. See
[Cursor's plugin documentation](https://cursor.com/docs/plugins).

Nodus does not yet have a public Cursor Marketplace listing. Installing the
local plugin or importing the repository does not depend on that listing.

## Use the tools

The first workload listing checks your connection without launching paid
compute. The plugin exposes:

- `validate_workload`
- `download_workload_output` locally or `get_workload_output` when hosted
- `submit_workload`
- `list_workloads`
- `get_workload`
- `cancel_workload`
- `get_workload_events`
- `get_workload_logs`
- `list_workload_outputs`

For a new run, give the agent your container image, command, GPU requirements
and spending limit. For example: **"Prepare my training command for Nodus
with a $10 maximum. Show me the request before submitting."** Preparing a
request does not start a workload.

The workload skill helps the agent preserve your budget, avoid duplicate
submissions after uncertain responses, and check results before reporting
success. Output listing returns metadata. Use the download tool to retrieve
the requested files. See the
[MCP tool reference](https://nodus-compute.ai/docs/guides/mcp/#tool-reference) for arguments and examples.

## Troubleshooting and updates

If you already configured Nodus manually, disable that duplicate MCP entry
when switching to the plugin. Keep other servers intact.

If `uvx` is missing, install uv and restart your client. GUI clients may need
the full path to `uvx` in their MCP command. Find it with `command -v uvx` on
macOS/Linux or `where.exe uvx` on Windows.

If sign-in fails, rerun the login command. A remote client must sign in on
the machine that runs the MCP server. Environment overrides take precedence
over saved settings. See [MCP authentication](https://nodus-compute.ai/docs/guides/mcp/#authentication-and-custom-deployments).

Update the plugin through your client's plugin manager. For a local Cursor
installation, replace the plugin folder with the version from a fresh
download and reload. The plugin pins its MCP dependency to a tested release.
Updating a separate global Python installation does not change that pin.

## Plugin development

All three client manifests live in the public repository's `plugins/nodus`
folder and share one `.mcp.json` and the same skills. The marketplace manifests
live at `.agents/plugins/marketplace.json`, `.claude-plugin/marketplace.json`
and `.cursor-plugin/marketplace.json` in the repository root.

From a checkout, use its absolute path in place of
`nodus-compute/Nodus-sdk-python` in the Codex or Claude marketplace command.
This tests the local files without publishing them. Cursor loads the same
folder from its local plugin directory.

Validate the plugin with `claude plugin validate plugins/nodus --strict`.
Run `python scripts/verify-plugins.py` with the SDK's development dependencies
and uv installed to test the declared launch commands against a local HTTP
fixture. This check downloads the pinned public package and does not start
paid compute.

See the [Claude plugin reference](https://code.claude.com/docs/en/plugins-reference)
and [Cursor plugin reference](https://cursor.com/docs/reference/plugins)
for their manifest contracts. A public Cursor listing requires submission at
[Cursor Marketplace](https://cursor.com/marketplace/publish) and vendor review.


---

Document: https://nodus-compute.ai/docs/guides/pools/
Markdown source: https://nodus-compute.ai/docs/source/guides/pools.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/pools.md

# Use your own GPU hosts

Sign in with `nodus login` or configure `NODUS_API_KEY`. Pools register
customer-owned GPU hosts for free, read-only measurement on deployments where
Compute is enabled. Your existing scheduler continues running your workloads.
Predict adds an optional paid forecast and advisory recommendations. Route
requires separate execution enrollment and explicit price consent.

Create a pool with `nodus pools create Research`. The command prints its pool
ID. Run `nodus pools token POOL_ID`, replacing `POOL_ID` with that returned ID,
to issue an observe enrollment token. This command prints a secret. Keep the
token out of shared logs and source control. Each token can enroll one host
and expires after 24 hours.

Use the Compute enrollment panel's installation instructions on your Linux
host from your infrastructure provider or data center. After enrollment, `nodus pools hosts POOL_ID` shows each host's ID, name,
health, agent mode, and device count. Hosts become lost when their heartbeat
has been absent for three minutes. Use the console to inspect their devices.

## Python methods

`client.pools` and `AsyncClient.pools` expose the same methods. Await methods
on the asynchronous client. IDs always come from the server.

| Method | Result |
|---|---|
| `create(name)` | A `Pool` configured for read-only measurement |
| `list()` | All pools owned by the authenticated team |
| `get(pool_id)` | A `Pool` with its current configuration |
| `update(pool_id, name=..., owned_cost_micros_per_hour=...)` | Updated `Pool`. Supply at least one setting |
| `enrollment_token(pool_id, mode="observe", host_id=None)` | An `EnrollmentToken` with `id`, `token`, `mode`, and `expires_at` |
| `utilization(pool_id, from_=..., to=..., bucket=...)` | `PoolUtilization` with a summary, host summaries, and time buckets |
| `set_route(pool_id, enabled, accepted_rate_version=..., accepted_rate_micros=...)` | Updated `Pool` with explicit price consent when enabling |
| `update_route_settings(pool_id, wait_policy=..., wait_alpha=...)` | Updated future placement settings |
| `hosts(pool_id)` | `PoolHost` objects with health, inventory, and `HostDevice` objects |
| `drain_host(pool_id, host_id)` | The host marked draining, without stopping customer processes |
| `remove_host(pool_id, host_id)` | Revokes the host credential and removes the host, preserving historical measurements |

`Pool.owned_cost_micros_per_hour` is your supplied hardware cost in USD micros
per hour. It does not create a charge. Missing server cost fields remain
`None`. Full pool and host metadata is available through their `raw` fields.
The token value is accessible through `EnrollmentToken.token` and is excluded
from its printed representation.

Pool requests do not automatically retry or follow redirects. After an
uncertain create response, inspect the pool list before creating another.
An uncertain token response may have issued a token whose secret was lost.
Issue a new token only when you intend to create another credential. Host
removal does not uninstall the agent or stop programs on the machine.


## Read the utilization ledger

Run `nodus pools utilization POOL_ID` to see measured allocated, busy, and
busy-of-allocated percentages. Add `--json` for host buckets and observed
foreign device IDs. Queued and fragmentation durations are workload-seconds. Other durations
are device-seconds. Unknown readings are `None` in Python and `null` in JSON.
A measured zero is distinct from an unknown reading.

The default window is the last seven days of complete UTC hours. Use `--from`
and `--to` with RFC 3339 timestamps to choose a window of at most 31 days.
Both boundaries must align to a UTC hour. The end is exclusive. Use
`--bucket hour` or `--bucket day` to choose the grouping. The corresponding
Python keywords are `from_`, `to`, and `bucket`. Omitting them leaves defaults
to the server.

`PoolUtilization.summary` contains `UtilizationMetrics`. Each entry in
`hosts` contains a host summary and `buckets`. Every bucket carries `start`,
`end`, `metrics`, and `foreign_device_ids`. The response's `from_`, `to`, and
`bucket` identify the measured window. The unchanged JSON is in `raw`.

`data_status` is `complete`, `partial`, or `no_data`. Partial coverage hides
derived totals and percentages. Foreign device IDs still identify allocation
observed during a partial bucket. Summaries cover retained ready device time. A complete summary does not claim
continuous coverage of every host hour. Empty buckets report `no_data`.
Missing history is never counted as idle.
Fragmentation, queued, and burst metrics require retained Route evidence and
appear in the pool summary. Host rows keep these fields unknown. Gaps,
ambiguous queue retries, and unsettled execution keep affected values unknown.
`summary.burst_cost_micros` contains exact settled customer cost in USD micros
when each relevant burst execution falls wholly inside the requested window.
A burst crossing a window boundary leaves cost unknown instead of prorating it.


## Forecasts and advisory recommendations

Observe measurements remain free. Predict costs **$99 per account per UTC
calendar month**, with no additional pool or device fee. The first activation
charges the full current month without proration. Enabling another pool in an
already-active period adds no charge. An active period can reflect an accepted
postpaid charge and does not mean an invoice has been paid. Disable Predict on every pool to stop
future renewal. Disabling does not refund the current period.

Read `client.pools.forecast(pool_id, horizon=7)` or use
`nodus pools forecast POOL_ID --horizon 7 --json` to inspect the server's
current subscription rate and cached forecast. The supported horizons are
7 and 30 days. Omit `horizon` to use the server default of 30 days. Existing
cached forecasts and recommendations remain readable without a refresh
charge when Predict is disabled or paused. Only administrators can change
the subscription or record recommendation outcomes.

A forecast response contains `subscription`, `refresh_status`, and `snapshot`.
A missing snapshot remains `None`. The snapshot identifies its model, creation
time, history coverage, hourly p10, p50, and p90 device-hour bands, owned device
count, and any advisory market price. Four complete weeks of measured history
are required. The weekly seasonal baseline is identified explicitly. Missing
history does not become zero demand, and the SDK does not invent a learned
model or a market price.

When Route provides known queued demand, `snapshot.forecast.queue` records its
known device-hours and the count of jobs with unknown runtimes. Each point's
`queue_device_hours` identifies the contribution added to its band. This assumes
known queued jobs start next hour and is not a placement promise. The evidence
separates demand included in the selected horizon from demand beyond it.
Older snapshots can omit this evidence.

The `calibration` object evaluates predictions issued before their target
hours against subsequently observed outcomes. Unknown coverage and pinball
losses remain `None`. Hourly coverage is the observed fraction within the
issued p10 to p90 band. Daily coverage counts fully evaluated UTC days whose
every hour fell within the band. A nominal hourly band does not guarantee
that a whole day falls inside it.

Enable Predict only after reviewing the returned price. Python callers use
`set_predict(pool_id, True, accepted_rate_version=...,
accepted_monthly_micros=...)`, supplying the exact rate version and integer
USD micros they accept. The SDK has no default consent or amount. With the
current rate, the CLI is:

```sh
nodus pools predict POOL_ID on \
  --accept-rate-version predict-account-monthly-v1 \
  --accept-monthly-micros 99000000
```

Disable with `client.pools.set_predict(pool_id, False)` or
`nodus pools predict POOL_ID off`. After an uncertain subscription response,
refresh the forecast response before deciding whether to try again.

Read advice with `client.pools.recommendations(pool_id)` or
`nodus pools recommendations POOL_ID --json`. Each recommendation includes
its expiration, advisory evidence, state, and any customer-reported outcome.
Rightsizing evidence exposes released device count, owned hourly cost,
expected burst device-hours, advisory burst price, and estimated savings.
These are scenario estimates, not measured savings or a guaranteed workload
completion price. Idle-reclaim advice identifies a host and device, sampled
low-utilization allocation, its recent trigger interval, and whether foreign
allocation was observed. Its estimated saving is unavailable. Nodus does not
identify or stop a customer process through this advice. Drain-window advice
identifies a host and a prospective UTC interval with at least three
consecutive hours whose forecast p90 demand is below one device. Its evidence
requires four complete weeks of host history. It is not an availability
guarantee, does not estimate a saving, and does not drain the host.
Placement-consolidation advice requires 336 complete hours of Route
observations and measured waits for requests needing multiple devices.
It reports workload-seconds above the stated policy threshold. Its packing
policy fills hosts first, excludes moving foreign jobs, and does not claim
an estimated saving. Missing Route observations produce no such advice.

Advice is paginated, with up to 100 records per response. Follow
`next_cursor` with the same pool and optional state filter:

```python
page = client.pools.recommendations(pool_id, state="expired", limit=25)
while page.next_cursor is not None:
    page = client.pools.recommendations(
        pool_id, state="expired", limit=25, cursor=page.next_cursor
    )
```

Omit `state` for all records, or use `open`, `done`, or `expired`.
The CLI accepts the same `--state`, `--limit`, and `--cursor` options and
prints the next cursor when older records remain.

After making a change yourself, record it with
`client.pools.recommendation_done(pool_id, recommendation_id, outcome)` or:

```sh
nodus pools mark-done POOL_ID RECOMMENDATION_ID \
  --outcome "Reduced capacity in our scheduler"
```

The returned `RecommendationOutcome` is explicitly customer-reported. Leave
`reported_saving_micros` unset when your saving is unknown. If you have an
independently assessed amount, pass nonnegative integer USD micros in Python
or add `--reported-saving-micros` in the CLI. Neither the SDK nor the console
copies an estimated saving into your reported outcome. Recording an outcome
does not execute a host action.

All these Python methods are also available on `AsyncClient.pools` and must
be awaited. Forecasts return `PoolForecast`, recommendations return
`PoolRecommendations`, and their unchanged response JSON is available in
`raw`. Subscription changes return `Pool`.


## Enable Route with explicit consent

An account admin can enable Route at **$0.02 per active customer device-hour**,
including optimization and apply. Your private hosts have no supplier rental
charge. Market capacity has separate compute charges. Enabling Route does not
create a Predict subscription or change an observe host's execution permission.

First choose an existing host from `client.pools.hosts(pool_id)`. Use the
Compute Hosts panel's **Enable execution** action to obtain a fresh token and
pinned installation command. Run that command in a root Bash shell on the
same Linux host. The installation rotates the host credential and installs
the execution service. You can prepare execution hosts before enabling Route.
The equivalent token request is explicit:

```python
token = client.pools.enrollment_token(
    pool_id,
    mode="execute",
    host_id=host_id,
)
```

Tokens are single-use secrets. Use `token.token` only when providing it to
the installation prompt. Do not log it or put it in command history. A token
response does not mean the host has installed execution support.

After reviewing the rate, enable Route:

```python
pool = client.pools.set_route(
    pool_id,
    True,
    accepted_rate_version="route-platform-v1",
    accepted_rate_micros=20000,
)
```

The same consent through the CLI is:

```bash
nodus pools route POOL_ID on \
  --accept-rate-version route-platform-v1 \
  --accept-rate-micros 20000
```

Disabling with `client.pools.set_route(pool_id, False)` or
`nodus pools route POOL_ID off` stops new admission. Existing work, accepted
terms, and exact cleanup remain tracked. Send Route changes separately from
Predict, pool name, and owned hardware cost updates.

`update_route_settings` accepts the following optional fields. Supply at least
one. These fields may also accompany `set_route` in one request.

| Field | Values |
|---|---|
| `wait_policy` | `never` keeps waiting for private capacity and never uses market fallback. `after_wait` allows fallback after waiting. `cheaper` may allow early fallback with active, paid Predict and usable forecast evidence |
| `wait_alpha` | Finite number at least zero. New pools default to 0.1 |
| `waiting_budget_pct` | Number from 0 through 100 |
| `burst_approval` | `auto`, `above_threshold`, or `always` |
| `burst_threshold_micros` | Nonnegative USD micros |
| `burst_timeout_behaviour` | `keep_waiting` or `cancel` |

The `cheaper` policy compares a current market quote's expected cost to completion
with the pool's forecast opportunity cost. It requires enabled, funded Predict,
a positive owned hardware cost, a trusted runtime estimate of at most 30 days,
and a ready forecast no more than two hours old. Missing or stale evidence does
not authorize early fallback. Burst approval and workload spending controls
still apply.

```python
client.pools.update_route_settings(
    pool_id,
    wait_policy="after_wait",
    wait_alpha=0.1,
    burst_approval="always",
)
```

## Choose private or market placement

Omit `placement` to prefer eligible private capacity. Set one pool explicitly,
or use `prefer="any"` to skip private pools. Do not set both fields.

```python
workload = client.run(
    command=["python", "train.py"],
    gpu_count=1,
    budget=5,
    placement=nodus.Placement(pool=pool_id),
)
```

Use `placement=nodus.Placement(prefer="any")` for market capacity. Your image,
GPU requirements, spending controls, and output selection still apply.
Unavailable, disabled, or inaccessible explicit pools are rejected. Accepted
submission does not mean execution has started. Observe progress and retrieve
results as for other workloads. The same arguments work with `AsyncClient`.

## Burst approval inbox

A burst proposal requests market fallback for one submitted workload stage and
execution generation. It requires Route, but not Predict. An account admin can
approve or reject the immutable proposed amount. Approval records intent and
does not itself rent capacity. Nodus rechecks the current quote, Route authority,
spending controls, and original expiry before admitting new work. The quoted
expected cost is not an absolute billing cap. Workload spending controls remain
separate.

```python
page = client.pools.proposals(pool_id, state="pending", limit=25)
for proposal in page.proposals:
    print(proposal.id, proposal.expected_cost_micros, proposal.expires_at)
```

Follow `page.next_cursor` with the same pool and state filter to read older
proposals. Amounts use USD micros. Review the server-reported amount before
calling `client.pools.approve_proposal(pool_id, proposal_id)` or
`client.pools.reject_proposal(pool_id, proposal_id)`.

`approved` means recorded intent. `applying` means the same execution generation
claimed that approval. Only `applied` means a matching winning execution was
observed. `expired`, `rejected`, and `no_op` retain their reason and do not silently
renew approval. An execution dispatched before expiry may be observed afterward.
A timeout does not prove a decision failed. Refresh the inbox before retrying.

The same methods are available on `AsyncClient.pools` with `await`.


---

Document: https://nodus-compute.ai/docs/guides/rl-runs/
Markdown source: https://nodus-compute.ai/docs/source/guides/rl-runs.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/rl-runs.md

# Run your own RL code or a prepared recipe

You can start with your own training command and optional data. You do not need
to select a catalog environment. To show reported RL task progress, add the
optional `rl` metadata to a normal run:

```python
def launch_custom_rl(
    client, *, image, source_asset_id, budget, stable_run_id,
    model_label, planned_tasks,
):
    if not isinstance(stable_run_id, str) or not stable_run_id:
        raise ValueError("stable_run_id must be a nonempty stable key for this run")
    if budget is None:
        raise ValueError("budget must be an explicit spending limit for this run")
    return client.run(
        image=image,
        source_asset_id=source_asset_id,
        command=["python", "train.py"],
        outputs={"results": "outputs"},
        budget=budget,
        compute_class="accelerator",
        idempotency_key=stable_run_id,
        extra={
            "rl": {
                "schema_version": 1,
                "environment_id": "custom",
                "mode": "train",
                "model": model_label,
                "planned_tasks": planned_tasks,
            }
        },
    )
```

Use an imported source asset containing your `train.py`, a compatible runtime
image and an explicit spending limit. The command must write final results
under `outputs` and write and load its own checkpoint state. Adjust the command
and output path to match your project.

The metadata describes your experiment. Your code implements the trainer,
model, task limit and task-event reporting. Use `mode="evaluate"` for evaluation
without training. Custom runs cannot set `rl.recipe` or claim managed recipe
validation. A completed command without task events has no reported RL score.

## Use a prepared recipe

RL recipes describe supported evaluation and training runs. Recipe availability
reflects operator qualification and current launch settings. Admission and
capacity are checked again at launch, so an available recipe can still fail to
launch.

Start by finding the recipe you intend to use and checking its current status:

```python
import nodus

def find_recipe(client):
    recipes = client.rl.list_recipes()
    recipe = next(
        (
            item for item in recipes
            if item.id == "reasoning-gym-leg-counting"
        ),
        None,
    )

    if recipe is None:
        raise RuntimeError("The requested RL recipe is not offered")
    if not recipe.available:
        reason = recipe.unavailable_reason or "No reason was provided"
        raise RuntimeError(f"RL recipe is unavailable: {reason}")
    return recipe
```

Build the configuration explicitly, then ask the server to normalize and
review it. Preview does not launch a workload.

```python
def prepare_run(client, recipe):
    configuration = {
        "recipe_id": recipe.id,
        "recipe_version": recipe.version,
        "mode": "evaluate",
        "evaluation_tasks": 16,
        "training_steps": 20,
        "max_cost_usd": 5,
        "seed": 42,
        "include_traces": False,
    }

    preview = client.rl.preview(configuration)
    print(preview.estimate)
    print(preview.phases)
    print(preview.outputs)

    if not preview.launchable:
        raise RuntimeError(
            "RL run cannot launch: " + ", ".join(preview.blocking_reasons)
        )
    return preview
```

Inspect the normalized configuration, phases, outputs, and estimate before
launching. The server returns a review token for that exact preview. A changed
configuration needs a new preview and review token.

Launch only after your application or a person has accepted the preview. Use a
stable idempotency key for one intended run and store it with your own run
record.

```python
def launch_run(client, preview, stable_run_id):
    workload = client.rl.launch(
        configuration=preview.configuration,
        review_token=preview.review_token,
        idempotency_key=stable_run_id,
    )
    print(workload.id)
    return workload
```

`launch()` requires all three values. It does not select a recipe, fill in a
configuration, or generate an idempotency key. This makes the paid action
explicit and lets an application retry safely.

If a timeout or connection failure leaves the launch outcome unclear, retry
the same configuration and review token with the same idempotency key. A new
key can create another paid run. Reusing one key with a different request
raises `IdempotencyConflictError`.

Normal SDK exceptions apply:

```python
def launch_with_error_handling(client, preview, stable_run_id):
    try:
        return client.rl.launch(
            configuration=preview.configuration,
            review_token=preview.review_token,
            idempotency_key=stable_run_id,
        )
    except nodus.IdempotencyConflictError:
        raise RuntimeError("The idempotency key belongs to a different request")
    except nodus.NodusError as error:
        raise RuntimeError(f"RL launch failed: {error}") from error
```

The asynchronous client has the same flow:

```python
async def prepare_async(client, configuration):
    recipes = await client.rl.list_recipes()
    recipe = next(
        (item for item in recipes if item.id == configuration["recipe_id"]),
        None,
    )
    if recipe is None or not recipe.available:
        raise RuntimeError("The requested RL recipe is unavailable")
    preview = await client.rl.preview(configuration)
    if not preview.launchable:
        raise RuntimeError(
            "RL run cannot launch: " + ", ".join(preview.blocking_reasons)
        )
    return preview

async def launch_async(client, preview, stable_run_id):
    return await client.rl.launch(
        configuration=preview.configuration,
        review_token=preview.review_token,
        idempotency_key=stable_run_id,
    )
```

## Read task evidence and grading receipts

Use `client.rl.events()` for reported RL task evidence. `workload.events()`
returns separate workload lifecycle events.

```python
def read_available_tasks(client, workload_id, cursor=None):
    while True:
        page = client.rl.events(workload_id, after=cursor, limit=100)
        for row in page.events:
            event = row.event
            print(event.task_id, event.outcome, event.reward)
        if page.truncated or page.dropped_events:
            print("Task evidence is partial", page.dropped_events)
        cursor = page.next_cursor
        if not page.has_more:
            return cursor
```

Pass `next_cursor` unchanged as `after`. Keep the returned cursor for the next
read. A row ID is a separate string identity and must not be used as a cursor.
Recovery can repeat an application event identity in a later execution
generation. Keep each row and inspect its `generation` instead of deduplicating
across generations.
An empty page or `has_more=False` means no more rows were available in that
snapshot. Continue polling while the workload is active and read again after
observing its terminal status. Task events are application-reported evidence.
They do not independently prove model quality or workload completion.

For a workload with a server-admitted private grading plan, request receipts
for the explicit revision you submitted:

```python
def read_grading(client, workload_id, revision):
    results = client.rl.grading_results(workload_id, revision=revision)
    for receipt in results.receipts:
        print(receipt.task_id, receipt.state, receipt.reward)
    return results

async def read_evidence_async(client, workload_id, revision, cursor=None):
    page = await client.rl.events(workload_id, after=cursor)
    grading = await client.rl.grading_results(workload_id, revision=revision)
    return page, grading
```

Receipt rewards are `None` when absent and `0` for a measured zero. An
infrastructure failure does not supply a model reward. Cleanup fields describe
the grading attempts in the selected revision and do not establish parent
cleanup or final billing. A missing grading plan raises `NotFoundError`.
Reading receipts does not enable private grading for a workload.


---

Document: https://nodus-compute.ai/docs/guides/services/
Markdown source: https://nodus-compute.ai/docs/source/guides/services.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/guides/services.md

# HTTP agent services

A service is a sandbox with one managed server command. It uses the sandbox
budget, lifetime and hourly meter. The server binds a loopback HTTP port inside
the sandbox. Nodus forwards authenticated requests over the runtime relay.
There is no inbound guest networking.

After authenticating, replace the image below with your qualified server image
and choose a budget. The example budget is illustrative:

```python
image = "your-account/inference:qualified"
budget_usd = 5
with nodus.Client() as client:
    box = client.sandboxes.create(
        image=image,
        service={
            "command": ["python", "server.py"],
            "port": 8080,
            "health_path": "/health",
        },
        budget=budget_usd,
        lifecycle={"max_lifetime_s": 90000, "idle_timeout_s": 1800},
        idempotency_key="inference-service-deployment-001",
    )
    print(box.id)
```

The image must already contain your server. Wait for the sandbox to become ready
before calling it. `box.request("POST", "/infer", port=8080, json=payload)` returns
response bytes. Use `json.loads(response)` for JSON or `response.decode("utf-8")`
for text. Send binary bodies with `content=payload_bytes` instead of `json=`.
Both synchronous and asynchronous service methods return bytes, including
`b""` for an empty response. The server
must return a successful 2xx response at its health path before an inference
request is forwarded.

The HTTPS API route is `/v1/sandboxes/{id}/ports/{port}/{path}` on your deployment
API origin. Send the normal Nodus bearer token. The route forwards the request
method, path, query, content type and body. Authorization and Cookie headers
never reach the guest. Other custom headers are not forwarded. Bodies are
limited to 64 KiB. WebSockets and streaming responses are not supported.
HTML responses run with an isolated sandbox origin.

HTTP calls have a 15 second relay deadline and are never automatically retried.
A timeout can mean the application received the request. Include an application
idempotency key in the JSON body before retrying a mutation. A suspended service
returns `service_waking` and starts recovery. That request was not sent to the
application. Retry after the sandbox becomes ready.

A stopped server restarts after a 30 second delay, up to 120 attempts per
runtime generation. Host recovery starts the server command again. Applications
must save and load their own state from `NODUS_CHECKPOINT_DIR`. This does not
restore process memory or make external tool calls exactly once.

Terminate a service with `box.terminate()` and verify its terminal state.
Read charges with `client.ledger(box.id)`. Passing local tests does not establish
24 hours of production availability.


---

Document: https://nodus-compute.ai/docs/
Markdown source: https://nodus-compute.ai/docs/source/index.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/index.md

# Run GPU workloads and agent sandboxes with Nodus

Run training, fine-tuning, and batch experiments that need GPU capacity beyond
your local machine. Submit your command from Python or a workload file, follow
its progress, and retrieve logs and output files through the same interface.
For interactive agents, create a durable sandbox and execute multiple commands
with streamed output and stdin.
Nodus uses qualified estimates of runtime cost when every eligible configuration
has comparable measurements. Otherwise it orders compatible on-demand
configurations by hourly price. Spending limits and independent price limits
apply in both cases.
Set a workload budget to limit spending. Optimization tiers are not supported.

Install [nodus-compute from PyPI](https://pypi.org/project/nodus-compute/)
with Python 3.10 or newer, then sign in to get started.

## For coding agents

Start with [Connect your coding agent](https://nodus-compute.ai/docs/guides/connect/) for Claude Code,
Codex, Cursor and other clients. It includes install commands, agent skills,
and a connection check that does not start paid compute.

Follow the [coding agent guide](https://nodus-compute.ai/docs/guides/agents/) to prepare workloads, observe
their progress, and collect results. Use the [parameter reference](https://nodus-compute.ai/docs/reference/parameters/)
for supported arguments and the [OpenAPI specification](https://nodus-compute.ai/docs/openapi.yaml)
for HTTP schemas. Set a budget, keep the workload ID, and check `succeeded` after
waiting. An accepted workload is not necessarily a completed workload.

## Get started

1. [Install and sign in](https://nodus-compute.ai/docs/getting-started/authentication/)
2. [Run your first workload](https://nodus-compute.ai/docs/quickstart/#2-run-your-first-workload)
3. [Read logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/)

## Run your code

- [Run a Python script](https://nodus-compute.ai/docs/guides/containers-and-scripts/)
- [Attach code and datasets](https://nodus-compute.ai/docs/guides/assets/)
- [Manage external data connections](https://nodus-compute.ai/docs/guides/connections/)
- [Train or fine-tune a model](https://nodus-compute.ai/docs/guides/gpu-workloads/)
- [Run from a workload file](https://nodus-compute.ai/docs/getting-started/workload-files/)
- [Run an agent sandbox](https://nodus-compute.ai/docs/guides/agent-sandboxes/)
- [Run from GitHub Actions and automation tools](https://nodus-compute.ai/docs/guides/automation/)
- [Connect MCP tools](https://nodus-compute.ai/docs/guides/mcp/)
- [Install Codex, Claude Code and Cursor plugins](https://nodus-compute.ai/docs/guides/plugins/)

## Go further

- [Concurrent experiments](https://nodus-compute.ai/docs/guides/async-sweeps/)
- [Stages and downloadable files](https://nodus-compute.ai/docs/guides/multi-stage-workloads/)
- [CI and safe retries](https://nodus-compute.ai/docs/guides/ci-and-idempotency/)

## Reference

- [Python client](https://nodus-compute.ai/docs/reference/python/client/)
- [Versioned workload and draft operations](https://nodus-compute.ai/docs/guides/operations/)
- [Terminal commands](https://nodus-compute.ai/docs/reference/cli/)
- [Workload parameters](https://nodus-compute.ai/docs/reference/parameters/)
- [GPU models and resources](https://nodus-compute.ai/docs/reference/parameters/requirements/#gpu-model)
- [Troubleshooting](https://nodus-compute.ai/docs/operations/errors/)


---

Document: https://nodus-compute.ai/docs/operations/errors/
Markdown source: https://nodus-compute.ai/docs/source/operations/errors.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/operations/errors.md

# Errors and troubleshooting

API and transport errors inherit `nodus.NodusError`. Inspect `.status_code`,
`.code`, `.payload`, and `.request_id`. Include the request ID when reporting an
issue. Python argument mistakes (`TypeError` / `ValueError`) are separate.

| Error | Action |
|---|---|
| `ConfigurationError` | Run `nodus login` or fix the reported configuration problem |
| `AuthenticationError` | Check deployment/key pairing, expiry, or revocation |
| `SignatureError` | Check signing secret and clock for signed requests |
| `ValidationError` | Correct the rejected request field |
| `IdempotencyConflictError` | Reuse the original payload or assign a new logical key |
| `NotFoundError` | Check workload ownership/ID. Logs may not yet be committed |
| `BudgetExceededError` | For `payment_method_required`, add a card in Billing. Otherwise inspect headroom and account limits |
| `SpendCheckUnavailableError` | Spending authorization is temporarily unavailable. Retry using the same submission key |
| `RateLimitError` | Pace requests. SDK honors bounded retry-after delays |
| `CapacityUnavailableError` | Retry later or relax feasible workload constraints |
| `APIConnectionError` / `APITimeoutError` | Preserve submission key. Outcome may be unknown |
| `AssetInUseError` | Finish or cancel dependent workloads before deleting the asset |
| `APIError` | Inspect HTTP status and response payload |

SDK request retries are finite. Capacity becoming available is not guaranteed.
A network exception does not prove a submission failed to reach the server.
See [idempotency](https://nodus-compute.ai/docs/guides/ci-and-idempotency/) and
[retry policy](https://nodus-compute.ai/docs/concepts/reliability/).

## Common first-run problems

- **Payment method required:** add a card in [Billing](https://console.nodus-compute.ai/?view=billing).
  This applies even when starter credits remain. Shared workspace members should
  ask their administrator. HTTP 402 with code `payment_method_required` uses the
  existing `BudgetExceededError` exception type.

- **Script not found:** put it in the container image and use an absolute path.
- **No bootstrap tool:** include `curl`, `wget`, or `python3` in the image.
- **No log yet:** inspect lifecycle events and retry when logs become available.
- **Login changed nothing:** environment variables override saved credentials.
- **Wait returned but work failed:** inspect `succeeded`, events, and logs.
- **Cancellation unconfirmed:** run `nodus cancel WORKLOAD_ID` and inspect status.
  A lost connection or force-killed process cannot confirm remote cleanup.
- **Download failed:** ensure the destination parent exists, choose a declared
  output name/stage, and retry. Integrity failures leave existing files intact.
- **Missing output methods:** upgrade with
  `pip install --upgrade nodus-compute`.

An omitted budget prints a short notice only in an interactive terminal. It
does not emit `UserWarning`. A known image without a bootstrap fetch tool can
still emit `UserWarning` before submission.

## Backend compatibility

The SDK and backend must support the same features. Upgrading the Python package
does not deploy backend changes. Login verification requires `GET /v1/me`, and
live log streaming requires `GET /v1/workloads/{id}/logs/live`. A 404 from these
routes can mean the deployment lacks the feature, even when saved credentials,
workload history, and committed logs still work.

GPU enforcement and spending limits also require their matching backend support.
An older server may ignore fields it does not recognize. Successful submission
alone does not prove those constraints were applied. Confirm support with the
deployment operator before relying on a specific GPU or a hard spending cap.

If login verification is unavailable, preserve the saved credentials and retry
after the backend is updated. Use committed logs to inspect output when live
streaming is unavailable. Optimization tiers are not supported.
Existing optimization arguments are accepted for compatibility
but have no preference effect on new runs.


---

Document: https://nodus-compute.ai/docs/reference/cli/
Markdown source: https://nodus-compute.ai/docs/source/reference/cli.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/cli.md

# Terminal commands

Use `nodus --help` for command groups and `nodus COMMAND --help` for options.
Replace `ID` with a workload ID. Use the installed command help to confirm
which capabilities your SDK version provides.

## Setup

| Command | What it does |
|---|---|
| `nodus mcp` | Start the local MCP server using your saved login. Requires `nodus-compute[mcp]` |
| `nodus login` | Reuse a valid login or open browser sign-in |
| `nodus login --force` | Start a fresh browser sign-in |
| `nodus logout` | Remove the locally saved key |
| `nodus init` | Create a starter `nodus.toml` without submitting work |

For headless machines and automation, see [authentication](https://nodus-compute.ai/docs/getting-started/authentication/).

## Run

| Command | What it does |
|---|---|
| `nodus run` | Submit `nodus.toml` and wait for completion |
| `nodus run train.toml` | Submit another file and wait |
| `nodus submit train.toml` | Submit and print the ID without waiting |
| `nodus list` | List your workloads |
| `nodus list active` | List active workloads |
| `nodus list mine` | List workloads attributed to your member login |
| `nodus list team` | List workloads across your team |

Personal history requires a member-associated login. Shared keys can use team history.

`submit` also defaults to `nodus.toml` when no path is given.
`list --limit N` accepts 1 through 100.
Set your image, command, budget, and advanced options in a
[workload file](https://nodus-compute.ai/docs/getting-started/workload-files/).

## Monitor and collect results

| Command | What it does |
|---|---|
| `nodus status ID` | Show status and current cost |
| `nodus wait ID` | Wait for a terminal status |
| `nodus logs ID` | Print committed logs |
| `nodus download ID` | Download published result files and archives under `outputs/ID` |
| `nodus cancel ID` | Request cancellation and remote cleanup |

Interactive waits show lifecycle events, live logs, elapsed time, and reported
training progress. Redirected output has no animation. `nodus logs` retrieves
saved log snapshots. Declare files as outputs to download them.
`logs --tail N` selects the last N lines, with 0 meaning all lines.
`logs --generation N` selects an attempt number starting at 1.

Ctrl+C during `run`, `wait`, or `events --follow` requests cancellation. Cleanup
happens remotely after acceptance. If the request fails, the CLI reports that
cancellation is unconfirmed and prints `nodus cancel ID`. A second Ctrl+C stops
the cancellation attempt. A wait timeout ends observation without cancelling.

If submission ends with an uncertain outcome, the CLI prints a recovery key.
Add that `idempotency_key` to the same workload file before retrying. Keep its
other settings unchanged to avoid submitting duplicate work.

## Agent sandboxes

SDK 0.5.1 accepts active names or exact sandbox IDs for the commands below.
Older releases require IDs for sandbox commands.

| Command | What it does |
|---|---|
| `nodus sandbox new IMAGE --name NAME --budget USD` | Admit a sandbox and print its ID while startup continues |
| `nodus sandbox ls` | Show sandbox IDs, names, states and costs |
| `nodus sandbox exec NAME_OR_ID COMMAND` | Run a command and stream its output |
| `nodus sandbox logs NAME_OR_ID EXEC_ID` | Read all currently stored command output |
| `nodus sandbox cost NAME_OR_ID` | Read the reported cost |
| `nodus sandbox rm NAME_OR_ID` | Request termination of the existing sandbox |

Name lookup selects one active exact match in your account. It never creates a
replacement. Use an ID for a terminated sandbox or when a name is ambiguous.
UUID-shaped sandbox IDs are treated as exact IDs even when absent. If you used
an ID-shaped name, use the actual ID returned at creation.
An accepted create request does not confirm runtime readiness, and accepting
termination does not confirm that remote cleanup has finished.

`sandbox new`, `sandbox exec` and `sandbox rm` accept
`--idempotency-key`. If a mutation has an uncertain outcome, preserve the printed
key and retry the unchanged operation. Use the printed sandbox ID when available.
For `exec`, put options before the sandbox reference so they are not interpreted
as part of the remote command.

If output observation fails after a command is accepted, the CLI prints its
execution ID and a `sandbox logs` command. Resume observation with that command
instead of submitting `exec` again.

```bash
nodus sandbox exec --idempotency-key tool-call-001 research-agent python agent.py
```

## Code and datasets

| Command | What it does |
|---|---|
| `nodus upload FILE` | Upload a file or archive and print its asset ID |
| `nodus assets` | List stored assets |
| `nodus asset get ID` | Inspect one asset and its safe export error |
| `nodus asset import-query CONNECTION SQL` | Export a database query and wait for its asset |

See [code and datasets](https://nodus-compute.ai/docs/guides/assets/) for imports and attaching assets to work.

## Customer-owned compute

| Command | What it does |
|---|---|
| `nodus pools create NAME` | Register a customer-owned host pool |
| `nodus pools token POOL_ID` | Print a secret single-use enrollment token |
| `nodus pools token POOL_ID --mode execute --host-id HOST_ID` | Print a secret token for explicit reenrollment of one existing host |
| `nodus pools route POOL_ID off` | Disable new private admission while retaining cleanup |
| `nodus pools route-settings POOL_ID --wait-policy after_wait --wait-alpha 0.1` | Update future placement policy |
| `nodus pools hosts POOL_ID` | Inspect enrolled hosts |
| `nodus pools utilization POOL_ID --json` | Read measured utilization and host buckets |
| `nodus pools forecast POOL_ID --horizon 7 --json` | Read cached forecast evidence and the subscription rate |
| `nodus pools recommendations POOL_ID --state open --limit 25 --json` | Read one page of advice, following `--cursor` for older records |
| `nodus pools predict POOL_ID off` | Disable paid refresh for one pool |
| `nodus pools mark-done POOL_ID RECOMMENDATION_ID --outcome TEXT` | Record a manual outcome without executing a host action |

Observe is free. Predict activation requires explicit consent to its account
monthly charge. See [customer-owned pools](https://nodus-compute.ai/docs/guides/pools/) for activation,
renewal, cached reads while paused, and optional reported savings. Enrollment
tokens are secrets and must not be written to shared logs.

## Advanced diagnostics

| Command | What it does |
|---|---|
| `nodus events ID` | Execution event history |
| `nodus artifacts ID` | Artifact manifests |
| `nodus explain ID` | Selected route and cost estimate |
| `nodus ledger ID` | Billing entries and settlement |

Use `nodus --debug COMMAND` for technical error details. Use command help for JSON output, polling, stage selection, and other diagnostic
options. Agents can use the [Python client](https://nodus-compute.ai/docs/reference/python/client/) for structured
results without parsing terminal output.

| Exit code | Meaning |
|---|---|
| 0 | Command succeeded |
| 1 | Failed/cancelled workload, unavailable logs, or route not yet selected |
| 2 | API/configuration error or invalid CLI usage |
| 130 | Interrupted with Ctrl+C |

## Upgrading from 0.1

Replace `nodus get ID` with `nodus status ID`, and `nodus get ID --wait` with
`nodus wait ID`. Submission flags have moved into workload files. Use `nodus run`
to submit and wait, or `nodus submit` to return immediately. Python `client.get()`
and `client.run()` keep their existing behavior.

### Burst proposals

`nodus pools proposals POOL_ID` reads retained burst intent. Optional `--limit`
accepts 1 to 100, `--cursor` follows the returned continuation, and `--state`
filters `pending`, `approved`, `rejected`, `expired`, `no_op`, `applying`, or
`applied`. Use `--json` for the typed public response.

`nodus pools approve POOL_ID PROPOSAL_ID` approves the immutable proposed amount.
`nodus pools reject POOL_ID PROPOSAL_ID` rejects pending intent. These commands
require a current account admin. Approval does not itself rent capacity and does
not change the original expiry. Inspect the amount with `proposals` first.

### Database output load state

`nodus workload outputs WORKLOAD_ID` lists output names, sizes and database sink
load state. Add `--json` for the API fields. Retry a saved sink output with
`--reload NAME` and add `--stage STAGE` when output names repeat across stages.
Loading happens independently of workload completion.


---

Document: https://nodus-compute.ai/docs/reference/parameters/continuity/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/continuity.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/continuity.md

# Continuity and recovery

`continuity` accepts a string, `nodus.ContinuityMode`, or dictionary.
Workload files accept a `continuity` string or a `[continuity]` table.

| Mode | Intended application behavior | Default `resume_on_interruption` |
|---|---|---|
| `checkpointed` | Restore progress from a committed checkpoint | `true` |
| `restartable` | Safely repeat unfinished work. Supported unit boundaries can preserve progress | `true` |
| `ephemeral` | Accept loss of this attempt | `false` |

Omitting continuity sends `{"mode": "checkpointed", "resume_on_interruption": true}`.
Dictionary input also defaults a missing mode to `checkpointed` and a missing
resume flag according to the mode. An explicit resume flag is retained.

```python
continuity = {"mode": "restartable", "resume_on_interruption": True}
```

These values select a recovery policy. They do not instrument arbitrary code.
A training program needs a supported integration or its own save and restore code.
Do not assume a default checkpoint mode guarantees lossless recovery for any
container. Confirm your framework and deployment's runner integration before a
long training run. Use restartable for short self-contained smoke tests.

## Application checkpoint integration

The optional `integration` field selects application save and resume support for
source submissions. A stage can override the workload selection.

| Value | Behavior |
|---|---|
| `auto` | Attempt a supported integration. Unsupported training configurations retain the original command and declared file-saving behavior |
| `none` | Use the application's existing save and restore code |
| `hf-trainer-v1` | Require the versioned Hugging Face Trainer integration. Unsupported configurations fail rather than silently restarting |

The SDK preserves omission. New submissions do not request application checkpoint
preparation unless `auto` or `hf-trainer-v1` is selected explicitly. A stage inherits
an explicit workload selection when omitted. Existing accepted submissions and
their idempotent retries retain their original selection.
Explicit `auto` and `hf-trainer-v1` require checkpointed continuity and the
dedicated `state` folder. Preparation availability depends on the deployment.

`hf-trainer-v1` accepts unmodified Transformers Trainer with PyTorch 2.7.1,
Transformers 4.57.6, Accelerate 1.12.0 and Datasets 4.4.2. Its supported profile
uses one training process and device, float32 model state, a fingerprinted map-style Hugging
Face Dataset, zero dataloader workers, the default data collator and standard
built-in callbacks. Trainer creates the optimizer and scheduler. Complete
model state must contain tensors only and fit within four billion bytes.
Mixed precision, distributed training, streaming data, PEFT, quantized models,
custom Trainer subclasses, custom loss functions and TRL require separate
qualification. The profile requires `save_only_model=False`,
`ignore_data_skip=False`, `load_best_model_at_end=False` and `push_to_hub=False`.

The integration saves complete model, optimizer, scheduler, random generator
and training-step state at an optimizer boundary. Nodus requests saves and
preserves completed versions. Recovery checks the program, dataset,
configuration, model structure, trainable parameters, device type and framework
identities before resuming. An incompatible
managed checkpoint fails even when `integration` is `auto`.

Check that your GPU environment is qualified before relying on automatic recovery.
Progress after the last committed checkpoint can still be lost.

## Files saved for recovery

New submissions save only the `state` folder by default. Write model weights,
optimizer state and training progress there and load them when your program
restarts. `NODUS_CHECKPOINT_DIR` points to this folder in the code directory.
Nodus does not search other folders for training state.

When your application manages its own state, write a complete version outside
the selected paths and publish it with an atomic rename on the same filesystem.
Do not overwrite files while Nodus may be copying them. A multi-file checkpoint
needs a complete, immutable version that remains available during capture.

Use `checkpoint_paths` when your program saves recovery files elsewhere:

```python
continuity = {
    "mode": "checkpointed",
    "resume_on_interruption": True,
    "checkpoint_paths": ["checkpoints", "progress.json"],
}
```

Paths are relative to the code folder and are literal, not glob patterns.
Specify up to 64 paths, with at most 512 UTF-8 bytes each. Absolute paths,
parent traversal, control characters and the runner-private `.nodus` folder
are rejected. Missing paths are skipped without expanding the selection.

Omitting the workload list or passing an empty list selects `["state"]` on the
server. The SDK preserves the supplied list without resolving this default.
A stage inherits workload paths unless it supplies a nonempty list, even when
it sets its own continuity mode. Use `["."]` to explicitly preserve the whole
code folder, except runner-private files, for a workload or stage. This can
include dependencies and caches that your command placed in the code folder.
Selecting other paths does not change `NODUS_CHECKPOINT_DIR`.
Framework shortcuts apply workload checkpoint paths to each generated stage.
Use explicit stages without a framework shortcut when each stage needs its
own selection. Stage-specific checkpoint paths combined with a framework
shortcut are rejected.

Your program must load its saved state when it restarts. Files outside the
selection are not restored from the checkpoint. Recreate dependencies from
the image or command, and put rebuildable custom installations under
`$TMPDIR/runtime` to keep them outside checkpoint storage. Download caches and
temporary files already use runner-private locations.

Existing submitted workloads retain their saved checkpoint selection.

Final downloadable result files are configured separately with `outputs`.
They can be outside the checkpoint paths.

`interrupt_tolerance` is not an input. The control plane derives interruption
behavior from continuity. Recovery enters a nonterminal `recovering` state.
[waiting](https://nodus-compute.ai/docs/concepts/reliability/) continues through it.


---

Document: https://nodus-compute.ai/docs/reference/parameters/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/index.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/index.md

# Submission parameters

Choose the environment and command for your code, then set any GPU, memory,
budget, and recovery requirements. `Client.run()` and `AsyncClient.run()` accept
the same named arguments. The table covers every explicit submission argument
and links to its accepted values, defaults, and examples.

The SDK translates these arguments into an HTTP request. Most omitted arguments
are not sent, except the default image and continuity policy.

| Python argument | HTTP location | Reference |
|---|---|---|
| `image`, `command` | `source.image`, `source.command` | [Container image and command](https://nodus-compute.ai/docs/reference/parameters/source/) |
| `source_asset_id`, `inputs`, `outputs` | Source asset and stage file declarations | [Files](https://nodus-compute.ai/docs/reference/parameters/source/#input-and-output-files) |
| `framework` | `framework` | [Framework execution](https://nodus-compute.ai/docs/reference/parameters/source/) |
| `gpu` | `requirements.gpu` | `"A100"`, `"H100"`, `"H200"`, `"B200"`, `"A10"`, `"A10G"`, `"L4"`, `"L40"`, `"L40S"`, `"T4"`, `"V100"`, `"RTX A6000"`, `"RTX 3090"`, `"RTX 4090"`, `"RTX 5090"`. Omit to let Nodus choose. [GPU models and examples](https://nodus-compute.ai/docs/reference/parameters/requirements/#gpu-model) |
| `optimization` | `requirements.optimization` | `"automatic"`, `"lowest_cost"`, `"lower_cost"`, `"balanced"`, `"faster"`, `"fastest"`. Compatibility only, omitted by default. No preference effect on new runs. [Optimization](https://nodus-compute.ai/docs/reference/parameters/requirements/#optimization) |
| `model`, `compute_class`, `peak_memory_gb` | `requirements.*` | [Resources](https://nodus-compute.ai/docs/reference/parameters/requirements/) |
| `requirements` | `requirements` | [Resources](https://nodus-compute.ai/docs/reference/parameters/requirements/) |
| `budget`, `finish_by` | `outcome.max_cost_usd`, `outcome.complete_by` | [Budget and deadline](https://nodus-compute.ai/docs/reference/parameters/outcome/) |
| `continuity` | `continuity` | [Recovery](https://nodus-compute.ai/docs/reference/parameters/continuity/) |
| `connections`, `sweep_id` | Same top-level keys | [Live connections](https://nodus-compute.ai/docs/guides/connections/#attach-live-wandb-to-a-run) |
| `data_regions`, `policy` | `policy.data_regions`, `policy` | [Policy](https://nodus-compute.ai/docs/reference/parameters/policy/) |
| `stages` | `stages` | [Stages](https://nodus-compute.ai/docs/reference/parameters/stages/) |
| `idempotency_key` | `Idempotency-Key` header | [Safe retries](https://nodus-compute.ai/docs/guides/ci-and-idempotency/) |
| `extra` | Additional top-level fields | [Extensions](https://nodus-compute.ai/docs/reference/parameters/#extensions-and-validation) |

Do not pass `env`, `interrupt_tolerance`, or `expected_runtime_hours`: they are
explicitly unsupported. Unknown Python keywords raise `TypeError` before submission. Use the dictionary fields listed
in these references. Each page defines their accepted values and defaults.

## Extensions and validation

`extra: dict` adds top-level request fields that the deployed server models but
this SDK version does not expose. It defaults to no additions and has no CLI flag.
It cannot replace a key already built in the request, including `requirements`,
`outcome`, or `continuity`. Collisions raise `TypeError` before network access.
Prefer named arguments and documented typed fields.

A non-colliding name is not proof that a server supports it. Unknown server
fields may be ignored on older deployments. Verify support in the deployed
contract before using extensions. Do not send secrets in arbitrary metadata.

`expected_runtime_hours` is also rejected inside requirements and stage dictionaries.
Stage inputs have a separate supported shape. See [stages](https://nodus-compute.ai/docs/reference/parameters/stages/).
Python typos raise `TypeError`, distinct from a server `nodus.ValidationError`.


---

Document: https://nodus-compute.ai/docs/reference/parameters/outcome/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/outcome.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/outcome.md

# Budget and deadline

| Argument | Type | Omitted | Workload file | HTTP field |
|---|---|---|---|---|
| `budget` | Positive finite number in USD | No separate per-run limit | `budget` | `outcome.max_cost_usd` |
| `finish_by` | RFC3339 string or `datetime` | No completion deadline | `finish_by` | `outcome.complete_by` |

Budget is a hard workload spending limit, not a completion-price promise.
Available credits and any configured account spending limit also apply.

Inside a `with nodus.Client() as client:` block:

```python
from datetime import datetime, timedelta, timezone

workload = client.run(
    image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
    command=["python", "-c", "print('deadline example')"],
    budget=5,
    finish_by=datetime.now(timezone.utc) + timedelta(hours=2),
)
```

Use timezone-aware datetimes. A naive datetime is interpreted in the submitting
machine's local timezone before conversion to UTC. A string passes through for
server validation. Use a future timestamp including an offset or `Z`.

`finish_by` expresses a completion deadline. `wait(timeout_seconds=...)` only
bounds local observation and does not cancel the workload.
See [costs](https://nodus-compute.ai/docs/concepts/costs/) for observed spend and settlement.


---

Document: https://nodus-compute.ai/docs/reference/parameters/policy/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/policy.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/policy.md

# Policy and data regions

| Argument | Type | Omitted | Workload file | HTTP field |
|---|---|---|---|---|
| `data_regions` | `list[str]` | No explicit region restriction from shortcut | `data_regions` | `policy.data_regions` |
| `policy` | Dictionary | Absent unless populated | Same key or table | `policy` |

`data_regions` is a list of exact region identifiers accepted by your deployment.
An empty list adds no location restriction. There is no universal region list
or region-discovery method in this SDK. For the hosted service,
[contact Nodus](mailto:nodus.infrastructure@gmail.com) for the enabled identifiers
before setting a location restriction. For a private deployment, obtain them
from its administrator. Do not assume a cloud provider's region codes are valid.

Restricting regions narrows eligible routes and can make a workload infeasible.
The SDK forwards the identifiers without translating them.

Inside a `with nodus.Client() as client:` block:

```python
workload = client.run(
    image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
    command=["python", "-c", "print('regional workload')"],
    data_regions=[],
    budget=5,
)
```

This example adds no region restriction. If your workload requires a particular
geography, replace the empty list with the exact approved identifiers before
submitting. Do not use unrestricted execution for a location-sensitive workload.

`policy["data_regions"]` wins over the flat `data_regions` argument. Regions
belong under policy, not requirements. Omission does not promise execution in any
particular geography. Deployment and account policies may still constrain it.
`policy.secret_refs` accepts tenant secret names or IDs. Admission pins each
version and supplies it as `NODUS_SECRET_<NAME>` in the execution environment.
`policy.egress_allow` accepts HTTPS hostnames to add to a live connection's
allowlist. These fields require an isolated execution provider.

Attach a wandb connection with `connections=["lab-wandb"]` and optionally set
`sweep_id="experiment-42"` to group runs. See [live connections](https://nodus-compute.ai/docs/guides/connections/#attach-live-wandb-to-a-run)
for credential delivery, network restrictions and captured run links.


---

Document: https://nodus-compute.ai/docs/reference/parameters/requirements/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/requirements.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/requirements.md

# Resource requirements

| Argument | Type / values | Omitted | HTTP field |
|---|---|---|---|
| `model` | Free-text workload description | No model hint | `requirements.model` |
| `compute_class` | `"accelerator"` for GPU workloads | Accelerator | `requirements.compute_class` |
| `peak_memory_gb` | Positive number in GB per GPU | No explicit memory hint | `requirements.peak_memory_gb` |
| `optimization` | `"automatic"`, `"lowest_cost"`, `"lower_cost"`, `"balanced"`, `"faster"`, `"fastest"`. Compatibility only | Not sent | `requirements.optimization` |
| `gpu` | `"A100"`, `"H100"`, `"H200"`, `"B200"`, `"A10"`, `"A10G"`, `"L4"`, `"L40"`, `"L40S"`, `"T4"`, `"V100"`, `"RTX A6000"`, `"RTX 3090"`, `"RTX 4090"`, `"RTX 5090"`. [Examples and aliases](https://nodus-compute.ai/docs/reference/parameters/requirements/#gpu-model) | Nodus chooses | `requirements.gpu` |
| `gpu_count` | Exactly `1`, `2`, `4`, or `8` on one machine | One GPU | `requirements.gpu_count` |
| `gpu_interconnect` | `"any"` | No topology guarantee | `requirements.gpu_interconnect` |
| `requirements` | Dictionary | Optional resource hints | `requirements` |

The workload file uses the same argument names. You do not need to predict how
long your program will run. Provide memory only when you know the requirement.
`model` describes your workload and does not download model weights.

## Optimization

Optimization tiers are not supported. New workloads use qualified estimates of
runtime cost when every eligible configuration has comparable measurements.
Otherwise Nodus orders compatible on-demand configurations by hourly price.
Spending limits and independent price limits apply in both cases. This does not
guarantee the lowest total cost or shortest runtime.

Omit `optimization` in new code. The SDK accepts `automatic`, `lowest_cost`,
`lower_cost`, `balanced`, `faster`, and `fastest` for backward compatibility.
These values have no preference effect on new workload or stage routing.
The API records `automatic` for newly accepted workloads. Empty nested values
remain accepted for compatibility. The flat shortcut does not accept an empty
string.

GPU, memory, CPU, disk, image compatibility, location and budget requirements
remain mandatory. An explicit GPU model is never replaced by another model.
Omit `gpu` to allow more compatible models. Accepted names do not establish
available capacity.

## GPU model

`gpu` is a hard requirement. Nodus never substitutes another model, including
when retrying a run. If matching capacity is unavailable, the run reports that
condition. Omit `gpu` to let Nodus choose compatible capacity.
An accepted model name does not guarantee matching capacity. GPU, memory,
and other resource requirements must all fit an available machine.

These are all accepted canonical model names. Use the Python argument shown in
`client.run()`, or the same quoted value for `gpu` in a workload file.

| GPU model | Exact Python argument |
|---|---|
| A100 | `gpu="A100"` |
| H100 | `gpu="H100"` |
| H200 | `gpu="H200"` |
| B200 | `gpu="B200"` |
| A10 | `gpu="A10"` |
| A10G | `gpu="A10G"` |
| L4 | `gpu="L4"` |
| L40 | `gpu="L40"` |
| L40S | `gpu="L40S"` |
| T4 | `gpu="T4"` |
| V100 | `gpu="V100"` |
| RTX A6000 | `gpu="RTX A6000"` |
| RTX 3090 | `gpu="RTX 3090"` |
| RTX 4090 | `gpu="RTX 4090"` |
| RTX 5090 | `gpu="RTX 5090"` |

Names are case-insensitive. Whitespace, hyphens, and underscores are ignored. An optional `NVIDIA`
prefix and compact RTX names are accepted, such as `"nvidia h100"` and
`"RTX4090"`. `"A6000"` is an alias for `"RTX A6000"`.
These names describe models, not a guarantee of current capacity.
Choose the model family and specify memory separately, such as `gpu="A100"`
with `peak_memory_gb=80`. Supplier names and machine IDs are not GPU names.

Inside a `with nodus.Client() as client:` block:

```python
workload = client.run(
    image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
    command=["python", "-c", "import torch\nprint(torch.cuda.get_device_name(0))"],
    gpu="H100",
    budget=5,
)
```

An explicit dictionary key wins over the matching flat shortcut:
`requirements={"peak_memory_gb": 48}, peak_memory_gb=24` sends 48.
Workload files reject duplicate flat and nested settings so the choice is clear.

## Additional dictionary fields

These fields belong inside `requirements={...}` in Python or `[requirements]`
in a workload file. They are not flat `run()` arguments.

| Field | Type and units | Omitted |
|---|---|---|
| `disk_gb` | Finite nonnegative number in GB | No explicit disk requirement |
| `vcpus` | Finite nonnegative number of virtual CPUs | No explicit CPU requirement |
| `dataset_bytes` | Nonnegative integer in bytes | No dataset-size hint |
| `notes` | Text with additional workload context | No notes |

Omitted or zero disk and CPU values in a stage inherit the workload requirements.
These fields do not transfer data or install dependencies.
`nodus.Requirements(...)` provides optional static typing. The SDK validates GPU
names, compatibility values, and numeric resource bounds for both typed and
ordinary dictionaries before submission. Booleans and nonfinite numbers are
not valid resource quantities. Explicit `peak_memory_gb` must be positive.

## Multiple GPUs on one machine

Set `gpu_count=8, gpu="H100", peak_memory_gb=80` to require eight H100s on
one machine, each with at least 80 GB of memory. Supported counts are 1, 2, 4
and 8. Omission retains single-GPU behavior. A smaller allocation or a group
of machines cannot satisfy this request. Matching capacity may be unavailable.

The count does not guarantee NVLink, NVSwitch or pooled device memory.
`gpu_interconnect="any"` declares no topology constraint. Other interconnect
values are rejected because the platform cannot yet verify that guarantee.
Do not submit topology-dependent training until its topology is supported.

The displayed node hourly price covers the whole allocation. Your budget
covers the run, including all devices, rather than applying separately to
each GPU. Recovery and saved-run reuse preserve the requested count.
Nodus preserves your command arguments. Supply your own distributed launcher
and application configuration, such as `torchrun --nnodes=1 --nproc_per_node=8`.


---

Document: https://nodus-compute.ai/docs/reference/parameters/source/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/source.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/source.md

# Container image and command

`image` chooses the container environment where your code runs. A container
image packages the runtime, system libraries, and installed dependencies. For example,
`pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime` selects an environment with
PyTorch and CUDA libraries. Choose an image containing the packages your program needs.

`command` tells that environment which program to start and which arguments to
pass. In `command=["python", "train.py"]`, the first item starts Python and the
second names the script. This list is also called an argument vector, or argv.
The script must already be in the image or attached as uploaded code. Naming a
local file in `command` does not upload it.

To attach your code, upload it with `client.assets.upload()` and pass the
returned asset ID as `source_asset_id`. Nodus extracts that code into the workload
working directory. See [run your own Python script](https://nodus-compute.ai/docs/guides/containers-and-scripts/)
for a complete upload-and-run example.

In the HTTP API, `source` groups the image, command, and optional code asset.
Python callers pass `image`, `command`, and `source_asset_id` directly to
`client.run()`. The SDK builds the `source` object for you.

| Argument | Type | Default / omission | Workload file |
|---|---|---|---|
| `image` | `str` | `python:3.11-slim` for a single source | `image` |
| `command` | `list[str]` or `str` | No command is sent | `command` |
| `framework` | `"train_eval"` | Absent | `framework = "train_eval"` |

For a single-source SDK call, omitting `image` or passing an empty string selects
`python:3.11-slim`. It does not request the native runtime available when leaving
the console image field blank. For GPU workloads, pass a compatible CUDA image
with the dependencies your code needs.

Use an explicit image and command. Omitting the command is accepted by this SDK,
but is not a portable way to invoke an image entrypoint: deployment bootstrap
controls execution. It is unsuitable for a first workload.

Inside a `with nodus.Client() as client:` block:

```python
workload = client.run(
    image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
    command=["python", "-c", "print('ready')"],
    budget=5,
)
```

A string command uses `shlex.split`. It does not invoke a shell. Prefer an argv
list. Pipes, redirects, variable expansion, and `&&` need an explicit shell, such
as `command=["sh", "-c", "python preprocess.py && python train.py"]`.

Images must contain a bootstrap fetch tool (`curl`, `wget`, or `python3`), plus
your program dependencies. Upload source files explicitly with `client.assets.upload()`. `framework` is
passed through to the control plane. It does not install a framework or replace
the need to prepare runnable code. The current compiler supports `train_eval`: it runs the same command in
prepare, train, and eval stages. Code must branch on `NODUS_STAGE_ID` and honor
the declared handoffs. Prefer explicit stages when each command differs. Do not
combine `framework` with `stages`, because framework expansion takes precedence.

When `stages` is nonempty, its stage sources replace the top-level source.
Combining it with nonempty `image`, `command`, or `source_asset_id` raises `TypeError`.

## Input and output files

| Argument | Purpose |
|---|---|
| `source_asset_id` | Uploaded/imported code asset extracted into the working directory |
| `inputs` | Named asset inputs such as `[{"name": "training", "asset_id": "ASSET_ID"}]` |
| `outputs` | Declared files such as `{"model": "model.bin"}` relative to the working directory |

`outputs` creates a single stage named `main`. Do not combine it with `stages`
or `framework`. Put `source.asset_id` on each explicit stage instead of using
`source_asset_id`. Top-level asset `inputs` can also supply staged workloads. See [code and datasets](https://nodus-compute.ai/docs/guides/assets/) for asset creation and input
paths, and [logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/) for downloads.

### File declaration constraints

Use asset IDs returned by upload or import, not paths or URLs. Their format is
`asset_` followed by 1 to 64 letters, digits, or hyphens. Omitted
`source_asset_id` attaches no code asset. Omitted `inputs` attaches no named assets.

`inputs` accepts at most eight dictionaries, each containing `name` and
`asset_id`, with an optional boolean `cache` flag. Names must be unique, start with a letter, and contain at most 64
letters, digits, or underscores. For example, `training_data` is valid and
`training-data` is not. The input directory is exposed as `NODUS_INPUT_training_data`.

Set `cache: True` to allow reuse of verified input content within your team and
execution region. Cache storage uses workspace size and count limits. The first
workload that fills a cache remains its storage billing owner, including after
termination, under its existing spending cap. Storage billing stays disabled
until a storage rate is configured. An unavailable cache falls back to the
ordinary input. This flag does not stream external bucket data.

Output names use only letters, digits, dots, underscores, or hyphens. They must
be distinct without regard to case, cannot be `.` or `..`, and cannot end in a
dot. Reserved file names `CON`, `PRN`, `AUX`, `NUL`, `COM1` through `COM9`, and
`LPT1` through `LPT9` are rejected without regard to case, including names with
extensions such as `CON.txt`. Stage IDs with declared outputs follow these
same portability rules in addition to the [stage ID rules](https://nodus-compute.ai/docs/reference/parameters/stages/).

Output paths identify files inside the workload working directory. Use `/` for
subdirectories, such as `results/model.bin`. Absolute paths, backslashes,
colons, control characters, empty path components, and `.` or `..` components
are rejected. When a stage has no declared outputs, non-empty `outputs/` and
`results/` folders are automatically preserved as `outputs.tar` and `results.tar`.
Explicit output mappings replace this default. Save complete model bundles in a
default folder, or declare files elsewhere. See
[logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/) for collection exclusions
and downloads.

## Stream a bucket object

A bucket input supplies a sequential file path through `NODUS_INPUT_<name>`.
The runner reads directly from the declared regional S3 or Google Cloud Storage
endpoint while your command consumes the file. It does not import the corpus
into Nodus storage. Read through EOF to verify the declared byte count and
SHA-256 digest. Seeking and reopening the stream are not supported.
The example descriptor below represents an object containing the three bytes `abc`.
Replace its location, byte count and digest with your own object metadata.

```python
job = client.run(
    command=["python", "train.py"],
    budget=2,
    data_regions=["us-east-1"],
    inputs=[{
        "name": "corpus",
        "bucket": {
            "uri": "s3://training-bucket/corpus.jsonl",
            "region": "us-east-1",
            "bytes": 3,
            "sha256": "ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad",
            "credential_source": "team_webhook",
        },
    }],
)
```

Use `gs://bucket/object` and the exact Google Cloud region for GCS. The execution
region must exactly match the input region. Do not put signed URLs, access keys,
or tokens in the workload definition. Bucket inputs do not accept `cache`.
Imported assets and OCI image layers use the separate optional digest cache.

A team administrator must configure the existing team webhook to handle a
synchronous `input.credentials.request`. Nodus signs this request using the
webhook timestamp and HMAC headers. The request identifies the workload, stage,
generation, input name and object descriptor. Return HTTP 200 with the same
`object` and a `credentials` object containing `expires_at` and either S3
`access_key_id`, `secret_access_key`, `session_token`, or GCS `access_token`.
Scope the temporary credential to reading the declared object and set its
expiry within one hour. Never log the response body.

Nodus keeps the credential only in memory and transfers it to the authenticated
runner over TLS. A failed, expired, unread or corrupt stream prevents successful
completion. Recovery requests a fresh credential and starts the stream from the
beginning. Your program remains responsible for loading its own saved progress
and skipping data already processed. Data received before EOF is not yet fully
verified against the declared digest.

## Database result sinks

An output value can be a path string or a dictionary containing `path` and
`sink`, where `sink` contains `connection` and `table`. String paths retain their
existing behavior. Database sinks support CSV, JSONL and flat Parquet files.
See [database result loading](https://nodus-compute.ai/docs/guides/connections/#load-results-into-a-database)
for types, limits, generation replacement and reload.


---

Document: https://nodus-compute.ai/docs/reference/parameters/stages/
Markdown source: https://nodus-compute.ai/docs/source/reference/parameters/stages.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/parameters/stages.md

# Stage parameters

`stages` is a list of dictionaries or `nodus.StageSpec` values. A nonempty list
replaces the top-level source. Use Python or `[[stages]]` entries in a
[workload file](https://nodus-compute.ai/docs/getting-started/workload-files/). Do not combine `framework` with explicit stages: a recognized
framework takes precedence in the current compiler.

| Stage field | Type | Omission / purpose |
|---|---|---|
| `id` | Unique string, 1–64 characters | Required. Letters, digits, `_`, `-`, `.`. Cannot start with `.` or `-` |
| `source` | `{image: str, command: list[str], asset_id?: str}` | Give explicit executable argv and image. Missing image defaults to Python image server-side |
| `depends_on` | List of stage IDs | Empty. Dependency edges must be acyclic |
| `inputs` | List of input references below | Empty |
| `outputs` | Mapping from logical name to relative output path. [Name and path constraints](https://nodus-compute.ai/docs/reference/parameters/source/#file-declaration-constraints) | Empty. Files must actually be produced |
| `requirements` | Requirements dictionary | Zero/empty fields inherit workload-level values |
| `continuity` | `nodus.ContinuitySpec` | Missing/empty mode inherits workload mode and resume behavior. Missing/empty `checkpoint_paths` inherits workload paths |
| `total_units` | Nonnegative integer progress-unit count | `0`. Describes units for compatible restartable work, not GPUs or replicas |

A stage-specific nonempty `continuity.mode` does not receive the SDK top-level
resume default: provide `resume_on_interruption` explicitly. Setting only that
flag without a mode does not override inherited mode and resume behavior.

A nonempty stage `checkpoint_paths` list overrides workload paths independently
of mode. New workloads default to `["state"]`. Use `["."]` to opt a stage into
preserving the whole code folder. See [checkpoint paths](https://nodus-compute.ai/docs/reference/parameters/continuity/#files-saved-for-recovery).

| Input field | Meaning |
|---|---|
| `name` | Local logical input name |
| `from_stage` | Upstream stage ID. Also include it in `depends_on` |
| `from_output` | Declared output name on the upstream stage |

Always declare producer outputs and complete input references so a typo can be
caught before execution. Output paths are relative to the runner's working
directory. Runtime integrations expose resolved inputs through `NODUS_INPUT_<name>`.
The value is a local file path, not the original storage URI.

Stage requirements support `model`, `compute_class`, `dataset_bytes`,
`peak_memory_gb`, `disk_gb`, `vcpus`, `gpu`, and `notes`. The `optimization`
field remains accepted for compatibility but has no preference effect on new runs. See [resources](https://nodus-compute.ai/docs/reference/parameters/requirements/) and the complete
[multi-stage example](https://nodus-compute.ai/docs/guides/multi-stage-workloads/).


---

Document: https://nodus-compute.ai/docs/reference/python/client/
Markdown source: https://nodus-compute.ai/docs/source/reference/python/client.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/python/client.md

# Python client reference

`Client(api_key=None, base_url=None, timeout=30.0, max_retries=2)` pools HTTP
connections. With no arguments it uses your saved login and the hosted service.
Prefer a `with` block. Otherwise call `close()`.
`AsyncClient` uses `async with` or `await aclose()` and mirrors the methods below.

| Method | Result / behavior |
|---|---|
| `run(**brief)` | Accepted `Workload`. [all parameters](https://nodus-compute.ai/docs/reference/parameters/) |
| `run_file(path="nodus.toml")` | Accepted `Workload` from a [workload file](https://nodus-compute.ai/docs/getting-started/workload-files/) |
| `assets` | [Upload, import, list, and delete code or dataset assets](https://nodus-compute.ai/docs/guides/assets/) |
| `operations` | [Typed version 1 workload and draft operations with contract discovery](https://nodus-compute.ai/docs/guides/operations/) |
| `pools` | [Measure customer-owned GPU hosts, review forecasts, and configure Route](https://nodus-compute.ai/docs/guides/pools/) |
| `sandboxes` | [Create, reconnect to, list, and control agent sandboxes](https://nodus-compute.ai/docs/guides/agent-sandboxes/) |
| `get(id)` | Refreshed `Workload` |
| `list(limit=50, offset=0, status=None, scope=None)` | One page of workloads |
| `list_page(limit=50, offset=0, status=None, scope=None)` | `(workloads, next_offset)` |
| `iter_workloads(page_size=50, status=None, scope=None)` | Iterator over offset-based pages |
| `wait(id, poll_seconds=2.0, timeout_seconds=None, progress=None, on_update=None)` | Terminal workload. Inspect `succeeded` |
| `cancel(id, idempotency_key=None)` | Request cancellation. Returns `None` |
| `events(id, after=0)` | One page of `Event` objects |
| `iter_events(id, after=0)` | Iterator over event history |
| `stream_events(id, poll_seconds=2.0)` | Poll events until terminal |
| `artifacts(id)` | List of `Artifact` manifests |
| `logs(id, stage=None, generation=None)` | Committed log text |
| `live_logs(id, after="")` | Live log chunks, cursor, and truncation state |
| `outputs(id)` | List of `Output` objects |
| `download_output(id, name, destination, stage=None, overwrite=True)` | Verified local `Path` |
| `routing(id)` | Placement-history dictionaries ordered by stage ID and generation |
| `ledger(id)` | `Ledger` |
| `set_webhook(url, secret=None)` | Webhook configuration response dictionary |
| `get_webhook()` / `delete_webhook()` | Read configuration / remove it |
| `healthz()` / `readyz()` | Deployment health/readiness dictionaries |

Optional settings after resource IDs are keyword-only. For `download_output`,
`name` and `destination` can also be positional. Status filters accept
`nodus.WorkloadStatus` members, strings, comma-separated strings, or lists.
Accepted status strings are `accepted`, `planning`, `reserving`, `provisioning`,
`running`, `recovering`, `completed`, `failed`, and `cancelled`. The `active`
preset selects nonterminal states and `terminal` selects `completed`, `failed`,
and `cancelled`. Omit `status` for no status filter.
Unknown statuses raise `ValueError`. Pagination uses offsets. Concurrent new
submissions can shift pages. It is not a consistent historical snapshot.

## Sandbox resources

`nodus.Sandbox(name=..., image=...)` is the direct get-or-create form. Reusing
the same name reattaches without an image. It is a context manager that
terminates on exit. Call `close()` to release only the local HTTP client while
keeping the remote sandbox alive.

`client.sandboxes.create(...)` accepts an image, resource requirements, a
budget, network policy, lifecycle, reservation, and continuity settings. It
returns an accepted `Sandbox` handle. Read `sandbox.state` or call
`sandbox.refresh()` before assuming the environment is ready.

`client.sandboxes.from_id(ID)` reconnects to a sandbox. `list()` returns one
cursor-based page and `list_page()` also returns `next_cursor`. Iterating over
`client.sandboxes` follows every page. The asynchronous client provides the
same methods and `client.sandboxes.iterate()`.

`sandbox.exec(command, ...)` accepts shell command text or an argument vector,
queues a process, and returns `SandboxExec`.
`execution.iter_output()` yields ordered `SandboxOutputFrame` objects with
`stream`, `data`, and decoded `text`. `execution.write(data, eof=False)` sends
stdin when it was enabled at execution creation. `execution.wait()` returns
when the process is terminal. Check `succeeded` and `exit_code`. Call
`sandbox.terminate()` to stop future execution and request resource cleanup.

Sandbox calls use idempotency keys for create, exec, stdin, and terminate.
Supply a stable key when your application retries after an uncertain response.
Generated keys protect the SDK transport retries within one method call.
If create, exec, or terminate returns an invalid receipt, the SDK raises
`APIError` with the sent key in `error.payload["idempotency_key"]`. Reuse that
key with the original arguments to recover the same operation.

`Workload` offers `refresh`, `wait`, `cancel`, events, logs, artifacts, outputs,
download, routing, and ledger methods without repeating the ID. Reads and waits
with `refresh()` and `wait()` update it in place. Useful attributes are `id`, `status`,
`succeeded`, `is_terminal`, `route`, `stages`, `meter`, `cost_now_usd`, `links`, and `raw`.
Each `WorkloadLink` has `kind` and `url`. Captured wandb links are available before completion.
Unknown server enum values remain strings for forward compatibility.

`workload.download(destination=None)` downloads all published customer outputs,
including automatically collected folder archives when no files were declared,
and returns a list of local `Path` objects. The default directory is
`outputs/WORKLOAD_ID`, with each file at `STAGE/NAME`. `await workload.download()` is the asynchronous equivalent.
Use `download_output(name, destination, stage=...)` for one specific file.

## Method arguments

| Argument | Meaning and default |
|---|---|
| `timeout` | HTTP request timeout in seconds, default `30.0`. Separate from a workload deadline or wait timeout. [Retry behavior](https://nodus-compute.ai/docs/concepts/reliability/#retry-behavior) |
| `max_retries` | Additional request attempts, default `2`, giving up to three total attempts. Downloads do not retry automatically |
| `limit`, `page_size` | Workloads requested per page, default `50` |
| `offset` | Number of workloads to skip, default `0`. `list_page()` returns the next offset, or `None` at the end |
| `poll_seconds` | Finite nonnegative seconds between successful polls, default `2.0` |
| `timeout_seconds` | Finite nonnegative local wait duration in seconds, default `None` for no deadline. A timeout leaves the workload running |
| `on_update` | Optional synchronous callback called with each successful workload read during `wait()`, including the terminal read. Available on sync and async clients |
| `progress` | `None` detects an interactive terminal, `True` enables output, `False` waits silently. [Live display](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/#live-display) |
| `events(after)`, `iter_events(after)` | Numeric sequence of the last event seen, default `0`. Returns events with later `seq` values, oldest first. `events()` returns at most 100 per page |
| `live_logs(after)` | Opaque `next_cursor` string from the previous response, default `""` for the first page. This is not an event sequence. [Live log response](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/#live-display) |
| `logs(stage)` | Stage ID to select, default `None` for no stage filter |
| `logs(generation)` | Stage attempt number to select, default `None` for no generation filter. Use a positive generation from the returned artifacts or live logs |
| `download_output(name)` | Declared output name, not its path in the container |
| `download_output(destination)` | Local file path with an existing parent directory |
| `download_output(stage)` | Stage ID to disambiguate an output name published by multiple stages, default `None` |
| `download_output(overwrite)` | `True` replaces the destination only after integrity verification. `False` refuses an existing target |
| `idempotency_key` | Stable key for a logical submission or cancellation. Omission creates a fresh key per call. [Character rules and retries](https://nodus-compute.ai/docs/guides/ci-and-idempotency/) |
| `scope` | `"mine"` or `"team"`. Omission sends no scope filter. [Personal and team history](https://nodus-compute.ai/docs/reference/python/client/#personal-and-team-history) |

For all declared files, `workload.download()` creates directories and refuses
to overwrite existing files. Use a new destination directory for another copy.

## Models

| Type | Useful fields |
|---|---|
| `Event` | `seq`, `id`, `type`, `payload`, `created_at` |
| `StageRun` | `id`, `status`, `completed_units`, `total_units`, optional `last_loss`, `metric_rate`, `metric_step`, `metric_total_steps`, `metric_epoch`, `metric_total_epochs` |
| `Artifact` | `manifest_id`, `stage_id`, `generation`, `sequence`, `final`, `files`, `outputs` |
| `ManifestFile` | `uri`, `sha256`, `bytes`, `media`, `is_tar` |
| `Output` | `name`, `stage_id`, `sha256`, `bytes`, `download` |
| `Route` | `sku`, `compute_class`, `fit_class`, `region`, `memory_gb`, `resources`, prices and estimated cost |
| `Meter` | `settled_usd`, `accruing_usd`, `total_now_usd`, `accruing_rate_usd_hour`, `as_of`, `compute_settled_usd`, `platform_fee_settled_usd`, `subscription_settled_usd`, `compute_accruing_usd`, `platform_fee_accruing_usd` |
| `Ledger` | `entries`, `charged_usd`, `settlement` |

`Event` has `type` and `payload`, not a `message` attribute. Output download
helpers use the authenticated API endpoint. Treat returned `download` as
server metadata rather than a URL to which you should forward credentials.

`Route.sku` is a catalog identifier, not a GPU model. When available,
`route.resources.get("accelerator")` reports the device model and
`route.resources.get("device_memory_gb")` reports its memory in GB. Missing
metadata does not prove that no GPU was used. The terminal shows `Not reported`
when it cannot identify the compute from the response. List responses may omit
the route, so use `client.get(ID)` or `workload.refresh()` for current details.

## Optional typed request dictionaries

`Source`, `Requirements`, `Policy`, `ContinuitySpec`, `StageInput`, and `StageSpec`
are `TypedDict` helpers exported by `nodus`. They support autocomplete and static
analysis while producing ordinary dictionaries:

```python
import nodus

requirements = nodus.Requirements(compute_class="accelerator", peak_memory_gb=24)
source = nodus.Source(image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime", command=["python", "-c", "print(1)"])
stage = nodus.StageSpec(id="example", source=source)
# Supply requirements= and stages=[stage] to client.run(..., budget=5).
```

They do not add runtime validation or defaults. Existing plain dictionaries remain
supported. Stage source commands are argv lists, not shell strings.

## Personal and team history

Use `client.list(scope="mine")` for your submissions or `scope="team"` for the
team. Scope also works with `list_page()` and `iter_workloads()` and combines
with status filters. Personal history requires a member-associated credential.
Listed workloads expose `owner_user_id`, which can be absent for shared keys or
older submissions. Scope filters history and does not change team access.


---

Document: https://nodus-compute.ai/docs/reference/python/secrets/
Markdown source: https://nodus-compute.ai/docs/source/reference/python/secrets.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/reference/python/secrets.md

# Tenant secrets

`client.secrets.put(name, value)` creates a new tenant secret version and returns
its ID, name, version and creation time. `client.secrets.list()` returns current
metadata without values. `client.secrets.delete(name)` retires the current
version. The same methods are available through `AsyncClient` with `await`.

Pass `secrets=["API_KEY"]` to `client.sandboxes.create()` to bind those names at
boot. Commands receive each value as an environment variable and as a file named
for the secret under `NODUS_SECRETS_DIR`. Names must be environment variable
names outside the reserved `NODUS_` prefix. A sandbox can select at most 32 names,
and each value can contain up to 4096 UTF-8 bytes without NUL characters.

Writing a new version leaves running sandbox bindings unchanged.
`sandbox.refresh_secrets()` binds current versions for subsequent commands.
Already running processes keep their environment. A wake binds current versions.
Retired versions are destroyed after seven days once no live generation remains
bound to them. A live generation keeps its bound versions for injection and output
redaction until it ends.

Stored and streamed command output replaces matching secret values with
`[redacted:NAME]`. Possible secret fragments at output frame boundaries are
conservatively redacted too, so a matching fragment of ordinary text can be
masked. Passing a retained secret value directly in an exec `env` is rejected.

Injected files reside in memory-backed storage outside the workspace and
recovery state. Customer code must keep secret values out of files it saves.

Use `policy={"secret_refs": ["API_KEY"]}` when creating a sandbox to pin the
current version at admission. A returned `sec_` ID can also be used. Commands
receive this value as `NODUS_SECRET_API_KEY`. Rotation, revocation, recovery, and
`refresh_secrets()` preserve this admission pin. Revocation prevents new
admissions from selecting the secret. Pinned ciphertext remains available until
the sandbox terminates, including while it is recovering or suspended.

The CLI reads exact UTF-8 input without stripping a trailing newline. Values are
never accepted as command arguments or printed in normal command output.

```bash
nodus secret set API_KEY --from-file /path/to/private-key
nodus secret ls
nodus secret rm API_KEY
```

You can also pipe the value on stdin to `nodus secret set API_KEY`.


---

Document: https://nodus-compute.ai/docs/unit-metrics/
Markdown source: https://nodus-compute.ai/docs/source/unit-metrics.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/unit-metrics.md

# Per-unit measurements

Print one complete JSON line when your command finishes a logical unit of work:

```python
import json

print("nodus.unit_done " + json.dumps({"id": "batch-42", "ms": 125.5}), flush=True)
```

Keep each unit ID stable if recovery repeats the same work. IDs must contain at
most 128 UTF-8 bytes. Durations must be finite, nonnegative milliseconds.

Read measurements from the workload returned by the service:

```python
workload = client.get(workload_id)
metrics = workload.unit_metrics
if metrics is not None:
    print(metrics.units_completed, metrics.p50_ms, metrics.p95_ms)
    print(metrics.cost_per_unit_usd, metrics.dropped_observations)
```

Cost per unit uses posted workload charges. It can change while charges settle.
It is not a final price estimate. Missing measurements remain `None`.

Measurements arrive periodically. A host failure can lose observations that have
not reached the service. A full local queue reports dropped observations, so
these measurements do not prove that every completed unit was counted.

These lines report performance only. They do not save application state or
advance the recovery position.


---

Document: https://nodus-compute.ai/docs/workspaces/
Markdown source: https://nodus-compute.ai/docs/source/workspaces.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/docs/workspaces.md

# Named workspaces

Create a workspace and attach it to a sandbox to preserve selected files between sandbox identities.

```python
with nodus.Client() as client:
    client.workspaces.create("research", size_gb=0.1)
    box = client.sandboxes.create(
        image="python:3.12",
        workspace={"name": "research", "mount": "/workspace"},
    )
    command = box.exec(["sh", "-c", "echo ready > /workspace/progress.txt"])
    command.wait()
    box.terminate()
```

One live sandbox can write a workspace. A conflict includes its current holder ID. Termination can return while the final save is pending. Observe the sandbox until it becomes terminated before attaching the same workspace to another sandbox. List workspace metadata with `client.workspaces.list()` and check `last_error` and `saved_at`.

A workspace can select another dedicated top-level mount. System directories cannot be used. Workspace contents and application recovery state are separate.

Periodic saves preserve the latest useful archive. An empty folder does not replace an earlier useful archive. Hard spending and lifetime cutoffs preserve the last successful save. They cannot guarantee files written after that save. Files must fit the configured workspace capacity.

Storage billing is disabled unless the deployment has a configured price. The metadata reports `disabled_no_approved_storage_rate` or `metered_subject_to_account_limits`. Account and workload spending limits still apply. The asynchronous client exposes the same workspace methods.


---

Document: https://nodus-compute.ai/docs/quickstart/
Markdown source: https://nodus-compute.ai/docs/source/quickstart.md
Repository source: https://github.com/nodus-compute/Nodus-sdk-python/blob/13906e802146fd7682a4fa5b1bebd71e54779c4f/README.md

# Quickstart

## 1. Install and sign in

```bash
pip install nodus-compute
nodus login
```

Get [nodus-compute on PyPI](https://pypi.org/project/nodus-compute/).
Requires Python 3.10 or newer. Upgrading an existing installation? Use
`pip install --upgrade nodus-compute`. These docs cover SDK 0.6.0.

Your browser opens Nodus sign-in. Sign in and approve the code matching your
terminal. You can then close the tab. The terminal finishes automatically and
saves your credentials. Python clients use that login without extra setup.

Running `nodus login` again reuses a valid login. Use `nodus login --force` for
a fresh sign-in.

For a machine without a browser, use `nodus login --no-browser`.
See [authentication](https://nodus-compute.ai/docs/getting-started/authentication/) for API keys and
custom deployments.

Before starting a workload, open [Billing](https://console.nodus-compute.ai/?view=billing)
and add a payment method. New accounts start with $30 in credits, but a card is
required to use them. Adding a card does not purchase credits. If you joined
a shared workspace, its administrator manages the payment method.

## 2. Run your first workload

This GPU smoke test prints the available GPU name. No local script is uploaded.
It submits paid compute with a $5 workload budget. Available capacity and account
limits still determine admission.

Save this as `first_workload.py`:


```python
import nodus

with nodus.Client() as client:
    workload = client.run(
        image="pytorch/pytorch:2.8.0-cuda12.8-cudnn9-runtime",
        command=[
            "python", "-c",
            "import torch\n"
            "assert torch.cuda.is_available()\n"
            "print(torch.cuda.get_device_name(0))",
        ],
        budget=5,
    )
    print("Workload:", workload.id)
    done = workload.wait()
    print(done.status, done.cost_now_usd)
    if not done.succeeded:
        raise RuntimeError(f"Workload {done.id} ended: {done.status}")
    print(done.logs())
```

Run it with `python first_workload.py`. It prints the workload ID and shows live logs, lifecycle events, and elapsed
time while waiting. Training workloads also show reported steps or epochs.
The final output includes status, current cost, and GPU name.

`run()` accepts the workload. `wait()` waits for a terminal status, so check
`succeeded` before using results. Ctrl+C while waiting requests cancellation
and remote resource cleanup.

The script prints the GPU name from the workload logs. For files produced by
your own program, see [logs and results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/).

## Run an agent in a sandbox

A sandbox is a durable execution environment with its own public API. It is
separate from a training or batch workload. Name it once, execute multiple
commands, stream ordered stdout and stderr frames, and reconnect with the same
name from another process.

This example requires a deployment with CPU sandbox preview enabled. Replace
the image with your published agent image, including a non-root `USER` and a
writable working directory.

```python
import nodus

with nodus.Sandbox(
    name="research-agent",
    image="ghcr.io/your-org/research-agent:1",
    requirements={"vcpus": 2, "peak_memory_gb": 4, "disk_gb": 10},
    budget=5,
) as sandbox:
    print("Sandbox:", sandbox.id)
    process = sandbox.exec("python -c \"print('agent tool finished')\"")
    for frame in process.iter_output():
        print(frame.stream, frame.text, end="")

    done = process.wait()
    if not done.succeeded:
        raise RuntimeError(f"Command ended: {done.state}")
```

Creating a sandbox can start paid infrastructure. Nodus checks the account
payment method, account headroom, and sandbox budget before billable placement.
Calling `nodus.Sandbox(name="research-agent")` reconnects to the named sandbox.
See the [sandbox guide](https://nodus-compute.ai/docs/guides/agent-sandboxes/).

The CLI mirrors the same resource and verbs.

In SDK 0.5.1, use the active name `research-agent` or the returned sandbox ID
for `NAME_OR_ID`. Use the returned ID with older releases. See
[CLI retry guidance](https://nodus-compute.ai/docs/reference/cli/#agent-sandboxes)
before retrying a request whose outcome is uncertain.

```bash
nodus sandbox new ghcr.io/your-org/research-agent:1 --name research-agent --budget 5
nodus sandbox ls
nodus sandbox exec NAME_OR_ID "python -c 'print(2 + 2)'"
nodus sandbox cost NAME_OR_ID
nodus sandbox rm NAME_OR_ID
```

## Prefer the terminal?

```bash
nodus init
nodus run
```

`init` creates `nodus.toml` with the GPU smoke test and a $5 budget. Review the
file, then `run` submits it and waits for completion. Edit the image, command,
and budget to run your own workload. See [workload files](https://nodus-compute.ai/docs/getting-started/workload-files/).

```bash
nodus status WORKLOAD_ID
nodus logs WORKLOAD_ID
nodus cancel WORKLOAD_ID
```

## Automatic capacity selection

Nodus uses qualified estimates of runtime cost when every eligible configuration
has comparable measurements. Otherwise it orders compatible on-demand
configurations by hourly price. Spending limits and independent price limits
apply in both cases. This does not guarantee the lowest total cost or shortest
runtime. Optimization tiers are not supported.
Existing optimization arguments remain accepted for backward compatibility
but have no preference effect on new runs.

Set `gpu="H100"` to require a GPU model, or omit it to let Nodus choose.
No runtime estimate is needed. See [resource options](https://nodus-compute.ai/docs/reference/parameters/requirements/).

GPU enforcement, live logs, login verification, and spending limits require a
compatible Nodus backend. Installing the SDK alone does not enable these server
features. See [backend compatibility](https://github.com/nodus-compute/Nodus-sdk-python/blob/main/docs/operations/errors.md#backend-compatibility) before relying on them with a custom or older deployment.

## Run your own code

Upload your script with `client.assets.upload()` or package it in a container.
Choose an image with your dependencies and pass its command to `client.run()`.

- [Run a Python script](https://nodus-compute.ai/docs/guides/containers-and-scripts/)
- [Attach code and datasets](https://nodus-compute.ai/docs/guides/assets/)
- [Train or fine-tune a model](https://nodus-compute.ai/docs/guides/gpu-workloads/)
- [Read logs and download results](https://nodus-compute.ai/docs/guides/monitoring-and-outputs/)
- [Use Nodus with a coding agent](https://nodus-compute.ai/docs/guides/agents/)
- [Measure customer-owned GPU hosts](https://nodus-compute.ai/docs/guides/pools/)
- [Run tool-driven agents in sandboxes](https://nodus-compute.ai/docs/guides/agent-sandboxes/)

For individual options, use the [Python reference](https://nodus-compute.ai/docs/reference/python/client/)
and [parameter reference](https://nodus-compute.ai/docs/reference/parameters/).
See [troubleshooting](https://nodus-compute.ai/docs/operations/errors/) if a run fails.

## Contributing

See [RELEASING.md](https://github.com/nodus-compute/Nodus-sdk-python/blob/main/RELEASING.md) for release steps. Licensed under [Apache-2.0](https://github.com/nodus-compute/Nodus-sdk-python/blob/main/LICENSE).

## Benchmark a workload

`client.benchmark()` accepts an API workload payload, `gpu_families`, `batch_sizes`, `regions`, `repetitions`, an explicit `budget`, and an explicit `idempotency_key`. Reuse the same key after an uncertain response. Both synchronous and asynchronous clients return the server report.

The server divides one total cap into fixed cell allocations. Unused allocations are not redistributed. Use `{{batch_size}}` in a command argument when varying batch size. Inspect the returned workload IDs, posted ledger costs and measurements with `client.get_benchmark(id)`.

`nodus benchmark run request.json --idempotency-key customer-attempt` accepts the API JSON shape with `workload`, `matrix` and `budget_usd`. `nodus benchmark get bm_ID` prints the report. These commands require a backend with the benchmark API.

See [durable steps](https://github.com/nodus-compute/Nodus-sdk-python/blob/main/docs/durable-steps.md) for serial recorded-result replay on deployments with the capability enabled.

### Action policies and shadow readiness

`client.pools.action_policies(pool_id)` reads all four per-kind settings and the
Act kill switch. Use `set_action_policy` with explicit `kind`, `level`,
`window_cron` and `parallelism_cap` to save one policy. The sync and async clients
support the same methods. The CLI provides `pools action-policies`,
`pools action-policy` and `pools act-kill-switch`.

Approve and auto require funded Predict and active Route with current consent.
The kill switch remains available after entitlement loss. Enabling it blocks new
Act authorization while preserving cleanup. Saving a policy does not execute an
action. Predict is needed to produce recommendations.

`start_shadow` starts a future 168-hour observation cycle for an explicit policy.
`shadow_runs` and `pools shadows` expose trusted hours, elapsed gaps and
counterfactual action counts. Follow `next_cursor` with the same pool and kind.
The response reports which action kinds currently have a trusted shadow
producer. Missing observations remain gaps. Generic completed evidence does not qualify automatic actions.
A qualified cycle is evidence readiness, not permission to execute, and
counterfactual counts are neither measured savings nor completed actions.

UTC maintenance windows use five fields. Day-of-month and month must be `*`.
Minute, hour and weekday accept integers, lists, inclusive ranges or `*`.
Weekday 0 means Sunday. Steps, names and macros are unsupported.

### Act approvals and observed outcomes

`client.pools.action_proposals(pool_id)` reads retained Act proposals with optional
`kind`, `limit` and `cursor`. `approve_action_proposal` and
`reject_action_proposal` submit only the proposal identity. The CLI equivalents
are `pools action-proposals`, `pools approve-action` and `pools reject-action`.
Burst approvals remain under `pools proposals`.

Approval records intent. Dispatch checks current permissions, funding, policy,
expiry and evidence again. The inbox preserves pending, approved, applying,
uncertain, applied, failed, no-op and expired states. Only a server-observed
outcome confirms application. Measured savings remain null when unavailable and
are distinct from customer-reported savings. Default policies become approve
when funded Predict and Route are active, while explicit per-kind overrides stay
in force. Auto still requires a recent matching trusted shadow cycle.

The Route `cheaper` waiting policy needs funded Predict and current forecast
evidence of lower expected market completion cost. Missing evidence keeps the
workload waiting. Wait-tuning advice uses complete Route coverage and settled
execution outcomes to suggest bounded changes for future waits. It makes no
saving or completion-time guarantee.

### Freeze and resume saved work

`client.freeze(workload_id)` requests a freeze of checkpointed batch work with a
useful saved checkpoint and a compatible runner. `client.freeze_status` reports
whether saving and exact compute cleanup have completed. `client.resume` starts
resumption only after the workload is frozen. Each method is also available on a
`Workload` and through the async client. The CLI provides `freeze`,
`freeze-status` and `resume` with a workload ID.

The workload states `freezing` and `frozen` are nonterminal. A freeze request does
not immediately stop billing for an unresolved compute resource. The response
reports retained checkpoint bytes. Retained storage is not separately metered,
so `storage_charge_micros` is null, not an inferred zero.

Resume restarts the same customer command with saved checkpoint files. Your
training program must load its model, optimizer and progress from those files.
This does not restore arbitrary process memory or add guessed resume flags.

Observed Act monetary outcomes identify their `measurement_basis`. The basis
`observed_platform_fee_reduction_30m_v1` compares Route platform fees over equal
30-minute windows. It is not total infrastructure saving or a causal estimate.

## MCP clients

For Codex, Claude Code and Cursor, install the
[Nodus plugin](https://github.com/nodus-compute/Nodus-sdk-python/blob/main/docs/guides/plugins.md)
to add the MCP tools and setup guidance together.

Sign in once, then connect Claude, Cursor, Codex or another MCP client:

```sh
uvx --from 'nodus-compute[mcp]==0.6.0' nodus login
```

```json
{
  "mcpServers": {
    "nodus": {
      "command": "uvx",
      "args": ["--from", "nodus-compute[mcp]==0.6.0", "nodus-mcp"]
    }
  }
}
```

Install [uv](https://docs.astral.sh/uv/getting-started/installation/) if needed.
The public package starts the server and reuses your saved login. See
[MCP setup and the seven tools](https://nodus-compute.ai/docs/guides/mcp/)
for Codex setup, pip installation and examples.