> ## Documentation Index
> Fetch the complete documentation index at: https://docs.sinas.co/llms.txt
> Use this file to discover all available pages before exploring further.

# System

> Workers, sandbox containers, and dependencies

Sinas has a dual-execution model for functions, plus dedicated queue workers for async job processing.

#### Sandbox Containers

The sandbox container pool is a set of **pre-warmed, generic Docker containers** for executing untrusted user code. This is the default execution mode for all functions (`shared_pool=false`).

**How it works:**

* On startup, the pool creates `sandbox_min_size` containers (default: 4) ready to accept work.
* When a function executes, a container is acquired from the idle pool, used, and returned.
* Containers are recycled (destroyed and replaced) after `sandbox_max_executions` uses (default: 100) to prevent state leakage between executions.
* If a container errors during execution, it's marked as tainted and destroyed immediately.
* A background replenishment loop monitors the idle count and creates new containers whenever it drops below `sandbox_min_idle` (default: 2), up to `sandbox_max_size` (default: 20).
* Health checks run every 60 seconds to detect and replace dead containers.

**Isolation guarantees:**

Each container runs with strict resource limits and security hardening:

| Constraint           | Default                                           |
| -------------------- | ------------------------------------------------- |
| Memory               | 512 MB (`MAX_FUNCTION_MEMORY`)                    |
| CPU                  | 1.0 cores (`MAX_FUNCTION_CPU`)                    |
| Disk                 | 1 GB (`MAX_FUNCTION_STORAGE`)                     |
| Execution time       | 300 seconds (`FUNCTION_TIMEOUT`)                  |
| Temp storage         | 100 MB tmpfs at `/tmp`                            |
| Capabilities         | All dropped, only `CHOWN`/`SETUID`/`SETGID` added |
| Privilege escalation | Disabled (`no-new-privileges`)                    |

**Runtime scaling:**

The pool can be scaled up or down at runtime without restarting the application:

```bash theme={null}
# Check current pool state
GET /api/v1/containers/stats
# → {"idle": 2, "in_use": 3, "total": 5, "max_size": 20, ...}

# Scale up for high load
POST /api/v1/containers/scale
{"target": 15}
# → {"action": "scale_up", "previous": 5, "current": 15, "added": 10}

# Scale back down (only removes idle containers — never interrupts running executions)
POST /api/v1/containers/scale
{"target": 4}
# → {"action": "scale_down", "previous": 15, "current": 4, "removed": 11}
```

**Package installation:**

When new packages are approved, existing containers don't have them yet. Use the reload endpoint to install approved packages into all idle containers:

```bash theme={null}
POST /api/v1/containers/reload
# → {"status": "completed", "idle_containers": 4, "success": 4, "failed": 0}
```

Containers that are currently executing are unaffected. New containers created by the replenishment loop automatically include all approved packages.

#### Shared Containers

Functions marked `shared_pool=true` run in **persistent shared containers** instead of sandbox containers. This is an admin-only option for trusted code that benefits from longer-lived containers.

**Differences from sandbox:**

|                    | Sandbox Containers                  | Shared Containers                       |
| ------------------ | ----------------------------------- | --------------------------------------- |
| **Trust level**    | Untrusted user code                 | Trusted admin code only                 |
| **Isolation**      | Per-request (recycled after N uses) | Shared (persistent containers)          |
| **Lifecycle**      | Created/destroyed automatically     | Persist until explicitly scaled down    |
| **Scaling**        | Auto-replenishment + manual         | Manual via API only                     |
| **Load balancing** | First available idle container      | Round-robin across workers              |
| **Best for**       | User-submitted functions            | Admin functions, long-startup libraries |

**When to use `shared_pool=true` (shared containers):**

* Functions created and maintained by admins (not user-submitted code)
* Functions that import heavy libraries (pandas, scikit-learn) where container startup cost matters
* Performance-critical functions that benefit from warm containers

**Management:**

```bash theme={null}
# List workers
GET /api/v1/workers

# Check count
GET /api/v1/workers/count
# → {"count": 4}

# Scale workers
POST /api/v1/workers/scale
{"target_count": 6}
# → {"action": "scale_up", "previous_count": 4, "current_count": 6, "added": 2}

# Reload packages in all workers
POST /api/v1/workers/reload
# → {"status": "completed", "total_workers": 6, "success": 6, "failed": 0}
```

#### Queue Workers

All function and agent executions are processed asynchronously through Redis-based queues (arq). Two separate worker types handle different workloads:

| Worker               | Docker service | Queue                   | Concurrency    | Retries               |
| -------------------- | -------------- | ----------------------- | -------------- | --------------------- |
| **Function workers** | `queue-worker` | `sinas:queue:functions` | 10 jobs/worker | Up to 3               |
| **Agent workers**    | `queue-agent`  | `sinas:queue:agents`    | 5 jobs/worker  | None (not idempotent) |

**Function workers** dequeue function execution jobs, route them to either sandbox or shared containers, track results in Redis, and handle retries. Failed jobs that exhaust retries are moved to a **dead letter queue** (DLQ) for inspection and manual retry.

**Agent workers** handle chat message processing — they call the LLM, execute tool calls, and stream responses back via Redis Streams. Agent jobs don't retry because LLM calls with tool execution have side effects.

**Scaling** is controlled via Docker Compose replicas:

```yaml theme={null}
# docker-compose.yml
queue-worker:
  command: python -m arq app.queue.worker.WorkerSettings
  deploy:
    replicas: ${QUEUE_WORKER_REPLICAS:-2}

queue-agent:
  command: python -m arq app.queue.worker.AgentWorkerSettings
  deploy:
    replicas: ${QUEUE_AGENT_REPLICAS:-2}
```

Each worker sends a **heartbeat** to Redis every 10 seconds (TTL: 30 seconds). If a worker dies, its heartbeat key auto-expires, making it easy to detect dead workers.

**Job status tracking:**

```bash theme={null}
# Check job status
GET /jobs/{job_id}
# → {"status": "completed", "execution_id": "...", ...}

# Get job result
GET /jobs/{job_id}/result
# → {function output}
```

Jobs go through states: `queued` → `running` → `completed` or `failed`. Stale or orphaned jobs can be cancelled via the admin API:

```bash theme={null}
# Cancel a running or queued job
POST /api/v1/queue/jobs/{job_id}/cancel
# → {"status": "cancelled", "job_id": "..."}
```

Cancellation updates the Redis status to `cancelled` and marks the DB execution record as `CANCELLED`. It also publishes to the done channel so any waiters unblock. This is a soft cancel — it does not kill running containers.

Results are stored in Redis with a 24-hour TTL.

#### System Endpoints

Admin endpoints for monitoring and managing the Sinas deployment. All require `sinas.system.read:all` or `sinas.system.update:all` permissions.

**Health check:**

```bash theme={null}
GET /api/v1/system/health
```

Returns a comprehensive health report:

* **`services`** — All Docker Compose containers with status, health, uptime, CPU %, and memory usage. Infrastructure containers (redis, postgres, pgbouncer) are listed first, followed by application containers sorted alphabetically. Sandbox and shared worker containers are included.
* **`host`** — Host-level CPU, memory, and disk usage (read from `/proc` on Linux).
* **`warnings`** — Auto-generated alerts at three levels:
  * `critical` — No queue workers running, or infrastructure services (redis, postgres, pgbouncer) down
  * `warning` — Non-infrastructure services down, unhealthy containers, DLQ items, queue backlog >50, disk/memory >90%
  * `info` — Disk/memory >75%

**Container restart:**

```bash theme={null}
POST /api/v1/system/containers/{container_name}/restart
# → {"status": "restarted", "container": "sinas-backend"}
```

Restarts any Docker container by name (15-second timeout). Returns 404 if the container doesn't exist.

**Flush stuck jobs:**

```bash theme={null}
POST /api/v1/system/flush-stuck-jobs
```

Cancels all jobs that have been stuck in `running` state for over 2 hours. Useful for recovering from worker crashes or orphaned jobs.

#### Dependencies (Python Packages)

Functions can only use Python packages that have been approved by an admin. This prevents untrusted code from installing arbitrary dependencies.

**Approval flow:**

1. Admin approves a dependency (optionally pinning a version)
2. Package becomes available in newly created containers and workers
3. Use `POST /containers/reload` or `POST /workers/reload` to install into existing containers

```
POST   /api/v1/dependencies              # Approve dependency (admin)
GET    /api/v1/dependencies              # List approved dependencies
DELETE /api/v1/dependencies/{id}         # Remove approval (admin)
```

Optionally restrict which packages can be approved with a whitelist:

```bash theme={null}
# In .env — only these packages can be approved
ALLOWED_PACKAGES=requests,pandas,numpy,redis,boto3
```

#### Configuration Reference

**Container pool:**

| Variable               | Default | Description                                |
| ---------------------- | ------- | ------------------------------------------ |
| `POOL_MIN_SIZE`        | 4       | Containers created on startup              |
| `POOL_MAX_SIZE`        | 20      | Maximum total containers                   |
| `POOL_MIN_IDLE`        | 2       | Replenish when idle count drops below this |
| `POOL_MAX_EXECUTIONS`  | 100     | Recycle container after this many uses     |
| `POOL_ACQUIRE_TIMEOUT` | 30      | Seconds to wait for an available container |

**Function execution:**

| Variable                          | Default | Description                      |
| --------------------------------- | ------- | -------------------------------- |
| `FUNCTION_TIMEOUT`                | 300     | Max execution time in seconds    |
| `MAX_FUNCTION_MEMORY`             | 512     | Memory limit per container (MB)  |
| `MAX_FUNCTION_CPU`                | 1.0     | CPU cores per container          |
| `MAX_FUNCTION_STORAGE`            | 1g      | Disk storage limit               |
| `FUNCTION_CONTAINER_IDLE_TIMEOUT` | 3600    | Idle container cleanup (seconds) |

**Workers and queues:**

| Variable                     | Default | Description                         |
| ---------------------------- | ------- | ----------------------------------- |
| `DEFAULT_WORKER_COUNT`       | 4       | Shared workers created on startup   |
| `QUEUE_WORKER_REPLICAS`      | 2       | Function queue worker processes     |
| `QUEUE_AGENT_REPLICAS`       | 2       | Agent queue worker processes        |
| `QUEUE_FUNCTION_CONCURRENCY` | 10      | Concurrent jobs per function worker |
| `QUEUE_AGENT_CONCURRENCY`    | 5       | Concurrent jobs per agent worker    |
| `QUEUE_MAX_RETRIES`          | 3       | Retry attempts before DLQ           |
| `QUEUE_RETRY_DELAY`          | 10      | Seconds between retries             |

**Packages:**

| Variable                     | Default   | Description                                     |
| ---------------------------- | --------- | ----------------------------------------------- |
| `ALLOW_PACKAGE_INSTALLATION` | true      | Enable pip in containers                        |
| `ALLOWED_PACKAGES`           | *(empty)* | Comma-separated whitelist (empty = all allowed) |