> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xynthis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Model server

> Run a local model behind an OpenAI-compatible endpoint on your machine.

`xynthis-llm-serve` runs a language model on your machine and exposes it through an OpenAI-compatible HTTP API at `http://127.0.0.1:8080/v1`. Anything that speaks the OpenAI API (an SDK, an editor integration, `curl`) can use it by changing one base URL. Nothing leaves your machine.

```sh theme={null}
xynthis-llm-serve
# xynthis-llm-serve listening on http://127.0.0.1:8080/v1
```

The first run downloads the model weights (roughly 1-3 GB depending on the backbone) from Hugging Face and caches them. Later starts load from the cache.

## Endpoints

| Method | Path                   | Purpose                                                                             |
| ------ | ---------------------- | ----------------------------------------------------------------------------------- |
| `POST` | `/v1/chat/completions` | Chat, OpenAI request/response shape. Set `"stream": true` for SSE.                  |
| `GET`  | `/v1/models`           | Lists the served model (`xynthis-local` by default).                                |
| `GET`  | `/health`              | Liveness probe. Returns `ok`.                                                       |
| `GET`  | `/status`              | Model id, which weights are serving, vocab size, whether workspace readouts are on. |

`/chat/completions` and `/models` also work without the `/v1` prefix, for clients that omit it.

A basic request:

```sh theme={null}
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "xynthis-local",
    "messages": [{"role": "user", "content": "Name one color."}]
  }'
```

Streaming follows the OpenAI convention: one SSE chunk per token delta, then a finish chunk, then `data: [DONE]`.

When the request omits them, `temperature` defaults to 0.7, `top_p` to 0.9, and `max_tokens` to 256. The `usage` block in responses is an estimate (character count divided by 4), not a true token count; tokenize client-side if you need exact numbers.

## Backbones

The server loads one backbone model at startup:

| Preset                | Model                 | Download                       |
| --------------------- | --------------------- | ------------------------------ |
| Default               | SmolLM2-1.7B-Instruct | \~3.4 GB                       |
| `XYNTHIS_LLM_SMALL=1` | TinyLlama-1.1B-Chat   | smaller, faster first download |

```sh theme={null}
XYNTHIS_LLM_SMALL=1 xynthis-llm-serve
```

The server holds one model and runs one generation at a time; concurrent requests queue. On Apple Silicon it uses Metal when available, otherwise CPU.

## Sessions and KV-cache reuse

Requests are stateless by default: every call re-processes the full message history. To skip that work across the turns of one conversation, send a stable `session_id` with each request:

```json theme={null}
{
  "model": "xynthis-local",
  "messages": [...],
  "session_id": "my-conversation-1"
}
```

The server keeps the attention KV cache from the end of each turn, keyed by `session_id`. When the next request's prompt extends the previous turn (the normal append-a-message chat pattern), generation resumes from the cache instead of re-reading the whole history. If the prompt diverges (edited history, different system message), the server detects it and starts fresh, so reuse never changes outputs. Sessions idle for 30 minutes are evicted; the sweep runs every 5 minutes.

## Configuration

| Variable               | Default         | Effect                                          |
| ---------------------- | --------------- | ----------------------------------------------- |
| `XYNTHIS_LLM_PORT`     | `8080`          | TCP port to listen on.                          |
| `XYNTHIS_LLM_BIND`     | `127.0.0.1`     | Bind address.                                   |
| `XYNTHIS_LLM_SMALL`    | unset           | Load the TinyLlama backbone instead of SmolLM2. |
| `XYNTHIS_LLM_MODEL_ID` | `xynthis-local` | The model id advertised by `/v1/models`.        |

<Warning>
  The server has no authentication. Keep the default loopback bind unless you are putting it behind something. Set `XYNTHIS_LLM_BIND=0.0.0.0` only behind a tunnel or VPN, never on an open network.
</Warning>

## What it serves

By default the endpoint serves the base backbone weights. The server can also learn from your data over time (see [Training](/llm/training)), and with `XYNTHIS_JSPACE=1` it reports the concepts the model weighed while generating (see [Workspace readouts](/llm/workspace-readouts)).