Model server

xynthis-llm-serve runs a language model on your machine and exposes it through an OpenAI-compatible HTTP API at http://127.0.0.1:8080/v1. Anything that speaks the OpenAI API (an SDK, an editor integration, curl) can use it by changing one base URL. Nothing leaves your machine.

xynthis-llm-serve
# xynthis-llm-serve listening on http://127.0.0.1:8080/v1

The first run downloads the model weights (roughly 1-3 GB depending on the backbone) from Hugging Face and caches them. Later starts load from the cache.

Endpoints

Method	Path	Purpose
`POST`	`/v1/chat/completions`	Chat, OpenAI request/response shape. Set `"stream": true` for SSE.
`GET`	`/v1/models`	Lists the served model (`xynthis-local` by default).
`GET`	`/health`	Liveness probe. Returns `ok`.
`GET`	`/status`	Model id, which weights are serving, vocab size, whether workspace readouts are on.

/chat/completions and /models also work without the /v1 prefix, for clients that omit it. A basic request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "xynthis-local",
    "messages": [{"role": "user", "content": "Name one color."}]
  }'

Streaming follows the OpenAI convention: one SSE chunk per token delta, then a finish chunk, then data: [DONE]. When the request omits them, temperature defaults to 0.7, top_p to 0.9, and max_tokens to 256. The usage block in responses is an estimate (character count divided by 4), not a true token count; tokenize client-side if you need exact numbers.

Backbones

The server loads one backbone model at startup:

Preset	Model	Download
Default	SmolLM2-1.7B-Instruct	~3.4 GB
`XYNTHIS_LLM_SMALL=1`	TinyLlama-1.1B-Chat	smaller, faster first download

XYNTHIS_LLM_SMALL=1 xynthis-llm-serve

The server holds one model and runs one generation at a time; concurrent requests queue. On Apple Silicon it uses Metal when available, otherwise CPU.

Sessions and KV-cache reuse

Requests are stateless by default: every call re-processes the full message history. To skip that work across the turns of one conversation, send a stable session_id with each request:

{
  "model": "xynthis-local",
  "messages": [...],
  "session_id": "my-conversation-1"
}

The server keeps the attention KV cache from the end of each turn, keyed by session_id. When the next request’s prompt extends the previous turn (the normal append-a-message chat pattern), generation resumes from the cache instead of re-reading the whole history. If the prompt diverges (edited history, different system message), the server detects it and starts fresh, so reuse never changes outputs. Sessions idle for 30 minutes are evicted; the sweep runs every 5 minutes.

Configuration

Variable	Default	Effect
`XYNTHIS_LLM_PORT`	`8080`	TCP port to listen on.
`XYNTHIS_LLM_BIND`	`127.0.0.1`	Bind address.
`XYNTHIS_LLM_SMALL`	unset	Load the TinyLlama backbone instead of SmolLM2.
`XYNTHIS_LLM_MODEL_ID`	`xynthis-local`	The model id advertised by `/v1/models`.

The server has no authentication. Keep the default loopback bind unless you are putting it behind something. Set XYNTHIS_LLM_BIND=0.0.0.0 only behind a tunnel or VPN, never on an open network.

What it serves

By default the endpoint serves the base backbone weights. The server can also learn from your data over time (see Training), and with XYNTHIS_JSPACE=1 it reports the concepts the model weighed while generating (see Workspace readouts).

Getting started

Core concepts

Guides

On-device model

Reference

Endpoints

Backbones

Sessions and KV-cache reuse

Configuration

What it serves

​Endpoints

​Backbones

​Sessions and KV-cache reuse

​Configuration

​What it serves

Endpoints

Backbones

Sessions and KV-cache reuse

Configuration

What it serves