Skip to main content
xynthis-llm-serve runs a language model on your machine and exposes it through an OpenAI-compatible HTTP API at http://127.0.0.1:8080/v1. Anything that speaks the OpenAI API (an SDK, an editor integration, curl) can use it by changing one base URL. Nothing leaves your machine. 
xynthis-llm-serve
# xynthis-llm-serve listening on http://127.0.0.1:8080/v1
The first run downloads the model weights (roughly 1-3 GB depending on the backbone) from Hugging Face and caches them. Later starts load from the cache.

​
Endpoints

MethodPathPurpose
POST/v1/chat/completionsChat, OpenAI request/response shape. Set "stream": true for SSE.
GET/v1/modelsLists the served model (xynthis-local by default).
GET/healthLiveness probe. Returns ok.
GET/statusModel id, which weights are serving, vocab size, whether workspace readouts are on.
/chat/completions and /models also work without the /v1 prefix, for clients that omit it. A basic request: 
curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "xynthis-local",
    "messages": [{"role": "user", "content": "Name one color."}]
  }'
Streaming follows the OpenAI convention: one SSE chunk per token delta, then a finish chunk, then data: [DONE]. When the request omits them, temperature defaults to 0.7, top_p to 0.9, and max_tokens to 256. The usage block in responses is an estimate (character count divided by 4), not a true token count; tokenize client-side if you need exact numbers.

​
Backbones

The server loads one backbone model at startup:
PresetModelDownload
DefaultSmolLM2-1.7B-Instruct~3.4 GB
XYNTHIS_LLM_SMALL=1TinyLlama-1.1B-Chatsmaller, faster first download
XYNTHIS_LLM_SMALL=1 xynthis-llm-serve
The server holds one model and runs one generation at a time; concurrent requests queue. On Apple Silicon it uses Metal when available, otherwise CPU.

​
Sessions and KV-cache reuse

Requests are stateless by default: every call re-processes the full message history. To skip that work across the turns of one conversation, send a stable session_id with each request: 
{
  "model": "xynthis-local",
  "messages": [...],
  "session_id": "my-conversation-1"
}
The server keeps the attention KV cache from the end of each turn, keyed by session_id. When the next request’s prompt extends the previous turn (the normal append-a-message chat pattern), generation resumes from the cache instead of re-reading the whole history. If the prompt diverges (edited history, different system message), the server detects it and starts fresh, so reuse never changes outputs. Sessions idle for 30 minutes are evicted; the sweep runs every 5 minutes.

​
Configuration

VariableDefaultEffect
XYNTHIS_LLM_PORT8080TCP port to listen on.
XYNTHIS_LLM_BIND127.0.0.1Bind address.
XYNTHIS_LLM_SMALLunsetLoad the TinyLlama backbone instead of SmolLM2.
XYNTHIS_LLM_MODEL_IDxynthis-localThe model id advertised by /v1/models.
The server has no authentication. Keep the default loopback bind unless you are putting it behind something. Set XYNTHIS_LLM_BIND=0.0.0.0 only behind a tunnel or VPN, never on an open network.

​
What it serves

By default the endpoint serves the base backbone weights. The server can also learn from your data over time (see Training), and with XYNTHIS_JSPACE=1 it reports the concepts the model weighed while generating (see Workspace readouts).