> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xynthis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Training

> How the local model fine-tunes itself from your brain's data, with eval-gated checkpoints.

The model server can fine-tune its model from what the brain has stored: your past agent sessions, working memory, and knowledge-graph facts. Training is opt-in and off by default. A server with training disabled loads none of the training machinery and pays no memory cost for it.

## The control file

`xynthis-llm-serve` polls `~/.xynthis/llm/train_config.json` every 5 seconds and obeys it:

```json theme={null}
{
  "enabled": false,
  "interval_secs": 1800,
  "force_tick": false
}
```

* `enabled`: when `true`, the trainer runs a tick every `interval_secs`. Default `false`.
* `interval_secs`: seconds between ticks. Clamped to between 60 and 86400 at read time.
* `force_tick`: a one-shot switch that runs a tick now, regardless of `enabled` or the interval. The server clears it after the tick. The app's "Train now" button sets this.

The app UI writes this file; you can also edit it by hand:

```sh theme={null}
mkdir -p ~/.xynthis/llm
printf '{ "enabled": true, "interval_secs": 1800, "force_tick": true }\n' \
  > ~/.xynthis/llm/train_config.json
```

The server writes its status to `~/.xynthis/llm/state.json` after every tick: cumulative steps, last loss, the latest checkpoint version, which version (if any) was promoted, and whether the gate passed.

## What a tick does

Each tick fine-tunes a LoRA adapter on the backbone: the base weights stay frozen, and only the small adapter trains. Training examples come from three sources, in priority order:

1. **Past agent sessions**: full transcripts (question, tool calls, results, final answer), about two-thirds of each batch. This teaches the model how tasks were actually done on your machine.
2. **Brain data**: working-memory perceptions and knowledge-graph facts, read from the brain over its socket. If the brain is down, this source contributes nothing and the tick proceeds.
3. **A built-in seed corpus**: a fallback when both are empty, so a fresh install can still complete a tick.

The tick runs a fixed number of gradient steps, then saves the adapter as a versioned checkpoint: `~/.xynthis/llm/student-v<N>.safetensors`.

## Eval-gated promotion

A new checkpoint does not automatically become the served adapter. After saving, the trainer scores the candidate's cross-entropy on a fixed probe set and compares it against the best score any promoted version has ever achieved. Only if the candidate is within a small tolerance of that best-ever score does `student-latest.safetensors` (the file the server resumes from and serves) flip to the new version.

When the gate fails, the candidate stays on disk as its versioned file for inspection, `student-latest` is untouched, and training continues; the next tick may recover. A candidate that can't be scored (a numerically broken adapter) never promotes. Gating against the best-ever score rather than the previous one prevents quality from drifting downward one tolerance step at a time.

Set `XYNTHIS_LLM_UNGATED_PROMOTE=1` to skip the gate and promote every tick unconditionally. This is an escape hatch, not a recommendation.

## Restarts

The trainer resumes from the promoted checkpoint on startup, so accumulated learning survives process restarts. Because a restart resumes from `student-latest` (the last version that passed the gate), a run of bad ticks before a crash can't carry forward.

## Serving what you trained

Training and serving are decoupled: the running server keeps the weights it loaded at boot even after a tick promotes a new checkpoint. To serve the fine-tuned adapter, restart with `XYNTHIS_LLM_SERVE_STUDENT=1`. If loading fails or no checkpoint exists, the server logs it and serves the base model, and `GET /status` reports `"serving": "teacher_backbone"` instead of `"student"` so you can verify which weights are answering.