The control file
xynthis-llm-serve polls ~/.xynthis/llm/train_config.json every 5 seconds and obeys it:
enabled: whentrue, the trainer runs a tick everyinterval_secs. Defaultfalse.interval_secs: seconds between ticks. Clamped to between 60 and 86400 at read time.force_tick: a one-shot switch that runs a tick now, regardless ofenabledor the interval. The server clears it after the tick. The app’s “Train now” button sets this.
~/.xynthis/llm/state.json after every tick: cumulative steps, last loss, the latest checkpoint version, which version (if any) was promoted, and whether the gate passed.
What a tick does
Each tick fine-tunes a LoRA adapter on the backbone: the base weights stay frozen, and only the small adapter trains. Training examples come from three sources, in priority order:- Past agent sessions: full transcripts (question, tool calls, results, final answer), about two-thirds of each batch. This teaches the model how tasks were actually done on your machine.
- Brain data: working-memory perceptions and knowledge-graph facts, read from the brain over its socket. If the brain is down, this source contributes nothing and the tick proceeds.
- A built-in seed corpus: a fallback when both are empty, so a fresh install can still complete a tick.
~/.xynthis/llm/student-v<N>.safetensors.
Eval-gated promotion
A new checkpoint does not automatically become the served adapter. After saving, the trainer scores the candidate’s cross-entropy on a fixed probe set and compares it against the best score any promoted version has ever achieved. Only if the candidate is within a small tolerance of that best-ever score doesstudent-latest.safetensors (the file the server resumes from and serves) flip to the new version.
When the gate fails, the candidate stays on disk as its versioned file for inspection, student-latest is untouched, and training continues; the next tick may recover. A candidate that can’t be scored (a numerically broken adapter) never promotes. Gating against the best-ever score rather than the previous one prevents quality from drifting downward one tolerance step at a time.
Set XYNTHIS_LLM_UNGATED_PROMOTE=1 to skip the gate and promote every tick unconditionally. This is an escape hatch, not a recommendation.
Restarts
The trainer resumes from the promoted checkpoint on startup, so accumulated learning survives process restarts. Because a restart resumes fromstudent-latest (the last version that passed the gate), a run of bad ticks before a crash can’t carry forward.
Serving what you trained
Training and serving are decoupled: the running server keeps the weights it loaded at boot even after a tick promotes a new checkpoint. To serve the fine-tuned adapter, restart withXYNTHIS_LLM_SERVE_STUDENT=1. If loading fails or no checkpoint exists, the server logs it and serves the base model, and GET /status reports "serving": "teacher_backbone" instead of "student" so you can verify which weights are answering.