> ## Documentation Index
> Fetch the complete documentation index at: https://docs.xynthis.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Binary Memory Core

> The engineering inside Xynthis memory: fixed 64-byte records, two-stage binary search, crash-safe writes, and measured throughput.

[Memory](/concepts/memory) explains what Xynthis remembers and why. This page goes under the hood of the store that holds it: the Binary Memory Core (BMC), the storage engine the brain links as an in-process library. There is no database server, no socket, and no background thread on the hot path. The brain calls a function, the function returns ranked results, synchronously, on the calling thread. Everything lives in plain files under `~/.xynthis/bmc/`.

## The record: 64 bytes per fact

Every durable fact is a fixed-layout 64-byte binary record:

| Field                              | Size | Purpose                                             |
| ---------------------------------- | ---- | --------------------------------------------------- |
| id                                 | 8 B  | Monotonic, never recycled                           |
| created\_at, valid\_from           | 16 B | Timestamps (epoch seconds)                          |
| valid\_until                       | 2 B  | Validity window length in days (0 = open)           |
| record type, flags, schema version | 3 B  | Store discriminant, soft-delete bit, format version |
| confidence, source trust           | 3 B  | Born-with confidence and source weighting           |
| signature                          | 32 B | 256-bit binary signature                            |

The signature is derived from the fact's embedding by a sign rule: bit *i* is set exactly when dimension *i* of the embedding is positive, over the first 256 dimensions. Two facts about similar things get similar bit patterns, so comparing signatures approximates comparing embeddings, at a fraction of the cost.

The verbatim text and its provenance (source, channel, timestamps) are stored alongside in a crash-safe key-value sidecar, keyed by record id. The binary record is the index and the ranking substrate; the sidecar is the payload.

Why fixed-width binary rather than a serialized format:

* **One cache line per record.** 64 bytes is the cache-line size on most CPUs, and the struct is declared with 64-byte alignment. A scan over records is a linear walk the prefetcher loves.
* **No parsing.** Loading a record is a size-checked byte copy into the struct. There is no deserialization step, no schema negotiation, no allocation per record.
* **mmap-friendly.** Fixed stride means record *n* lives at a computable offset, so the store files can be memory-mapped and read in place.
* **A training substrate.** Records are already tensors of bits and small integers. The same files that serve recall can feed a training loop without an export step. A binary-native classifier substrate exists behind a feature flag today; it runs in shadow mode only and is not on the live decision path yet.

## Search: propose cheaply, verify precisely

Recall is two-stage.

**Stage 1 proposes.** A locality-sensitive hash index over the bit signatures returns candidate ids. Matching is Hamming distance: XOR two signatures and count the differing bits, which modern CPUs do in a handful of instructions per 64-bit word. If the first probe comes back thin, a multi-probe pass widens the net.

**Stage 2 verifies.** Only the proposed candidates get the expensive treatment: a precise cosine similarity against the stored quantized embedding, which replaces the cheap Hamming score for ranking. The full-precision comparison never runs over the whole store, only over the shortlist.

Both stages run in-process and synchronously. No network hop, no query queue, no round trip. At the benchmark profile of a 10,000-record store (batched writes, consumer Apple-silicon laptop), search p95 is 792 µs and a full multi-store router query is 799 µs p95: under a millisecond end to end. Store-level behavior at 100,000 records and beyond is where the current optimization work is aimed; the 10k profile is measured and healthy, the million-record profile is not yet proven.

## Durability: assume the power cord gets pulled

Writes go through a per-store write-ahead log before they touch the main file. Each WAL entry carries a CRC32 checksum. Two durability modes exist: strict mode (the default) fsyncs per record, batched mode does group commit, one WAL write and one fsync per insert batch.

On every open, the store replays the WAL and then repairs it:

* A torn tail (the file ends mid-entry) or a zero-filled tail (the classic power-cut artifact) is detected and truncated back to the last good entry. Everything before the tear survives.
* A CRC mismatch on bytes that are fully present is fatal by design. That pattern means corruption, not a tear, and silently skipping it would hide data damage.

This behavior is pinned by a test harness that tears the files at every byte offset of the final write and verifies both prefix recovery and continued appendability. Any future throughput work has to pass it.

One process writes, ever. Each store takes an exclusive OS-level advisory lock on a lockfile in its directory at open and holds it for the store's lifetime. A second writer, whether another process or a second handle in the same process, fails fast with a store-locked error instead of silently corrupting shared state. The OS releases the lock on exit, so a crash never wedges the directory.

## Scale: fewer bytes, more cores

Two levers keep the store fast as it grows, both measured.

**int8 rescore payloads.** The stage-2 cosine pass needs the embedding, and that payload dominates disk volume: at 768 dimensions, the legacy f16 encoding costs 1,544 bytes per record, the int8 encoding costs 780, a 49.5% reduction. Recall is unaffected: cosine similarity is scale-invariant, so per-vector symmetric int8 quantization produced rank-for-rank identical results across every query in the recall eval, and a round-trip test pins cosine fidelity above 0.9999. int8 is the default for new stores; existing f16 files are auto-detected from their header and keep working with no migration.

**Sharding.** The store can split into N fully independent shards, each with its own WAL, index, and files, searched in parallel with reversible id routing. On a 10-core Apple-silicon machine with 100,000 records, search throughput went from 1,428 to 3,422 queries per second at 4 shards, with p50 latency dropping from 697 µs to 249 µs. Inserts moved from 170,715 to 196,228 per second at 4 shards, and int8's smaller payload added a further 13.6% insert win in the disk-bound 4-shard regime. Eight shards regressed (efficiency-core and fsync contention), so 4 is the measured sweet spot on this class of hardware.

Measured single-threaded throughput at the 10,000-record batched profile, same laptop:

| Operation           | Throughput |    p50 |
| ------------------- | ---------: | -----: |
| Fact store insert   |  145,735/s |   6 µs |
| Vector store insert |  178,098/s |   5 µs |
| Graph edge insert   |  457,836/s |   2 µs |
| Fact store delete   |   18,161/s |  55 µs |
| Search              |    1,427/s | 697 µs |

These are real numbers from the crate's benchmark tool, not projections. The write path got there by eliminating syscall storms: the rescore encoder once issued 769 write syscalls per 768-dimension record; batching that into one buffered write per batch took inserts from roughly 1,300 to 145,000+ per second.

## The knowledge graph

Alongside the fact stores, BMC keeps a temporal knowledge graph of subject-predicate-object triples. Three properties make it more than an edge list:

* **Time is first-class.** Every triple carries a validity window and a confidence. Queries can ask "as of" a moment: a fact that ended stays queryable inside its window instead of vanishing.
* **Entities canonicalize.** The entity index is built on a deterministic canonical key: trim, strip quotes and punctuation, drop a leading article, collapse whitespace, casefold. "Ada Lovelace", "ada  lovelace", and " The Ada Lovelace. " land in one bucket, while the stored triple keeps its original surface form for display.
* **Facts supersede.** When a newer perception contradicts an older one (same subject and predicate, conflicting object), a consolidation sweep marks the older record stale. The sweep processes records in id order, so older facts get superseded by newer ones and never the reverse. Stale facts stop surfacing in recall but are not destroyed.

## What this buys you

The point of all this engineering is a single property: memory that costs nothing to consult and nothing to keep. Sub-millisecond recall means the brain checks its memory on every single turn rather than rationing lookups. Sixty-four bytes per fact plus a sub-kilobyte payload means years of facts fit in the space of a few photos, so nothing has to be forgotten for capacity reasons. And because the records are already fixed-width binary with confidence, trust, and validity built into the layout, the same store that answers queries can feed the [learning loop](/concepts/learning) directly. That combination, memory fast enough to always use, cheap enough to keep forever, and structured enough to train on, is the substrate the rest of the [vision](/vision) is built on.