Configuration
BrainLayer uses one shell-compatible config file as the local source of truth:
Launchd templates source this file through brainlayer-env-run.sh before they
exec their service command. API keys stay out of rendered LaunchAgent plists.
Environment Variables
Core
| Variable | Default | Description |
|---|---|---|
BRAINLAYER_DB |
~/.local/share/brainlayer/brainlayer.db |
Database file path. Set to override the default location. |
Launchd environment
| Variable | Default | Description |
|---|---|---|
BRAINLAYER_SYSTEM_ENABLED |
1 |
Global launchd gate. Set to 0/false/off to disable launchd-managed BrainLayer jobs. |
BRAINLAYER_ENV_FILE |
~/.config/brainlayer/brainlayer.env |
Override the config file sourced by launchd templates. |
BRAINLAYER_DISABLED_SLEEP_SECONDS |
3600 |
Sleep duration for disabled KeepAlive jobs to avoid restart loops. Use 0 in tests. |
BRAINLAYER_SUPERVISOR_SOCKET |
unset | Opt in to owner-only Unix component control for health/watchdog checks; leave unset until consolidated installer migration. |
BRAINLAYER_SUPERVISOR_READY_GRACE_S |
90 |
Export for install.sh consolidate: readiness, service-exit and custody waits use this integer budget (10–900 seconds; invalid values abort). Exit waits retain two extra seconds. Polls remain 0.1 seconds, with progress on stderr about every 10 seconds; readiness timeout still rolls back. |
BRAINLAYER_PAUSE_SENTINEL |
existing pause.sentinel |
Override the shared hold path for standalone watchdog adapters; explicit health-check configuration takes precedence. |
Control defaults to ~/.local/share/brainlayer/supervisor/control.sock; the environment override is resolved identically by supervisor and adapters. Read-only health never connects to control. Healing with unavailable or malformed control returns a critical supervisor_control_unavailable result, preserves badge/canary checks, and records failed heals. Mutations wait for the advertised component exit grace plus five seconds.
The staged supervisor uses a 15-second shutdown cap; its future plist must allow at least 25 seconds.
Missing-count comparisons accept ticks at least 240 seconds apart. A slow healthy hotlane during big ingest can still look stalled when arrivals outrun embeddings; net counts alone cannot distinguish overload from a wedge.
Enrichment (retired)
LLM chunk and session enrichment is retired. Provider, backend, model, concurrency, and rate controls are no longer activation instructions. History: retirement details.
Google credential compatibility gate
Older installed hotlane plists can still set BRAINLAYER_REQUIRE_GOOGLE_API_KEY=1.
The installed brainlayer-env-run.sh then exits 78 before exec if neither a
GOOGLE_API_KEY/GOOGLE_GENERATIVE_AI_API_KEY value nor a Google key declaration
in the env file is present. Keep the 1Password-backed GOOGLE_API_KEY configuration,
the require-key flag, and this exit-78 gate until the release re-renders the installed
hotlane plists. Retirement alone does not authorize deleting or bypassing them.
| Variable | Default | Description |
|---|---|---|
GOOGLE_API_KEY |
empty | Retained Google key configuration for the installed-hotlane compatibility gate. Prefer a 1Password op read reference. |
BRAINLAYER_REQUIRE_GOOGLE_API_KEY |
0 in env-run |
When an installed plist sets 1, require Google key configuration before executing its command. |
Launchd Toggles
Each install-managed plist in scripts/launchd/ sources the config file and
checks its own BRAINLAYER_LAUNCHD_*_ENABLED gate before exec.
| Variable | Default | Controls |
|---|---|---|
BRAINLAYER_LAUNCHD_HOTLANE_ENABLED |
1 |
Gates com.brainlayer.hotlane-brainbar. |
BRAINLAYER_LAUNCHD_DECAY_ENABLED |
1 |
com.brainlayer.decay |
BRAINLAYER_LAUNCHD_DRAIN_ENABLED |
1 |
com.brainlayer.drain |
BRAINLAYER_LAUNCHD_WATCH_ENABLED |
1 |
com.brainlayer.watch |
BRAINLAYER_LAUNCHD_INDEX_ENABLED |
1 |
com.brainlayer.index |
BRAINLAYER_LAUNCHD_BACKUP_DAILY_ENABLED |
1 |
com.brainlayer.backup-daily |
BRAINLAYER_LAUNCHD_JSONL_BACKUP_ENABLED |
1 |
com.brainlayer.jsonl-backup |
BRAINLAYER_LAUNCHD_MAINTENANCE_NIGHTLY_ENABLED |
1 |
com.brainlayer.maintenance-nightly |
BRAINLAYER_LAUNCHD_MAINTENANCE_WEEKLY_ENABLED |
1 |
com.brainlayer.maintenance-weekly |
BRAINLAYER_LAUNCHD_REPAIR_FTS_ENABLED |
1 |
com.brainlayer.repair-fts |
BRAINLAYER_LAUNCHD_WAL_CHECKPOINT_ENABLED |
1 |
com.brainlayer.wal-checkpoint |
Hotlane CPU budget
On macOS the hotlane sets its own CPU nice value to 20, without changing disk I/O policy.
BRAINLAYER_HOTLANE_THREADS caps torch and BLAS threads (default 1 on Apple M1, 2 otherwise).
BRAINLAYER_HOTLANE_BACKLOG_MAX_LOAD sets the one-minute load ceiling (default performance-core count).
BRAINLAYER_HOTLANE_BACKLOG_MAX_WAIT_SECONDS (default 300) guarantees a reserved backlog batch
at least every five minutes under high load. This uses the existing backlog lane and never lifts hot-slice backpressure.
A hotlane.pause file beside the resolved database also pauses backlog embedding; removing it resumes work.
The pause marker overrides minimum duty completely.
Hot embedding continues during these pauses unless store-queue backpressure suppresses it.
Startup records the applied settings in hotlane-budget.json
beside that database and in the log.
The daemon also writes hotlane-cpu.json beside the resolved database. It records a rolling ten-minute
CPU-time delta, backlog vectors actually encoded, and current running/yield/pause state, with no full-DB scans.
For the budget, it separately sums backlog_cpu_seconds and backlog_wall_seconds only around
completed cycles that encode backlog vectors. Yielded intervals contribute to neither sum;
only complete cycles wholly within the rolling window are retained.
The ratchet table automatically reads this file on a Mac and verifies its live installed PID with ps.
BRAINLAYER_HOTLANE_CPU_BUDGET_PERCENT defaults to 60% of one core and compares against
100 * backlog_cpu_seconds / backlog_wall_seconds. Less than 30 seconds of active backlog work,
an incomplete window, or a current pause marker is n/a, never GREEN. High load after a minimum-duty
batch does not dilute its measured CPU rate or invalidate sufficient active evidence.
The budget is unmeasured until installed; hosted CI cannot produce installed-Mac evidence.
An optional independent ps audit remains available via
python scripts/hotlane_cpu_ratchet.py --pid PID --db DB --out REPORT.json and the table's
--hotlane-cpu-report REPORT.json override. A ps-only report is diagnostic n/a for the budget:
without active-cycle CPU/wall timing, it cannot certify a pass or an overrun.
Privacy
| Variable | Default | Description |
|---|---|---|
BRAINLAYER_SANITIZE_EXTRA_NAMES |
(empty) | Comma-separated names to redact from indexed content |
BRAINLAYER_SANITIZE_USE_SPACY |
true |
Use spaCy NER for PII detection during indexing |
Database Location
The database path is resolved in this order:
BRAINLAYER_DBenvironment variable (highest priority)- Canonical path
~/.local/share/brainlayer/brainlayer.db(default)
First run: an absent database
The scheduled (calendar) brainlayer index pass never creates a database. When the resolved path
does not exist it prints INDEX_BOOTSTRAP_REQUIRED db=<path> run="brainlayer bootstrap-db" and
exits 78, writing nothing. A dangling symlink at the path prints INDEX_DB_UNAVAILABLE and exits 1.
An existing database is never bootstrapped: a partial, foreign or unwritable file still fails the
runtime schema check. Interactive brainlayer index keeps the writer's own refusal.
brainlayer bootstrap-db is the explicit first-run creator. It builds the canonical schema in a
sibling temp file, verifies the runtime schema fingerprint, and publishes it only if nothing is at
the path. Any existing path, including one created by a concurrent run, is refused unchanged
(BOOTSTRAP_REFUSED, exit 1).
Data Sources
BrainLayer reads from these locations by default:
| Source | Location |
|---|---|
| Claude Code conversations | ~/.claude/projects/ |
| Deduplicated system prompts | ~/.local/share/brainlayer/prompts/ |
| Daemon socket | /tmp/brainlayer.sock |
| Historical enrichment lock | /tmp/brainlayer-enrichment.lock (legacy safety/cleanup only) |
T3 projection selection
ingest-t3 automatically prefers ~/.t3/userdata/statev2.sqlite with the V2
projection reader. It also ingests state.sqlite when that file carries
projections, deduplicating shared thread/message IDs with the newer file winning.
Without V2, it falls back to state.sqlite. Both sources open read-only and
schema or payload failures remain fatal. An explicit --state-db selects only
that source and detects V2 from its projection tables when no version is supplied.
--projection-version overrides detection when deliberate selection is needed:
V2 reads settled user/assistant messages from their JSON payloads and retains
all projects, stable message IDs, source timestamps, and t3-thread provenance.
Streaming messages become eligible when settled. Missing schema or invalid
payloads raise an alarm; the reader never falls back to legacy tables. When V2
succeeds, an unavailable legacy source does not block its import, but health stays
alerting with legacy_source_failure. Dry-run
writes the health file but does not write the destination database.
The packaged T3 LaunchAgent runs every 300 seconds and resolves default sources
on every invocation. To override the source and interval durably, add these
settings to the private BrainLayer env file, then rerender with the installed
scripts/launchd/install.sh t3-ingest:
BRAINLAYER_T3_INGEST_STATE_DB=/Users/example/.t3/userdata/statev2.sqlite
BRAINLAYER_T3_INGEST_PROJECTION_VERSION=2
BRAINLAYER_T3_INGEST_INTERVAL_SECONDS=60
An explicit projection-version override (V1 or V2) requires an absolute source path. The importer reads all
projects. The renderer pins --db to the configured BRAINLAYER_DB (or canonical
DB) and --health-path beside that DB.
BRAINLAYER_T3_INGEST_HEALTH_PATH may explicitly override health placement.
Invalid configuration fails before the installed plist is replaced.
The public ingest and health-check CLIs and direct health-check configuration
share this path resolver; explicit --health-path/--t3-health-path options
take precedence over the environment. Rerender the job after changing its DB
or health selection so the loaded importer and scheduled consumer agree.
The optional interval accepts 60–86400 seconds; unset restores 300 seconds.
RunAtLoad remains enabled. launchd runs one instance of this label, and the existing DB writer arbitration still
applies. Freshness is the interval plus import runtime, model startup, and any
writer contention, so five minutes is a scheduling target rather than a hard
searchability guarantee. Writer contention exits cleanly for the next interval
and records alerting writer_busy health. A short writer preflight checks IDs
before embedding, then releases the writer for model loading/embedding. The
importer reacquires it and rechecks IDs before upserting precomputed vectors;
content already persisted by another writer is skipped. Other indexing failures
still fail. Indexed chunks with unchanged scrubbed content skip both embedding
and upsert, so repeat runs do not increment seen counts or add dedupe audit rows.
Each run embeds at most 200 changed chunks across both source files. Within each
source, changed chunks are ordered newest first before applying the cap; V2
remains the preferred source. CLI output
and health report chunks_deferred for the remainder, which later intervals
continue using stable IDs. First V2 execution imports historical settled messages
across as many bounded runs as needed. Verify loaded argv,
schedule, health and exact persisted message IDs after a reviewed deployment.
These settings select the SQLite importer. BRAINLAYER_T3_STATE_DB configures
watcher provider linkage separately. Without overrides, provider linkage uses
the same automatic source selection and unions linked session IDs across files.
Config File
Create or update the config file with:
Secure 1Password-backed form:
Plaintext Google keys are intentionally not supported in generated BrainLayer
env files. Existing configs retain the credential reference for the compatibility
gate; it does not enable the retired enrichment feature. See
scripts/launchd/brainlayer.env.example for the packaged schema and launchd job gates.
Personal lexical aliases (optional)
The packaged lexical_defense_dictionary.json holds only public product terms. Personal
search aliases (names, nicknames, private spellings) go in a file that is never shipped:
~/.config/brainlayer/lexical_defense.local.json, or the absolute path in
BRAINLAYER_LEXICAL_DEFENSE_LOCAL. Nothing creates this file for you.
{"version": 1, "entries": [{"canonical": "Zorbleth", "aliases": ["zbx"], "split_forms": ["zor bleth"]}]}
brain_search expands a query token that matches any surface of an entry into that entry's
surfaces. Both search paths read the file: the hybrid helper (lexical_defense.py) and the
BrainBar native fallback (LexicalDefenseUserAliases.swift).
| Case | Policy |
|---|---|
| File missing | No user aliases. Search behaves exactly as without the feature. |
Over 64 KiB, not UTF-8 JSON, wrong version, unknown keys, more than 256 entries, more than 16 aliases+split_forms per entry, or a surface that is empty, longer than 64 characters, or has no letter or digit |
The whole file is ignored, with one warning per file version (Python logger / BrainBar NSLog). |
| An entry whose surface is already claimed by a packaged entry, or by an earlier user entry (compared by letters and digits only, case-folded) | That entry is skipped whole and logged. Packaged entries never change. |
Cycles (A→B plus B→A) |
The second entry conflicts, so it is skipped. Expansion is one hop and never transitive. |
| Freshness | The file is stat()ed on every search and re-read only when its mtime or size changes. Edits apply to the next search, with no restart. |
User entries never set protect_from_split, so they stay out of the grammar, prompt and
VoiceLayer exports, which read only the packaged dictionary.
Scheduled Tasks (macOS)
BrainLayer includes launchd plist templates for automated operation:
| Service | Schedule | Description |
|---|---|---|
com.brainlayer.index |
Nightly | Incremental indexing of new conversations |
com.brainlayer.watch |
KeepAlive watcher | Watch and queue new conversation writes |
com.brainlayer.drain |
Queue/WatchPaths trigger | Drain queued writes as the single writer |
com.brainlayer.decay |
Weekly | Refresh decay metadata |
com.brainlayer.repair-fts |
Weekly | Read-only FTS row-count check |
com.brainlayer.wal-checkpoint |
Daily 09:30 | Checkpoint the WAL |
com.brainlayer.backup-daily |
Daily | Backup the BrainLayer DB |
com.brainlayer.jsonl-backup |
Daily | Backup Claude JSONL files |
com.brainlayer.maintenance-nightly |
Nightly | Light maintenance |
com.brainlayer.maintenance-weekly |
Weekly | Full maintenance |
Weekly calendar runs use index → backup-snapshot → full maintenance → JSONL backup,
counters, checkpoint, decay and FTS checks → backup-daily in upload-only mode. Snapshot and upload use the existing
calendar process-group supervisor sequentially; the upload keeps the existing job/history/log name; no detached worker is created. Nightly runs retain
maintenance first and DB upload last. BRAINLAYER_BACKUP_PHASE=snapshot writes a durable local
receipt after raw SQLite checks and archive verification; upload drains retained snapshots
newest first from those same bytes, preserving resumable sessions and retrying completed-session
verification. The default all phase keeps the combined daily command.
On a staging filesystem that passes the macOS hole-punch probe, every raw snapshot must pass
full integrity_check and sentinel validation before compression. Compression hashes the raw
stream and writes durable independent 64 MiB chunks (zstd -3 -T1 --check frames, or gzip -6
members), then punches the consumed raw extents. Each punch must reduce the raw file's allocated
blocks. For full chunks, volume free space is sampled up to five times over a two-second window
and must increase by at least 32 MiB against the same pre-punch reading. This allows delayed
macOS accounting while retaining a bounded safety check on a busy volume. A persistent deficit
aborts with raw_reclaim_unconfirmed and records the chunk, block counts, free-space samples,
deltas, threshold and window. It does not establish a snapshot pin. Unrelated disk usage between
punches does not accumulate against this check. The 2 GiB
raw-free-space floor remains enforced before writes and after punches. The archive is decoded
as a stream and SHA-256-compared with the checked raw before publication. Subsequent local and
Drive verification use the same SHA-256 proof; full Drive verification also compares compressed
MD5. Drive reads use 8 MiB requests and the upload path's bounded transient-network retry policy.
Drive uploads use 64 MiB chunks on a reused HTTP session. Each PUT has a total deadline of
max(120 seconds, chunk bytes / 32 KiB per second), capped by the remaining backup run budget
with five seconds reserved for the receipt. Override the floor or rate with
BRAINLAYER_DRIVE_UPLOAD_DEADLINE_FLOOR_SECONDS or
BRAINLAYER_DRIVE_UPLOAD_MIN_BYTES_PER_SECOND. Ambiguous network writes are probed before
retransmission. Private resumable receipts are fsynced at session creation, every confirmed
advance/retry, and every 30 seconds while waiting for a PUT. They include bytes_confirmed,
upload_rate_bps (newly confirmed bytes per elapsed upload second, excluding an existing resumed
prefix), retries, upload_probes, upload_requests, upload_deadline_exceeded,
upload_bytes_attempted (scheduled bytes, including retries; not measured wire traffic), and
upload_updated_at. A long unconfirmed chunk can show zero confirmed rate while bytes are in flight.
This path reserves raw size + one 64 MiB chunk + at most 64 KiB expansion per frame, plus the
2 GiB floor and 1 MiB margin. For a 22 GB database this is about 24.24 GB. If the staging probe
cannot reclaim blocks, the log names the reason and selects the legacy pipeline and its larger
raw-plus-compressed/restore/download peak budget. That fallback uses quick_check on ordinary
nightly runs, full integrity_check for snapshot/full-verification runs, a single zstd -3 -T0
process (or gzip), and the existing on-disk restore verifier. Older receipts without a raw hash
also retain that restore verifier. A runtime reclamation failure aborts; it never switches to an
unbudgeted fallback after consuming raw bytes.
Consumed raws live only in job-owned attempt directories. Before the first punch, an inode-bound
.consumed marker and the raw's move are fsynced. At the next run's start, before capacity
admission, only a marked consumed raw with the matching inode is deleted. Unmarked/unconsumed
raws, links, foreign files and archives are retained. A failed consuming attempt needs a fresh
snapshot; legacy failures retain their untouched raw.
Verified Drive uploads are removed locally (removal requires matching recorded local/Drive MD5s; older archives also need matching name/size receipts); weekly pins expire after their date, unreadable pins protect only today's archives, and explicit removal opt-out is honored.
Snapshots use YYYY-MM-DD.db.zst by default. BRAINLAYER_BACKUP_COMPRESSOR=gzip selects
.db.gz; any other value is refused. The zstd binary is resolved from
BRAINLAYER_BACKUP_ZSTD_PATH (absolute and executable, or the run fails), then
/opt/homebrew/bin/zstd, /usr/local/bin/zstd, then PATH, because launchd jobs run with
PATH=/usr/bin:/bin. If zstd is absent, gzip is selected and compressor_fallback is recorded.
Standard zstd/gzip decoders accept the concatenated frames/members. Every reader (verify,
restore, raw cleanup, resume, retention, observability, the weekly gate and BrainBar's local-copy
listing) accepts both formats. Drive retention keeps the newest verified copy of each
format; a verified local archive may be removed by the job. For opt-in Drive retention,
"verified" means the exact Drive file ID's latest backup-log receipt says uploaded and
verified (no verification_error or MD5 mismatch); a format with
no such object has nothing trashed, and an unreadable log trashes nothing. Each zstd chunk child
runs in the backup's process group, so the calendar's process-group stop reaches it. A child
failure stops consumption and cleans the attempt; the legacy streaming child is also killed and
reaped on timeout, stop or free-space floor breach.
Weekly VACUUM first reads the freelist/page-count ratio through a read-only connection. Below
BRAINLAYER_VACUUM_FREELIST_RATIO (default 0.10, range 0–1), VACUUM is not_needed: the rest
of full maintenance completes with exit 0, clears the maintenance alert, and advances the weekly
calendar marker. The calendar still takes its snapshot before maintenance for the final upload-only step.
When needed, weekly VACUUM requires a local verified snapshot of the same database, created since local midnight
and at most BRAINLAYER_BACKUP_LOCAL_MAX_AGE_HOURS old (default 6, shared by snapshot reuse and
the maintenance gate). Only the latest receipt for each snapshot counts, with raw integrity_check: ok.
The receipt binds SHA-256, size and path; missing, altered, unverified, failed remote verification,
future or yesterday's copies cannot authorize VACUUM. Remote upload is not part of that gate.
Calendar runs without that proof defer VACUUM with exit 75. A standalone full run may create a
local snapshot under the daily supervisor within the remaining quiet window. A held backup lock
still waits up to two hours in a standalone run; a calendar step, whose writers are already paused,
never waits and defers VACUUM with exit 75. Snapshot failures, timeouts and holds too short to
start VACUUM also defer it, preserving the reason and clearing a stale maintenance-full alert.
Deferred runs leave the weekly marker unchanged, so the next night retries; the calendar retains
its existing notice threshold for consecutive deferrals. VACUUM errors, expiry during VACUUM and
full-maintenance restore failures remain failures with exit 76 and an alert.
Quiet-window, queue-idle and writer checks run again after backup;
post-backup gate failure skips service bootout and all DB maintenance with exit 76.
python -m brainlayer.maintenance exit codes (#1040): 0 ok; 75 (EX_TEMPFAIL) only for a
deliberate, retryable deferral (a quiet-window, queue-idle or writer gate, a refused window-bound
hold, or an optional VACUUM step skipped for backup custody); 70 (EX_SOFTWARE) for a genuine
failure — services that could not be resumed (light/burn), a burn drain that failed, pathological
search latency, a quiesce or watchdog-hold failure, or any abort that does not declare itself a
deferral; 76 for a weekly full-pass failure; 77 when the maintenance lock timed out. A
deferral that then cannot restore its services exits 70 (76 in a full pass), never 75. Every
terminal record the run prints, and its row in the maintenance log, carries status, outcome
(ok/deferred/failed), exit_code, a UTC ts and run_id. Under the calendar, run_id is the
maintenance_run_id on that step's spawn and finished records (passed in
BRAINLAYER_MAINTENANCE_RUN_ID), so readers correlate on the record, not on a log's mtime; a manual
run gets a fresh id. Readers fall back to the reason allowlist only for older records without outcome.
Every writer pause (a supervisor MaintenanceHold) is bounded. It lasts at most
BRAINLAYER_MAINTENANCE_HOLD_MAX_S (default 3600). A calendar or scheduled maintenance hold is
also window-bound: it starts only inside the quiet window and ends 10 minutes before the window closes
(05:50 for the default window). A held step with no time left, including calendar FTS repair after
05:50, defers with exit 75 (maintenance-window) and the rest of the plan still runs. Manual FTS
repair keeps only the cap. The deadline is measured on a boot-scoped elapsed clock, not the wall clock,
so changing the system time neither extends nor shortens a hold. Suspend counts toward it, so a hold
that ran out while the Mac slept ends on wake. The hold clock is Darwin CLOCK_MONOTONIC or Linux
CLOCK_BOOTTIME.
The ordering is fixed on that one clock: 1. VACUUM is interrupted (SQLite rolls it back) 30 s before the deadline. 2. The calendar cuts the step at the deadline. 3. 60 s after the deadline, the supervisor sends TERM to the hold's owner, then KILL 30 s after a delivered TERM.
Only VACUUM unwinds itself; the checkpoints around it rely on the cut and the watchdog. The journal records the hold generation: owner token, PID, kernel birth time and boot session. The supervisor revalidates that same generation (lease still held, journal unchanged) immediately before each signal, and never signals a dead, reused or previous-boot PID. A hold is refused when its owner identity cannot be read. Escalation state sits beside the journal, so a supervisor restart does not grant a fresh TERM cycle. On each tick the supervisor restores a hold whose owner has died.
Older pending copies of the same DB become reclaimable only after a newer copy is offsite-verified;
capacity errors list retained archives. Weekly maintenance holds the archive lock across quiesce/VACUUM
and pins its exact snapshot for the rest of that local date. Missing or changed pending copies become
terminal abandoned receipts with local_verified: false and one durable alert; later runs can progress.
An empty upload drain exits successfully without appending a receipt or clearing the snapshot alert.
Observability keeps backups.db_snapshot at the last offsite uploaded-and-verified receipt, preserving
existing BrainBar behavior. Schema v2 adds the optional sibling backups.local_snapshot with
local_verified, uploaded, upload_state, upload_error and local integrity evidence. Upload failures
still set backups.error_type. BrainBar shows a fresh verified local snapshot with a pending or
in-progress upload as “Upload queued” or “Upload in progress” alongside the green last offsite copy,
only while that offsite copy is also within threshold_hours (normally 36 hours). A stale offsite
copy turns red even if each nightly job produces a new local copy. Failed or abandoned
uploads remain red and show upload_error; missing/unverified/stale local proof remains red. A
nothing_pending state makes both local snapshot and freshness rows red with an action to run
backup-daily or check the snapshot job; current producers do not emit this state. Schema v1 is frozen.
Schema v2 also exposes optional backups.transcript_attempt evidence (at, status: "no-op",
verified: true, uploaded: false) only for a valid fresh verified no-op receipt. BrainBar presents
that idle state neutrally under “Transcript coverage”, showing the last upload date and relative check
time with the caller’s calendar/locale; the actual archive ID and count remain in details.
Unrelated DB, Drive and weekly job failures retain their own attention lines without blaming transcript
coverage. Only transcript attempt errors, retention or freshness invalidate the idle evidence.
Missing, stale, failed or unverified attempt evidence never enables the idle presentation. Job failures,
expired Drive access and retention failures still require attention. Older documents without these
optional fields keep their previous presentation.
The snapshot step has a separate timeout (BRAINLAYER_BACKUP_SNAPSHOT_TIMEOUT_SECONDS). By default it is
sized from the database file: 200 s per GB, rounded up to five minutes, with a 45-minute floor and a
90-minute ceiling (21 GB gets 70 minutes). The 2026-10-20 MBP run needed at least 2843 s for 20.75 GB
(VACUUM INTO, full integrity check, live gzip, verify) and was killed by the old flat 45 minutes. The
ceiling still leaves weekly maintenance about a 35-minute hold after a 03:45 snapshot start. An explicit
override replaces the sized default. The step is also clipped to finish its process-group cleanup
before 06:00, and the backup's own deadline is set up to 60 s inside the calendar budget so its
timeout receipt is written before the calendar stops the process group. Both phases honor the existing
BRAINLAYER_LAUNCHD_BACKUP_DAILY_ENABLED gate; snapshot failures have their own alert key.
--full --dry-run reports a held backup lock without starting backup work.
Scheduled FTS row-count checks use a read-only connection and do not acquire the maintenance writer lock. Full FTS rebuilds require an explicit offline database copy path.
Install the remaining packaged agents with:
Keep existing Google key configuration until the release re-renders installed hotlane plists. Do not install, load, or resume the retired enrichment service.
The install-managed plists in scripts/launchd/ render without embedding
GOOGLE_API_KEY. Their ProgramArguments call the installed
brainlayer-env-run.sh loader, which sources ~/.config/brainlayer/brainlayer.env
and then execs the service command.
Configuration precedence is:
- Existing process environment.
- Simple assignments in
~/.config/brainlayer/brainlayer.env. - Built-in defaults.
The shared Python config loader does not read repo-root .env. Command
substitution such as GOOGLE_API_KEY="$(op read 'op://...')" is intentionally
left for the shell-based launchd env runner so plaintext secrets are never
written to disk by BrainLayer.
Migration for an existing hardcoded LaunchAgent: move the existing key value into
~/.config/brainlayer/brainlayer.env using brainlayer setup or
brainlayer init, preferably as a 1Password op read reference, then have the
deployment lead reinstall the repo-generated plist. Do not paste the key into
shell history, logs, PRs, or chat.
Manual credential cleanup
brainlayer scrub-at-rest --providers google_oauth --db /path/to/offline-copy.db --dry-run
reports matched row counts per table, column and provider. Provider modes are
google_oauth (the default), context7, and exa_labeled. EXA cleanup requires
a label with an exa segment and key; separators (_, ., -) or a
camelCase capital can end the segment. Labels like example_key and bare UUIDs
are preserved. Use the same mode for the survey and apply. Remove --dry-run to
apply to an offline copy; --batch-size defaults to 100. An explicit database path
is required. A live dry run is read-only and needs no opt-in. Live apply, including
symlink/hardlink aliases, refuses without --allow-live-db and --expect-rows N.
Use the sum of all table row counts from the preceding dry run for N (including
FTS/history copies, not just chunks). Apply re-surveys after quiescence and refuses
if the current total differs.
The guarded mode holds the maintenance lock, requires an active enrichment pause
sentinel and a verified backup receipt for that DB no older than 24 hours, and
reuses VACUUM's quiet-window (04:00–06:00 local), idle-queue and writer gates.
--wait-for-backup-seconds N (default 0) lets guarded live apply poll at intervals of up to 30 seconds
for a qualifying receipt before taking the lock or quiescing services. Dry runs and
offline applies ignore it. Poll intervals shorten near the deadline. Waiting ends
at N seconds or when only 20 minutes remain in the quiet window, reserving that
time for the guarded run. A refusal after sleeping reports verified-backup-timeout;
if no wait is possible before the first sleep, it reports verified-backup-required,
as does the default. Receipt freshness uses snapshot_created_at (the snapshot's VACUUM time), not attempted_at.
It quiesces the fleet/throughput/tier-0 watchdogs, health-check healer, BrainBar UI/daemon,
hotlane, watcher, drain, index, tier-3 ingest, decay and enrichment using maintenance's service helpers. It checks
that jobs remain unloaded, BrainBar processes are gone and no writable database
descriptors remain before applying. Any failed gate aborts before writing.
Service restoration runs in finally; services previously down or deliberately
paused remain down, and resume failures are reported by count. Refusal JSON adds
cause (a fixed category such as maintenance-abort, sqlite-busy, os-error,
scrub-refusal, or unknown) and gate (quiet-window, queue-idle, lsof-writers,
maintenance-lock, or null); neither ever carries exception text. Proof and rehearsal
must use copies and synthetic service stubs; canonical execution belongs to the
deployment lead's maintenance window.
The schema survey includes logical tables, FTS and repair/history copies, including text stored in numeric-affinity columns. Only the selected provider spans change; assignment/quarantine and other providers stay intact. Selected provider regexes see full values so window boundaries cannot truncate spans. Changed chunk content gets canonical hashes, SimHash bands and character counts; summaries/previews retain all unmatched text. Existing chunk/KG/git FTS triggers run, and session FTS refreshes explicitly. Bitemporal preimage capture and preview regeneration are transactionally suppressed during chunk redaction and restored verbatim; independent previews keep their unmatched text and existing history is scrubbed. Identity/reference matches appear in dry counts but refuse apply. Unsupported virtual/external-content index layouts refuse rather than silently skip. Unique-index collisions caused by redaction (for example in KG entities/facts or git memories) abort the current batch rather than deleting or merging rows. A failed batch rolls back; earlier committed batches may remain, so rerun after resolving the error. Writer/maintenance locks serialize apply, with bounded busy retries and checkpoints on the same writer connection before/after. The final survey must find zero selected matches. This is logical redaction, not physical erasure of old WAL/free pages, embeddings, source transcripts or backups. During the entire command, SQLite diagnostics log only error codes/classes, including file handlers; the process's normal SQLite callback is restored after stores close. Failures retain value-free causal error categories and cleanup notes.
Google Drive OAuth client mode
BRAINLAYER_DRIVE_OAUTH_TESTING=1 enables the local seven-day consent deadline
for OAuth clients whose Google publishing status is Testing. Leave it unset for
Production clients (the default): consented_at remains metadata, status has no
local expires_at, and only actual credential failures require reconnecting.
Google rejecting a refresh with invalid_grant still raises a consent alert in
either mode. Set this variable in the environment of the BrainLayer process
checking or loading Drive credentials.
Daily Drive upload recovery
backup-daily probes an interrupted resumable upload before retransmitting bytes.
Network failures use exponential delays (30 seconds up to 5 minutes), bounded by
BRAINLAYER_DRIVE_UPLOAD_RETRY_BUDGET_SECONDS (default 1200, per period without
confirmed progress). An explicit BRAINLAYER_DRIVE_UPLOAD_STALL_MAX_ATTEMPTS
additionally caps attempts; without it, uploads allow 30 attempts. Metadata-read
retries retain their existing default of three attempts. HTTP 401/403 and other
client errors fail immediately; transient HTTP 429/5xx retain short bounded retries.
The whole-backup timeout still applies.
A stalled daily receipt records resumable_upload with the session URL,
destination folder ID, and the artifact's absolute path, byte count, and pre-upload
SHA-256. Receipts containing this private upload capability are restricted to mode 0600. The next upload run
requires the same DB, machine, target folder, provenance, and unchanged artifact,
then asks Drive for its current offset. A completed session goes through the usual
verification. An expired session (404/410) starts a fresh snapshot and upload.
See the Drive resumable upload protocol.
After a newer snapshot is uploaded and verified on Drive in the same run,
reclamation only considers older dated .db.zst or .db.gz files
directly in the staging directory whose stalled receipt and current SHA-256 prove
ownership. Uploaded/verified paths, symlinks, hard links, changed bytes, malformed
receipts, and legacy receipts without identity evidence are retained. Each removal
is logged in superseded_stalled_deleted; uncertainty emits a warning. This does
not change Drive retention, successful-copy retention, or restore formats.
Insufficient snapshot capacity fails explicitly and names retained stalled files;
these files are never removed to make space.
Daily receipts record snapshot_created_at at VACUUM start; resumed receipts
preserve it. Verified-backup reuse and observability age the snapshot, rather
than its later upload attempt. Legacy dated snapshots use conservative UTC
midnight; unknown creation times fail closed. Resume requires an age below
BRAINLAYER_BACKUP_RESUME_MAX_AGE_HOURS (default 36). The legacy combined-upload
reuse override remains supported for older callers; weekly local reuse uses the shared local-age knob above.
The upload session is checkpointed privately before sending bytes, so even a supervisor exit retains a probeable session. Timeout, stop, and mid-upload auth failures preserve its identity and confirmed offset. Recovery sleeps and request timeouts share the remaining whole-run deadline, reserving five seconds to write the receipt. Session-creation POST network failures use the same bounded backoff; HTTP client/auth failures still stop immediately.
A fresh snapshot cannot overwrite a same-day stalled archive of either format. An expired or unsafe session with such a collision fails explicitly and retains the existing file. A later dated run can create a replacement without changing backup filenames.
Weekly maintenance rechecks local proof immediately before quiesce and VACUUM. Upload runs after weekly maintenance and retains its independent remote verification contract.