0.54.2 2026-09-18
Added
GET /indexes?details=true— and so the MCPlist_indexestool, which proxies it — carries per-index vector-lane health:semantic_coverage(vectors_present,vectors_unavailable_reason,vector_quant,chunk_count,embedded_this_boot),stages,search_capabilities,chunk_count,lexical_onlyandskip_vector. A zero-vector index reportsstages.semantic: readyand so appears in neither/health'sindexes_stage_failed_idsnor the console's Indexes roster; the roster now has the same three inputs — a nonzerochunk_count,vectoramong the capabilities, andvectors_present: 0— that the per-index expanded panel already flags on (#6699, follows #6689). Every field is computed byservice::server::vector_health, the one functionGET /indexes/{id}/statuscalls, so the roster and the panel cannot disagree about an index. Additive: existing entry fields keep their names, types and presence rules, and the plainGET /indexesarm still returns a bare array of ids. Costs two O(1) reads per index (Index::size()on the wired store, one redbtable.len()) beside the per-index directory walk the endpoint already pays forsize_bytes— no corpus scan.GET /indexes?details=truealso carrieslast_indexed, the same RFC 3339 timestampGET /indexes/{id}/statusreports. The endpoint's directory walk already produced it and this arm discarded it, so serving it costs nothing and lets a roster drop its per-row/statuscall (#6699).GET /healthreportsresident_index_capandresident_index_cap_source. The cap is the acting number (nullwhen it is off); the source is"env","env (off)", or"tier default". The same pair is logged once at startup. Without them the number that decides whether an index gets parked was only readable from the daemon's environment (#6821)SearchAppState::machine_tier. The residency sweep and/healthboth need the machine tier, and reading total RAM spawnssysctlon macOS — resolving it once per state keeps a process spawn off a per-poll endpoint (#6821)trusty-search quantize(andPOST /indexes/:id/quantize) re-encodes an existing index's vectors at a different scalar precision in place. This is the only way an index built before thef16default becomes quantized, since a forced reindex upserts into the store built at warm-boot and therefore re-embeds at the old precision. It reports before it writes (--dry-runnames the index, root, chunk count, vector count and snapshot bytes), prompts unless--yes, is a no-op when the index already holds the target precision, and refuses while a reindex is in flight.GET /indexes/:id/statusreportssemantic_coverage.vector_quant: the precision the LIVE index holds, which for any index built before the default flip differs from whatTRUSTY_VECTOR_QUANTwould suggest.TRUSTY_HNSW_DEMOTE_COOLDOWN_SECS— seconds of write-idle before a written HNSW store is persisted and demoted back to an mmap view. Default300;0,off,false,no,disabled, ornonedisables it; an unparseable value warns and falls back to the default.TRUSTY_HNSW_REVIEW_IDLEremains the kill switch for heap→view demotion as a mechanism and disables this path too; this knob turns off the write-cooldown path alone (#6826)- The embedded dashboard (
ui-dist/, mirrored fromcrates/trusty-console/ui-search) carries a stale-registration panel at#/indexes/cleanup. It lists this daemon's ownGET /registry/orphanscensus and prunes a confirmed batch throughDELETE /indexes/{id}, pinning each delete to the root the census reported so a path wiped and recreated in between is refused rather than deleted. No daemon-side change: the panel moved out of trusty-console (#6923, DOC-73 §13) onto the API this crate already served (#6941). - Expose bounded structured CSV, XLSX, HTML and TSV preparation for API chat attachments through the document extraction service.
- The eight result-returning MCP tools —
search,search_lexical,search_semantic,search_kg,search_all,list_chunks,grep, andget_call_chain— enforce a 48 KiB ceiling on the serialized response. Over it, whole hits are dropped from the tail andmetareportstruncated,returned,withheld, and atruncation_noticenaming the arguments that fetch the rest. A per-callmax_bytesoverrides the ceiling up to 512 KiB, above which it is clamped and the clamp reported;full: truedisables it for that call. A hit larger than the ceiling is returned alone rather than as an empty result, under a notice that says it exceeds the ceiling rather than claiming the response fits. A cappedlist_chunkspage rewrites its cursor to the last chunk it returned so paging does not skip the withheld ones. Amax_bytesthat is not a byte count, and afullthat is not a boolean, are rejected rather than ignored (#7493). - All five search tools —
search,search_lexical,search_semantic,search_kg, andsearch_all— take an optionalcompactflag (default false). A compact hit carries onlypath,start_line,end_line,compact_snippet,score, andmatch_reason;content,id,file, and the KG/ranking metadata are absent from the JSON. Measured at 63% fewer bytes on a 10-hitsearch(#7676). list_chunkstakes an optionalpath_prefix, matched at a path-segment boundary, so one file or directory can be enumerated in a single call with no seed cursor. The filter runs before paging, sototalandnext_cursordescribe the scoped set (#7677).GET /indexes/:id/statusreportsmigration_error, the set of outstanding per-stage migration faults (stage,detail,at), andsearch_healthreturns a newindex_migration_failedverdict instead of theindex_emptyone that prescribed a reindex. A failedchunks.json→index.redbmigration, or a failed M001–M005 chain, was visible only as one WARN line while both endpoints rendered the index as ordinarily empty. Faults are keyed by stage, so a schema chain with nothing to do no longer clears a still-true JSON migration fault (#7979).
Fixed
- A named chunk id now carries its end line —
{file}::{type}::{name}::{start}::{end}(#6581). It used to stop at the start line, so two declarations sharing a name and a start line shared a primary key. A minified bundle is one physical line, soapp/hiring/static/assets/index.jscollapsed 2,299 declarations onto 225 ids and the #6571 dedupe dropped the other 2,070 — those symbols were absent from search entirely. Two declarations with an IDENTICAL span still build one base id; the second and later occurrences now take a deterministic::dup::{n}suffix instead of being dropped. Positional ids for unnamed chunks are byte-identical to before. - Schema migration M005 (schema version 4 → 5) clears each existing index's corpus and re-chunks it from source on daemon startup (#6581). Operators will see a one-time re-chunk per index in the daemon log; a large index logs the before/after chunk counts when it completes. It re-embeds nothing: a chunk's text is unchanged, so its stored vector is handed to the new id by re-pointing the HNSW key sidecar, and only text the corpus has never held is left to the ordinary embed catch-up.
TRUSTY_MAX_CHUNKSapplies exactly as it does on a fresh index; an index whose re-chunk exceeds the cap is logged as migrated but not fully re-chunked. - The path filter accepts both the old and the new named chunk-id shape until an index's own M005 marker is set, then only the new one (#6581). Before this,
is_chunk_id_suffixhard-coded a three-segment named shape, which also rejected every method chunk outright because a qualified method name isType::methodand the check requirednameto be colon-free. get_call_chainrenders a named chunk id asfile:lineinstead of mis-parsing it (#6581). An id likeassets/index.js::Function::e::1::74came out asassets/index.js::Function::e:in the report.- M005 is crash-recoverable and never advances an index's schema version over missing chunks (#6581). The pass clears the corpus before rebuilding it, so the corpus itself cannot say whether the pass finished: an interruption anywhere between the clear and the final flush left no pre-#6581 id to find, the idempotency guard read that as "already migrated", and the daemon stamped schema version 5 over a partial — or, if the crash landed before the first batch, an entirely empty — index, permanently and with no error. A recovery plan is now written durably into the corpus
_metatable BEFORE the clear and removed only after the whole pass succeeds; its presence, not the corpus contents, is what says a pass is outstanding. An interrupted migration is resumed to completion on the next daemon start, and any failure leaves the version pinned at 4 so the pass retries rather than being recorded as done. - A search that lands while a migration is rebuilding an index now returns
503 index_migration_in_progress(retryable: true) instead of200with an empty result set (#6581). The re-chunk empties the corpus for the length of the pass while every read still succeeds, so a concurrent query was answered as a genuine miss — the same "outage rendered as nothing matched" failureindex_corpus_unavailablecovers for a corpus that cannot be read at all. - A schema migration and an incremental reindex can no longer run at once on the same index (#6581). M005 now takes the same per-index mutual-exclusion permit a reindex holds for its whole run, so whichever starts first completes and the other waits. Without it, a reindex's
copy_all_from— the #839 snapshot that carries hash-skipped unchanged files across the staged-corpus swap — could snapshot the corpus while M005 held it emptied, and the swap would then rename that staging file over the live one, discarding every unchanged file's chunks with no error and no distinguishable log. APATCH /indexes/:id/confignow refuses while a migration runs, as it already did during a reindex. - The global search fan-out reports an index it dropped because a migration is rebuilding its corpus, as
migration_in_progress_indexes_skipped(#6581). It previously fell through to the generic error arm and vanished from the payload with no counter at all, so the caller saw a complete-looking sweep. Present on every fan-out response, like itscorpus_failed_indexes_skippedandcorpus_read_failed_indexes_skippedsiblings. - Every corpus- and graph-backed endpoint now returns
503 index_migration_in_progresswhile a migration is rebuilding that index, not onlysearch(#6581).GET /indexes/:id/chunksreported a mid-rebuild page as the whole corpus,totalincluded;POST /indexes/:id/grepandPOST /grepreported{matches: [], total: 0};GET /indexes/:id/call_chainreported404 entry point not foundfor a symbol that exists;GET /indexes/:id/graph/neighborsreportedcount: 0with an empty neighbour list; andPOST /indexes/:id/graphmerged a contribution into a graph the migration had emptied, then answered200with totals read from that emptied corpus. All six now refuse with the same bodysearchalready sent. The ingest refuses BEFORE it persists, so a caller never holds a durable contribution behind a 503 that says nothing about it.GET /indexes/:id/statusandGET /healthare deliberately unchanged — a caller polling them to learn why the others are refusing needs them to answer. - A schema migration now stops at a checkpoint when the index it is migrating is deleted, instead of running to completion first (#6581).
DELETE /indexes/:id?delete_data=truesignals a cancel and then waits 30s for every writer to drain; M005 held the writer side for its whole re-chunk, so on a large index the wait always expired and the delete abandoned itself — telling the operator to re-issue a delete that would abandon again for as long as the migration lasted. M005 now polls the same cancel flag a reindex does, before its corpus clear and at every batch boundary. An index cancelled before the clear is left whole; one cancelled mid-pass keeps its recovery marker and its schema version, so the pass resumes from where it stopped if the index survives the delete. - An unreadable M005 recovery marker now fails the migration loudly instead of being treated as no marker at all (#6581). The marker is only ever written before the corpus clear, so an unreadable one means a pass started and its recovery inputs are gone; treating it as absent fell through to the corpus-contents check over the corpus that pass had already emptied. The schema version stays at 4 and the log names the remedy (reindex).
- A written HNSW index now returns to its mmap view after a write-idle cooldown, instead of holding its heap copy for the daemon's whole life.
TRUSTY_HNSW_MMAP_SERVEopens a snapshot as anIndex::view, and the first write —index-file, a reindex, a file-watcher commit — promotes it to a heap copy. The #2164 demote could only reverse that for a store whose graph already matched disk, which a written store never does until something saves it, and nothing did: the 128 GB reference host measured 76 MB of mapped file against 9 GB of heap across 56 mostly-edited indexes. The idle sweep now saves such a store and re-views it once it has been quiet forTRUSTY_HNSW_DEMOTE_COOLDOWN_SECS(default 300 s;0/offdisables).id_to_key/key_to_idstay heap-resident, exactly as they did in view mode before (#6826) save()no longer marks a store clean when a write raced it. The HNSW write lock covers only the FFI serialize, not the renames that follow, so a writer landing in that window had its mutation flagged as persisted; a later demote would thenIndex::viewa snapshot without it, dropping the vector from the graph while it still sat inid_to_key.save()now compares a mutation epoch across the whole operation and leavesdirtyset when one changed, costing at most a skipped reclamation cycle (#6826)serve's working-directory auto-pin now scans the index listing it already fetched for an entry rooted at the working directory before refusing. A derived id served from another tree — or served by nobody — pins that entry instead of leaving the session UNPINNED, and the startup line names it (#6864).- Cold-parking an index no longer drops its unpersisted vector writes.
cold_park_indexdetached the handle and dropped the last reference to it without callingsave(), so a heap-resident HNSW store holding writes the incremental persister had not yet checkpointed lost them; the next query rebuilt from the stale on-disk snapshot with the index reporting healthy. The park now saves the store first, through the samesave_vector_storepath the shutdown flush takes, and refuses to park when that save fails — the next sweep retries.indexes.toml,roots.toml, and the redb corpus are still untouched; the one durable write is to the index's own snapshot. This was rare whileTRUSTY_MAX_RESIDENT_INDEXESshipped off by default and is the ordinary path now that #6821 turns it on (#6870) - M005 re-reads each file through
core::extract::read_content, the same seam the ingest and watch paths use, so.docx,.pdfand spreadsheet files are extracted rather than read as raw UTF-8. Reading them as text produced no chunks, and the orphan sweep then dropped every vector they had — matsuoka-com fell from 64,517 chunks to 3,526 on its first query after upgrading (#6910). - A file that is still on disk but yields no chunks keeps its vectors instead of being swept as an orphan. That covers both an extraction error and an extraction that succeeds with no usable text —
extract::pdfreturnsOkwith empty text for a scanned/image-only PDF, which reached the sweep by a different route and dropped the same vectors. Removal now needs affirmative evidence that the file is gone from disk; an existence probe that errors keeps the vectors (#6910). - M005 reads each 64-file batch concurrently, as the reindex batch reader does. Sequential extraction cost the sum of the batch's files, so a pathological batch could hold the pass for ~32 minutes at the 30s per-file extraction timeout before reaching the next cancel checkpoint — long past the 30s an index delete waits for the pass to quiesce (#6910).
trusty-search index -n <name>on a root carrying atrusty-search.yamlmanifest no longer discards-n(#6920). A-nthat names no declared index is now refused, with an error naming the manifest path, every declared index name, the conflicting value, and the two ways forward. It used to be dropped with only an informational line, soindex --force -n flyr-duetto-monolith <root>reindexed the manifest'sduetto-backendinstead and tore that index's HNSW snapshot during the #6910 recovery. A-nthat matches one declared name now indexes only that index; it previously ran every declared index. Omitting-nstill fans out over every declared index, and a root without a manifest is unchanged..pptxdecks are extracted and indexed (#6938).pptxwas absent from bothEXTRACT_EXTSand the walker'sSOURCE_EXTSand no PowerPoint extractor existed, so every deck in an indexed tree produced zero chunks — 53 files under one measured corpus root yielded nothing. The new extractor readsppt/slides/slide<N>.xmlin numeric slide order (not entry order, whereslide10precedesslide2), emits one paragraph per<a:p>, and appends a slide's speaker notes (ppt/notesSlides/notesSlide<N>.xml) directly after that slide's own text. A corrupt, truncated, or non-zip.pptxreturns anExtractError::Pptx, never a panic; a zip carrying no slide part is an error rather than a successful empty extraction. An existing index gains no.pptxcontent until it is reindexed (trusty-search index <path>, orPOST /indexes/:id/reindex): the M005 re-chunk reads through the sameread_contentseam and so picks the extractor up automatically, but it re-chunks only files already in the corpus, and a.pptxwas never walked into one.- The
.docxand.pptxextractors share one bounded zip-part reader and one run-text decoder (core::extract::ooxml), rather than parsing the OOXML container and its<w:t>/<a:t>runs two ways (#6938). Each caller passes its own error constructor, so a failure still names the format the caller handed in. No.docxbehavior changed. - The residency sweep can no longer detach an index out from under a reindex that started mid-park. The sweep read
reindex_progressonce and then calledcold_park_index, which does a filesystem write (persist_before_park, #6870) before it removes the registry entry. APOST /indexes/<id>/reindexarriving during that write inserted itsRunningprogress and spawned against the SAMEArc<IndexHandle>, so the #3995Arc::ptr_eqidentity guard saw an undisturbed handle and the detach succeeded anyway. The reindex task then kept writinghandle.stageson anArcno later request could reach, and the first query afterwards lazily rebuilt a fresh handle from the cold entry — the reindex completed into nothing and its status stayed wedged. The park now re-reads the caller's reindex state immediately before the detach and aborts when a reindex isRunning, rolling its own cold-store insertion back identity-guarded so the index is left exactly as it was found: hot in the registry, absent from the cold store. The next sweep tick retries (#6957) - HNSW snapshots now capture the graph and key maps at one completed mutation boundary, preventing concurrent indexing from producing a binary that warm boot rejects against its sidecar. Key rewrites stay dirty until persisted, and snapshot publication stages both files, restores the previous sidecar on binary rename failure, and preserves deletion accounting until publication succeeds (#6961).
- Keep sidecar health PIDs current across restart gaps, and retire old PID forwarding tasks safely during shutdown and replacement (#6967).
- Preserve the corpus's applied migration version during a forced reindex so restarting does not replay completed migrations and change vector coverage (#6985).
- A forced reindex no longer leaves chunks for files that were deleted or that
stopped matching the walker's include set. Force stages an empty corpus and
skips the prune pass, so the promoted redb rows were already correct — but
the warm chunk map, the BM25 index and the vector store kept the obsolete
chunks, deferred embedding re-created their vectors, and the next graceful
shutdown flush wrote them back into the promoted corpus. A confirmed force
promotion now reconciles the warm state to the promoted corpus's chunk ids,
re-confirming each candidate against the corpus immediately before dropping
it so a concurrent
index-filecannot lose its vector. Incremental reindexes are unchanged. (#7004) - Apply each index's current subtree, extension, exclusion, and ignore rules to filesystem watcher updates and dropped-event reconciliation, so excluded metadata stays out of search after configuration changes.
- A filesystem error while resolving a saved file no longer deletes that file's chunks (#7396).
apply_modifiedrouted every negative answer fromadmitsinto the removal path, andadmitsanswered negative whenpath.canonicalize()merely failed — so oneEACCES, one network-mount hiccup, or a save landing inside an atomic-rename window purged the file from the index. Admission is now three-valued: only a definite policy exclusion, or a path confirmed absent by a second check, may remove anything. An undecidable answer leaves the index untouched, warns, and re-arms a rescan to settle the path from disk. - A dropped-event rescan whose index has no registry handle now re-arms instead of being discarded (#7396). The watch loop answered an absent handle with a bare
continue— no log line, no failure count, no retry — while the paths the OS dropped were already unrecoverable, so a handle missing for a moment cost the index an unknown set of file changes permanently. The registry lookup moved inside the pass: an absent handle is nowRescanError::UnregisteredIndex, which flows through the same failure counting and backoff retry every other incomplete pass uses. - A batch of undecidable live admissions now costs one full-tree reconcile rather than one per event (#7396). The causes are not per-file — a mount that answered with an error, a directory the daemon lost access to — so every file under them defers at once, and each defer armed its own detached timer and its own reconcile of the whole watched tree. Both the defer path and the failed-pass retry now arm through one
RescanGateper watch task, which drops a defer while a rescan is already outstanding and re-opens as the watch loop takes thatRescanoff its channel. - Refuse a
TRUSTY_DATA_DIRthat names the operator's production data directory when the process is acargo testbinary, falling back to the isolated per-process test directory. The override was read before the #4255 test-harness guard, so anyone who exports it — the documented isolated-instance workflow — had no guard at all and every persisting handler under test wrote fixture indexes and roots into the liveindexes.tomlandroots.toml. Any other override is honoured unchanged (#7599). grep'sglobparameter no longer returns a silent zero for real indexed paths (#7674). A glob with no/now matches by basename at any depth (rg -gparity, sosavings.rsreachesa/b/savings.rs), and an absolute glob is resolved against the index root so afilevalue copied out of asearchresult can be pasted verbatim. A request that supplies aglobnow carries ametaobject reporting the normalized glob, how many indexed files it selected, and the corpus size —glob_matched_files: 0states in words that the filter, not the pattern, produced the emptymatchesarray. An unparseable glob remains a400, never an empty result.- Hybrid
searchno longer ranks a semantic guess above a literal occurrence. A query naming an explicitly quoted string, an identifier carrying an underscore,::or a camelCase boundary (optionally behindfn/struct/const), a bare filename or path-shaped suffix (session_mcp_scope.rs,indexer/search/exact.rs, matched against each chunk's path), or an issue reference (#7675) now gets an exact-match floor: every chunk carrying that literal ranks above every chunk that does not — declaration first, live before archived, then the caller's branch preference — with the vector lanes ordering only the remainder. An unquoted multi-word phrase is a conceptual query and earns nothing. Candidates come from the BM25 postings rather than a scan of every chunk's content, so the lane stays sub-millisecond on a 100k-chunk index. A filename/path match's contribution to the floor is capped at a small constant (FILENAME_HIT_CAP) so a basename as common asmod.rscannot flood the page with an arbitrary subset of unrelated files, and an exact path-suffix match ranks ahead of a same-basename-only one.meta.exact_match_floorandmeta.exact_match_literalreport which literal decided the top hit;meta.exact_match_degradedandmeta.exact_match_full_scanreport a lane that could not run and a postings prefilter that was unavailable, so an absent floor is never mistaken for an absent literal. A query naming no literal, or one that occurs nowhere in the corpus, ranks exactly as before (#7675). - A non-boolean
compactonsearch,search_all,search_lexical,search_semantic, orsearch_kgis now rejected withInvalidParamsinstead of being read asfalse.compact: "true"used to return full hits byte-identical to a call that never asked for compaction (#7676). - Index
src/bin/**andsrc/coverage/**sources again.bin,coverage— and the other names that mean build output in one ecosystem and source in another (out,build,dist,classes,generated*) — now prune only at a build-output location, never directly undersrc/,tests/,benches/orexamples/. Matching on basename at any depth had hidden 301 tracked.rsfiles in this workspace, the wholetmCLI among them. The walk, the file watcher and the reconcile mtime walk share one predicate, so they cannot disagree about a path. An index built before this change is missing those files and needs onetrusty-search index <path> --forceto pick them up;trusty-search doctorsays so (#7694). - Exclude Terraform state blobs from indexing by default:
*.tfstate(any stem —terraform.tfstate,dev.tfstate,prod.tfstate) and*.tfstate.backup, matched on file name/extension at any depth. A repo-wide grep for a common word once returned noise dominated by a large.tfstateblob instead of relevant source lines. A directory merely namedtfstateis unaffected — only the basename is matched (#7722). trusty-search index <root> --forceon a root already registered under another index id now reindexes that index instead of failing withdaemon returned 409 Conflict for POST /indexes, which contradicted the flag's own--help(#7758). Without--forcethe refusal is unchanged.- The daemon's Unix socket path now honours
TRUSTY_DATA_DIR/--data-dir, so a second instance no longer tries to bind the production daemon's socket and refuses to start (#7801).TRUSTY_DATA_DIRis the authoritative override for isolating an instance end to end; a client reaches an isolated daemon by exporting the same value or by settingTRUSTY_SEARCH_SOCKET. trusty-search index --no-kgwith no other filter set no longer dropsskip_kgon the wire — the empty-filter fast path substituted default filters (#313).- The shutdown flush and the incremental persister no longer overwrite a populated
chunks.jsonwith an empty in-memory corpus, or with a corpus this indexer never loaded from that file. The write is refused, logged at ERROR with a running refusal count, and returned as an error; the snapshot is left byte-identical (#7920) - A reindex whose staged corpus swap cannot re-attach the durable corpus (for example because another daemon holds
index.redb) now quarantines the index instead of leaving it detached and reporting healthy. The index reports its corpus unavailable, logs an ERROR naming the index, and the shutdown flush and incremental persister refuse to writechunks.jsonor the HNSW snapshot for it (#7920) - That detached-corpus quarantine is now refused, in release builds too, when a corpus was re-attached between the swap's release and the quarantine call. Quarantining then would have left an index marked write-quarantined while holding a working corpus — a state no later corpus open clears — and only a
debug_assert!stood against it (#7920) - The M002/M004 absolute-to-relative path migrations no longer lose chunks. A chunk whose root-relative id is already held by another row keeps its absolute id and is reported as a warning with a count and a reason, instead of overwriting that row and deleting itself; a chunk whose id does not embed its file path has
filerewritten in place instead of being upserted and then deleted under the same key. The rewrite runs in one redb transaction that re-checks each target id and the row count, so a crash, a concurrent write, or a count mismatch commits nothing and a row written after the migration started is never overwritten. A row whose content changed after the migration loaded it, even under the same id and path, keeps its new content and its absolute id, and is counted and logged as a warning (#7923) - The
chunks.json→index.redbmigration no longer stamps schema v1 after a failure: a corrupt snapshot, a missing corpus store, a failed redb write, or a redb row count below the migrated count now fails the step so the next boot retries it, and the snapshot is left in place. Duplicate ids inside the snapshot are counted and logged instead of being folded away silently (#7923) - The
core::memguardRSS-liveness test no longer fails under parallel test load. It used to sample the whole test binary either side of its own 128 MB allocation, so a sibling test freeing memory inside that window drove the reading down and looked like a stale sampler. The growth pair is now sampled against a quiet child process, which also exercises the arbitrary-pid path the sampler exists for (#7926). - Every MCP boolean flag now rejects a wrong-typed value with an
INVALID_PARAMSerror naming the parameter and the expected type, instead of silently reading it as the default.exclude_archived,serial,full_content,include_source,follow_links,check,confirm, andgrep'scase_insensitive/multiline/fixed_strings/files_with_matches/invert_match/word_regexpused to map"true"(a string) ontofalsewith nothing in the response to say the flag was dropped. An absent flag still means the documented default (#7927). - The
chunks.jsonoverwrite guard no longer refuses every save from a store-less, non-quarantined indexer whose target file is unreadable. The unreadable bytes are renamed to a.corruptsidecar (numbered.corrupt.1,.corrupt.2, … on collision, so a later occurrence never clobbers an earlier one) and the write lands; a write-quarantined or corpus-detached writer is still refused, a preservation that cannot run falls back to the refusal, and nothing is ever destroyed (#7980). - A staged reindex promotion whose rename or re-open FAILED no longer reports
ReindexStatus::Complete. That arm releases the staging store and quarantines the index, so the run ended holding no corpus while stampinglast_indexed_atand a HEAD-SHA marker claiming the live corpus was current; the terminal status is nowFailedand neither stamp is written. Only the refused-promotion arm was excluded before (#7991, #7920). - The unreadable-snapshot sidecar now claims its
.corrupt[.N]slot atomically withcreate_newinstead of probing withexists()and renaming afterwards. Two writers preserving the same snapshot could compute the same free name between the probe and the rename, andrename(2)replaces its destination silently — destroying the copy the first writer had just preserved (#7980). - Staged reindex promotion no longer renames over a live
index.redbanother process holds open. The commit takes redb's own advisory lock (flockon macOS and Linux, the same primitiveredb::Databasetakes) on the live file and holds it across the rename; a held, unlockable, or unopenable live file defers the promotion with the staged corpus and the live corpus both intact. A deferred run reports the newReindexStatus::PromotionDeferredterminal status instead ofComplete, andGET /indexes/:id/statuscarries apromotion_deferredobject saying why — before, a refused promotion looked like a healthy completed reindex while the live corpus stayed stale (#7991).
Changed
TRUSTY_EMBEDDER=http://…is refused at startup instead of dialling a listener that no longer exists. #6289 retiredtrusty-embedderd --httpunder ADR-0032, so the HTTP arm and itsRemoteEmbedderAdapterare deleted; the error names ADR-0032 and points atTRUSTY_EMBEDDER=unix:/path/to/trusty-embedderd.sock, which reaches the same daemon over its hardened socket. Falling through to the auto-spawn default would have left an operator believing their manually managed daemon was in the loop when it was not- An MCP read tool called with no
index_idon an unpinned session now returns the registered indexes plus a retry hint as a successful result, instead of erroring. Applies tosearch,search_lexical,search_semantic,search_kg,typeahead, andindex_status;grepandsearch_allalready fanned out.index_idis no longerrequiredin those tools' schemas. The mutating tools —index_file,remove_file,delete_index,reindex— keep the error (#6317). - A host below 16 GB now degrades instead of refusing to start.
trusty-search startused tobail!with "requires at least 16 GB of RAM", so a 12 GB laptop got no search at all rather than a smaller one. It now warns once and serves on the reducedMemoryTier::Degradedcaps (embedding cache 500, BM25 corpus 50 000, KG nodes 75 000, batch-size hard cap 64 — half of Medium's row).TRUSTY_SKIP_RAM_CHECK=1still works: it silences the advisory now that there is no exit for it to bypass (#6820) - The tier primitives moved to
trusty_common::machine_tierand are re-exported unchanged. The RAM read,MemoryTier, and the proportional formulas were private to this crate, so trusty-memory could not read them and drifted.core::memory_policy::{detect_total_ram_mb, MemoryTier}still resolve, andMemoryPolicynow layers this daemon's own caps and env overrides on top of a sharedMachineBudget. A 16/24/32/64/128 GB host resolves to exactly the numbers it did before —MemoryTiergained aDegradedvariant and is now a re-export, which is why this is a MINOR bump - The
core::memory_policy::computemodule is renamedcore::memory_policy::coreml; nothing a caller names moved, so there is nothing to change. It holds only the CoreML batch-size and tripwire knobs, which are this daemon's embedding-pipeline concern rather than machine-tier policy; the proportional formulas that shared the file went to trusty-common.resolve_coreml_batch_size,resolve_coreml_tripwire_mb, and the fourCOREML_*constants keep their public paths - The resident-index cap is on by default and scales with the machine.
TRUSTY_MAX_RESIDENT_INDEXESshipped built and tested in #2161 and then sat disabled, so any index queried even once stayed resident for the daemon's whole lifetime — 56 resident indexes and 15 GB of heap on the 128 GB reporting host, 35 of them never queried. An unset value now resolves to the machine tier's default (Degraded2,Medium8,Large12,XLarge16) instead of to "off". A number is still honoured verbatim,0included, so a host that already sets the variable is unaffected (#6821) TRUSTY_MAX_RESIDENT_INDEXES=offis the new way to turn residency off.0could not take that role because it already means "park every resident index on the next sweep". An unparseable value warns and falls back to the tier default rather than silently restoring the unbounded pre-#6821 growth (#6821)- Warm-boot inherits the same cap when
TRUSTY_WARMBOOT_MAX_INDEXESis unset. Otherwise a boot eagerly loads every registered index and waits up toTRUSTY_RESIDENCY_SWEEP_SECSfor the sweep to park them back down; it is that transient peak, not the steady state, that a 16 GB host cannot absorb. Ordering is unchanged — most-recently-used first, and an index that was never queried and never indexed is always in the deferred remainder (#6821) - New indexes store their vectors as
f16by default; existing indexes keep the precision they were built with untiltrusty-search quantizeis run.TRUSTY_VECTOR_QUANTnow defaults tof16instead off32, so every index created from this version on stores half-precision vectors — half the vector bytes in RAM and on disk. Recall@10 measures 1.00, the same as f32, on theooc_quick_winsfixture's query set. SetTRUSTY_VECTOR_QUANT=f32to keep full precision;i8is unchanged and stays opt-in. An empty value now means "unset" and resolves to the default rather than tof32. - Existing indexes are untouched by the flip: usearch records the scalar kind in the snapshot header and rebuilds the metric from it on every open, so opening an f32 index under the new default reads it as f32 and rewrites no bytes. Converting one is the explicit
trusty-search quantizebackfill. - Corrected the
TRUSTY_VECTOR_QUANTmemory-tuning entry, which claimed a forced reindex adopts a new precision. It does not — the vector store is built once at warm-boot and a reindex upserts into it, soreindex --forcere-embeds at the old precision. - Internal:
lazy_loader::cold_park_indextakes a fifth argument. No daemon behaviour, CLI flag or environment variable changes here — the effect an operator sees is the park/reindex race fix listed under Fixed above.reindex_in_flight: impl FnOnce() -> boolis the late reindex re-check the park now runs immediately before it detaches the handle (#6957). The caller supplies it as a closure so the residency module keeps depending on neitherSearchAppStatenorReindexStatus— the same shaperoot_gate::evaluate_root_moveuses for its registry read. A caller that tracks no reindex state passes|| false(#6957) VectorStore::search_filteredtakes a fifth argument,shapes: ChunkIdShapes. An out-of-treeVectorStoreimplementation, or a caller invoking the trait method directly, must add it; every in-tree store and every daemon call site already pass it, so no CLI flag, MCP tool, HTTP endpoint or environment variable changes here. The filter needs the caller index's chunk-id shape policy to tell a pre-M005 index's legacy named ids from a migrated index's new-only ids while it parses them (#6902)
Removed
- Delete
core/output.rs, an orphaned module nomoddeclaration ever referenced, so it never compiled and none of its tests ran (#6053).
Security
- The chat-attachment XLSX bounds validator now follows the package and workbook
relationships to every sheet instead of scanning only
xl/worksheets/*.xml, requires an explicitrcoordinate on each cell, boundssharedStrings.xmlallocation hints, and rejects a package whose ZIP parts differ only by letter case or slash direction as ambiguous — a crafted workbook can no longer reach calamine's range allocation through a relationship target outside the conventional prefix (#7655).
Documentation
service::rpc::streams' module doc described the pre-#7217 transport, where the server learned of a disconnect only at its next write and an abandoned reindex producer could stay parked for a stall's duration or until the progress record's 60 s garbage collection.write_streamnow reads the peer off the socket, so a closed client ends the handler within one poll interval and the dropped receiver reaches every producer through itsSender::closed()arm.REINDEX_PROGRESS_TTL_SECSstill expires the progress record to bound daemon memory (#75); it is no longer what frees the producer. No behaviour change here — doc only. (#7217)