A shared memory for me and my AI agents: from 9 to 12 out of 15

My home lab’s written record of tasks, mistakes, runbooks and gotchas holds 5,587 entries in one shared store. Three kinds of readers use it every day: me, AI coding assistant sessions, and my local AI agent. On 18 October 2026 I moved it into a context database that all three read and write, and measured whether they could find things in it. The first score was 9 out of 15 questions answered correctly.

This post covers how the shared memory is laid out, how two secrets slipped past my redaction on the first day, and what took retrieval to 12 out of 15 questions.

Contents · 11 sections

TL;DR #

• A shared memory for humans and agents needs per-reader permissions: who may write where, and whose writes are trusted.
• Redact by value, not only by pattern. A token stored as a bare value, with no label in front of it, slips past every regex.
• Store one fact per file. Packing many entries into one chunk dilutes each fact until vector search cannot find it.
• Vector scores from one embedding model cluster tightly; a content reranker over the top hits put the right answer first.
• Memory is history. Status questions need a live check, and nothing may be recorded as observed before it happened.

Why a shared memory #

The record started as a SQL database with a command-line tool. It worked for keyword lookups, but my agents asked questions in plain words (“which log viewer do I use?”) and the entry said “Dozzle”. I also wanted one store for everyone, where an AI session’s notes from yesterday are visible to the agent today, and both can see mine.

I chose OpenViking, an open-source context database (version 0.5.0, AGPL-3.0). It presents a viking:// filesystem, writes short summaries for folders and files, runs vector search and speaks MCP, so agents use it as a set of tools.

Layout and permissions #

Three accounts: me (owner), one account shared by the AI assistant sessions, and one for the local agent. The tree and its access:

Area	Assistant sessions	Local agent
the roadmap	manage	write (since 18 October)
git snapshots of repositories	manage	read
shared notes	write	read
the agent’s own area	read	write

Two rules matter more than the table. First, content the agent writes is untrusted data for the other sessions: they read it as information, never as instructions. Second, the agent’s client has its own guard: it does not get the tools to add resources, add skills, delete, set ACLs or cancel watches, and its writes are restricted by URI prefix.

That client guard exists because the server has gaps. In this version, the “shared skills” area had no server-side write control: any account’s key could write a skill there that every agent would then load. A nightly drift check now fails on any skill that is not on an approved list. Also, the root folder allows any user to create a new top-level folder, which only the client guard and the drift check cover.

Two more gotchas from the first day:

• Creating a folder accepted an acl field and silently ignored it. My setup script reported “ok” while every folder stayed open to all users. ACLs are now applied with a separate call and verified after setting.
• A key-minting script re-issued keys when run a second time, and the step that stored them failed quietly, so the stored keys stopped working a few minutes later. Scripts that mint credentials must be safe to re-run, fail loudly when storing fails, and test the stored value.

How secrets slipped past redaction, twice #

Before the first import, an export script turned the database into Markdown and redacted credentials. An independent leak scanner, written by a second AI session, found the first problem before anything was ingested: an old API key written as API key: <value>. The regex expected api_key or api-key; a space was not a separator it knew.

The second one got further. One entry in an infrastructure table held a live bot token as a bare value, with no label at all. Pattern redaction cannot see that. The scanner found it in one ingested chunk before any summary copied it. Deleting it through the API was impossible mid-ingest (the server answered 409 path busy), so containment was an in-place edit on the storage volume.

The fix is the most reusable idea in this post:

# Redact by verbatim value: collect every live secret value, then scrub any text containing one.
def secret_values(min_len=10):
    vals = set()
    for secret in list_cluster_secrets():                 # every namespace
        for key, value in secret.items():
            if looks_secretish(key) and len(value) >= min_len:
                vals.add(value)
    return vals

def redact(text, known=secret_values()):
    for v in known:
        text = text.replace(v, "[REDACTED]")
    return text

Repository snapshots are gated the same way: a file containing any known value is dropped. The value list then caused the opposite problem: some short credential hints were plain six-letter words, so ordinary sentences were refused. Hints now need the same “secret-ish” test as cluster secrets, meaning at least 10 characters and high entropy or a mix of letters and digits.

Ingest is not free #

Summaries are written by the shared 9B model on the same GPU as everything else. A test of 16 KB of Markdown in 4 files cost about 23,000 input and 4,600 output tokens and took 416 seconds at one request at a time. The full import took about 5 hours of shared GPU time for 610 files, with 6 to 7 parallel requests, no errors and about 7,995 embeddings; the GPU memory temperature peaked at 82 °C.

Folder summaries are built bottom-up, but the parent refresh is best effort and skipped when the workers are busy, so under load the top-level descriptions went stale (one described the roadmap as a pastebin repository). A script now reindexes folders level by level after an import. And a recursive listing silently stopped at about 640 of 1,023 folders, so completeness checks walk folder by folder.

Measuring retrieval #

A sealed test set of 15 real questions, run with the agent’s key, top 4 results:

Setup	Correct (of 15)
SQL keyword search on the old database	8
vector search, entries bundled per month and project	9
both combined	11
one fact per file + content rerank	12

The misses were never “not stored”. They were “stored but not retrieved”, for two reasons.

Bundled chunks. The importer packed several roadmap entries into one chunk, so a one-line fact shared its vector with a dozen others. The new export writes one file per task, observation or mistake, each with a short plain-language header describing its project: 5,587 files, imported with vectors only, so no per-file summary cost. Folder summaries still cost model time, so the import took about an hour.

Near-ties. Search is one global vector search, and the embedding scores clustered between 0.75 and 0.79, so paraphrased questions lost to near-identical scores. The built-in reranker scored only the summaries, which are empty in vectors-only mode. The fix sits on the client:

vector search, top 20
  -> read the first 1,500 characters of each hit
  -> cross-encoder reranker (bge-reranker-v2-m3) on the question + content
  -> top 4

It takes about one second. The “log viewer” question went from rank 9 to rank 1. With the old bundled tree, the same reranker scored 9 of 15 because the right entry was not in the top 20 at all: both changes were needed.

How agents should use a memory #

Two mistakes by my local agent taught the usage rules:

• Asked for the status of a pipeline, the agent ran 6 rounds of memory searches and then delegated a worker that browsed files for over 7 minutes, with no answer. One systemctl status call had it. Memory is history; status needs a live tool. After three history-only rounds, the agent now gets a nudge to check live state.
• On 18 October the agent recorded a “post-fix verification” that no bounce had arrived after midnight UTC on the 19th, 14 hours before that time. The bounce came at midnight. A check can only be recorded after its time has passed; the memory tool now refuses future-dated results unless they are worded as a plan.

Operations #

A nightly job runs the ACL drift check, syncs operational state, snapshots repositories (leak-gated) and mirrors the whole roadmap into git (leak-gated again). The storage folder is in the hourly host backup; the vector index can be rebuilt, the accounts file and the agent’s area cannot. One rule is written in capitals in my notes: never re-run the old diff import against the new roadmap, because it would delete everything written directly.

Checklist #

• Give each reader its own account and least-privilege areas; treat agent writes as untrusted data.
• Verify effective ACLs after setting them; do not trust a success message.
• Redact by verbatim secret value as well as by pattern, and scan with an independent tool before ingest.
• Make credential scripts re-run safe and verify the stored value.
• Store one fact per file with a plain-language context header.
• Add a content reranker over the top vector hits.
• Evaluate with a sealed question set before and after every change.
• Use memory for history; use live tools for status.

Frequently asked questions #

Why not just use keyword search for an agent memory? #

Keyword search failed on vocabulary: a question about a “log viewer” never matches an entry that names the product. In my test it scored 8 of 15, about the same as plain vector search, and the two together reached 11.

How do you keep secrets out of a vector database? #

Redact before ingest, by pattern and by the exact value of every live secret, and run an independent scanner on the export. Once a secret is embedded and summarised, removing every copy is much harder than never sending it.

Does chunking matter for retrieval? #

Yes. When several facts share one chunk, each fact’s signal is diluted. One fact per file, with a short header that says what it is about, was the biggest single improvement.

What does a reranker add to vector search? #

A cross-encoder reads the question and the candidate text together and scores relevance directly. When embedding scores are nearly tied, that is often the difference between rank 9 and rank 1.

The lesson #

A shared memory is only useful if every reader can find the fact, and only safe if no reader can poison or leak it. Both properties came from the same discipline: measure retrieval with a fixed test set, verify permissions after setting them, redact by value, and keep the line between “what happened” and “what is true right now” sharp.