Blog · tag

#lessons-learned

31 posts tagged #lessons-learned.

RSS · Atom · JSON Feed · All posts

  1. Four agents, four lessons: how my local AI agent grew up

    I built four AI agents on one home server in seven months. Each failed in a new way: code that was not running, reports that said all clear, tasks closed with no work, a deleted home folder. Each failure moved trust out of the model.

  2. After the wipe: six habits that changed how I run my home lab

    A test deleted my home folder in ten seconds. Recovery took two days; the habits it left took longer. Freeze before you fix, reboot on purpose, back up the boring files hourly, commit everything, and give agents their own user.

  3. One machine, everything: what I would do differently, and what's next

    One workstation runs my whole lab: Kubernetes, a 120,000-work library, honeypots, local AI. Seven months of records hold 253 mistakes. What one machine gave me, what it cost, what I would do from day one, and what comes next.

  4. When my datasets started talking, the loudest link was my RSS feed

    In May 2026 a linker joined my lab's separate security datasets into one graph. On 8 July its strongest cross-layer link turned out to be an RSS reader polling my own blog. 1,885 pairs became 6, and I learned to ask what chance would give.

  5. One docker rm -f took down my whole k3s cluster

    Freeing a busy port with docker rm -f by filter removed the external Postgres that holds my k3s cluster state: about 20 minutes of total outage. How k3s datastores work, why recovery was quick, and the guard rails I added.

  6. The cleanup job that deleted 239,000 live search chunks

    A nightly orphan-chunk sweep in my Elasticsearch library deleted 239k live chunks from 19.4k documents for four weeks. Why its liveness check was wrong, how I recovered, and the count-first circuit breaker that followed.

  7. 13 hours of question marks: a canary for LLMs that look healthy

    After a llama.cpp router restart my main model answered every request with question marks for 13 hours while every health check stayed green. A known-answer canary now catches it within 10 minutes.

  8. How small local models finish hour-long agent tasks

    Left alone, my 9B agent closed tasks without doing them and invented a 10 KB report from an empty sub-agent result. Scaffolded task trees, evidence checks and deterministic gathering let it finish 66-minute projects.

  9. A post-reboot check script for a one-machine home lab

    A cold boot of my lab produced 37 failed jobs; three were real defects hidden for up to two weeks. Here is the read-only OK/FAIL script I now run after every reboot, and the failures it catches.

  10. One GPU, many LLMs: the VRAM guards that stopped a driver deadlock

    With 33.9 of 34.2 GB of VRAM in use, a desktop app triggered a 24.7 GB eviction and froze my GPU driver. Here is the VRAM budget and the two guards that now enforce it.

  11. Swapping an 8 GB NVIDIA card for a 32 GB AMD GPU: what broke

    I replaced an RTX 4060 Ti with a 32 GB Radeon AI PRO R9700 for local AI. Ten things broke, mostly silently; here they are with fixes, and the before-and-after throughput.

  12. Hybrid search with bge-m3 and Elasticsearch: six lessons

    What I learned building BM25 + kNN hybrid search with reciprocal rank fusion and bge-m3 on one shared GPU: an analyser mismatch (38 to 2,631 hits), RRF pitfalls, embedding timeouts and a highlighting limit.