Letting my AI agent act: autonomy with guardrails, a work in progress

On 11 October 2026 my local AI agent was given two things to do with nobody watching. Restart one named service in my Kubernetes cluster. Delete a Kubernetes object.

It restarted the service on its own, logged why, and sent a note to my phone saying it had done so. It did not delete anything. The deletion arrived in my inbox as a proposal, with the exact command and a reason, waiting for me.

That was the first time since an AI test deleted my home folder, eight days earlier, that my local AI agent was allowed to change something without asking me first. One action. One kind of action. This post is about why I opened that door at all, how the door is built, and where it is heading. It is not finished. I will tell you which parts are not.

Contents · 10 sections

In one minute #

• After the wipe, every unattended run on my machine could only propose changes. Safe, and a growing pile of work for me.
• On 11 October I approved a plan: let the agent do small, reversible things on its own, behind fixed rules in code and a second opinion from a separate decision model.
• The order of checks is fixed: a deny list, an allow list, a cooldown, a dry run by default, then the decider. Any “no”, any doubt, any error: the action is proposed instead.
• The allow list today holds one action. It grows only when a test case for the new action exists.
• The first autonomous project ran to the end on 12 October. Watching it fail produced about a dozen fixes. Watching it succeed produced three corrections.
• This is a work in progress. The design is stable. The trust is still being earned.

The before: a very safe agent and a very full inbox #

After 3 October, the rule on my machine was simple: if no human is watching, nothing changes. Scheduled reports, monitors and helper agents could read anything they were allowed to read, and every change they wanted became a proposal. I described that gate in read, write, critical: permission tiers for a local AI agent.

It worked. It also meant that a failed nightly job, a pod stuck after running out of memory, or a service that needed a restart waited for me, every time, even when the fix was obvious and harmless. My cluster runs around 150 pods on one machine. Small things fail every day.

On 18 October I said what I actually wanted, in two words: “open more”. The agent was too strict. I wanted it to fix out-of-memory cron jobs and failed jobs by itself. But I put the condition right next to it: no rm -rf, no database changes without my consent.

That tension is the whole post. Too strict, and the agent is a very expensive notification system. Too open, and you get a Saturday morning like 3 October.

The trigger: a plan with a frame #

The plan I approved on 11 October had a sentence at the top that I still like. The frame I set was: cooperative and trusted, watched on both sides. Autonomy with guardrails. Everything recorded in the roadmap, in Git and in tests.

It had three phases:

1. Foundations. Let a scheduled job move a project forward one bounded step at a time. Build an auto-approve lane for a few safe actions. Make it easy for me to watch any run live.
2. A second agent of mine that needed native tool calls on my local model server. (I will leave that one for another day.)
3. Watched autonomy. Give the agent real items from my backlog, as scoped projects, and watch it work through them.

And it had a list of principles. Two of them carry the weight. Work must be sized for local models: small, scaffolded tasks that a 9-billion or 27-billion-parameter model can actually finish, with heavy steps routed to bigger local models. And the safety spine is absolute: the deny list, the second-factor code for critical actions, dry runs, cooldowns and the audit log do not move, whatever the model says.

The technical heart: how an action is allowed to run on its own #

Here is the lane, in the order it is checked. An unattended action runs only if every step says yes.

unattended action requested
  1. deny list        never: cluster datastore, volumes, namespaces, secrets ...  -> blocked
  2. allow list       is this exact action class allowed at all?                   -> else propose
  3. cooldown         did we just do this? rate limit                              -> else propose
  4. dry run          default unless the action is explicitly live                 -> else propose
  5. decider          typed question to a separate model: auto_approve or propose?
                      act only on auto_approve with confidence >= 0.85             -> else propose
  -> run, log the reason, notify my phone

Steps one to four are plain code. The model cannot talk its way past them, because they never read what the model wrote.

Step five is the interesting one. The decider is my small decision service. It does not generate text. It asks a model a question with lettered options and reads the probability the model gives to each option label, straight from the model server. The model can only choose among answers I wrote; it cannot invent a third one. I wrote about that technique in typed decisions from logits.

The preset for this lane asks one thing: given this action, the allow-list entry it matched, whether it is reversible, whether it is idempotent and how wide its blast radius is, should it be auto-approved or proposed to the owner?

The description inside the preset says the design better than I can: the decider picks whether; the agent’s spine of deny list, allow list, cooldown and dry run decides if.

Earning the decider #

The first version used my 9B model. On the exact format of its test cases it scored 13 of 13, with no unsafe answers. Then I reworded a perfectly justified restart, the way a real agent would phrase it on a real day, and the 9B flipped to “propose”.

That is a safe failure. It is also a useless one. A decider that only says yes to sentences it has seen before is not a decider.

The 35B model was more robust, but on my single GPU it was often not loaded when the question came, and the request failed. So the lane stayed built, tested and switched off.

The answer was a 27-billion-parameter model that stays loaded. On varied, free-form rationales it scored 9 of 9, with zero unsafe answers. On 11 October I switched the lane on. A live canary: an allowed restart was auto-approved, logged and announced; a delete was proposed.

The rule behind that sequence is written in the decision service’s own README: never trust an unattended threshold that was not fitted on labelled data. The lane did not go live because the design was good. It went live because the decider passed a test it had not been shaped for.

Why only one action #

The allow list holds one action class: restarting a single named Deployment. A restart is reversible, idempotent, narrow and the most common fix in my cluster. Every new entry needs a matching test case in the decider’s evaluation set before it is added. Anything I am not sure about is not in the list.

Every auto-approval is logged with the decider’s reason, and every one is sent to my phone. One line in a configuration file turns the whole lane off.

The build: watching it fail #

The other half of phase one is the project driver: a scheduled tick that resumes a project, picks the next task, does one bounded step and stops. The project itself lives in a database, so the model never needs to remember anything between ticks.

The first real project was deliberately harmless: a read-only audit of the container images that my cluster runs, with a plan for improving them. Then I watched. Each failure became a fix, the same day:

• The first tick loaded about 470 tools into the model’s context. That is noise for a 9B model, and it kept every mutating tool within reach. Ticks now see only the tools of their project kind, and a read-only kind drops every mutating tool from the session, below the permission gate.
• One tick ran for more than 11 minutes on the shared model and was killed mid-work. Ticks now have their own round limit, 18 by default.
• The 9B picked the wrong arguments for a cluster query and delegated to a file that did not exist. The 27B wrote the right query. Ticks now use the heavier model.
• Stalled ticks kept handing work to sub-agents, which ran unbounded and lost the data gathered in the conversation. Ticks can no longer delegate.
• Even the 27B dropped one segment of a nested query path, and a whole tick produced empty columns. So the agent got a small, deterministic, read-only tool that lists images itself. The model no longer composes that query at all.

On 12 October the audit finished on its own, driven by the scheduler: a 20 KB report with findings per namespace and a four-phase fix order.

It was also wrong in three places. It counted one category as 3 when it was 30. One section header promised 15 items and listed 18. A restart count was off by five. All three were caught by checking the report against the data, and they are recorded next to the result.

That is what “watched on both sides” means in practice. The agent did the long, boring part. The checking stayed human.

The after: opening more, and the mistake that came with it #

On 18 October, after “open more”, the agent got the ability to fix common cluster problems by itself: delete a failed job or a stuck pod, restart a deployment, or raise a memory limit for a job that ran out of memory. The memory raise is bounded: at most double, never above a fixed cap, only after re-reading the out-of-memory evidence, and only by editing and committing that one manifest. These operations are not run by the agent. They are run by a small root daemon that re-validates each request itself.

The same day, I made a mistake that this post would be dishonest without.

Another operation added that day let the agent run commands as me, the owner, for admin work. The daemon executed those commands without asking for my code, trusting that the agent’s own permission gate had already approved them. But the daemon cannot see the agent’s gate. Anything running as the agent’s user could reach the daemon and ask. I had applied “open mode” to an operation that crosses a privilege boundary.

A review caught it the same day. The fix: every such call now needs my code, either as a one-time ticket or within a 60-minute window that my code opens. The lesson went into the record in a form I now repeat to myself:

Anything a more powerful process does on behalf of a less powerful one must be authorised by the more powerful side itself. Not by the caller’s promise that it checked.

Open mode is fine for actions at the agent’s own privilege level, and for narrow operations that the privileged side validates on its own. It is never fine for a doorway into more power.

What it costs #

Speed, first. Every auto-approved action passes five checks, and the decider runs on the same GPU as everything else. On 11 October the first watched autonomous project was paused for a while, on purpose, to free that GPU’s model slot for my own work. One machine means one GPU, and an autonomous agent competes for it with me.

Attention, second. Watching is real work. The report needed three corrections. The ticks needed a dozen fixes. Autonomy does not remove the human; it moves the human from doing to reviewing.

And humility. I designed a careful lane and, a week later, opened a careless door next to it. The system caught it because review is part of the design, not because I was careful.

Where this is heading #

Phase three has started, slowly, with that first audit. The next projects come from my real backlog: scoped, one bounded tick at a time, with a live view of each run that I can open from any device. Projects that would change things propose.

The next steps, in the order I expect them:

• A longer allow list, one entry at a time. Each new action class gets its test cases in the decider’s evaluation set first. Candidates are other small, reversible actions such as restarting a known service unit or re-applying a manifest whose diff shows it only restores drift.
• A better decider, measured. The option wording is the biggest quality lever: in my decision service, rewording one preset took it from 23 % to 85 % accuracy. New wording means a new preset version and a new evaluation run, never an edit in place.
• Ticks that fit the machine. Bounded steps, deterministic tools for anything a model keeps getting wrong, and heavy reasoning only where it pays.
• Better watching. A proposal inbox and a live view exist. The next step is making the trail of every unattended action easy to read the morning after.

What will not change: the deny list, the second factor for critical actions, dry-run defaults, the audit log, and “propose” as the answer to any doubt.

Questions people ask me #

Why not let the model decide directly whether an action is safe? #

Because a model that writes free text can write a convincing reason for anything. My decider can only choose between options I wrote, it reports a probability, and even a confident “yes” only counts after four checks in plain code have passed. The model contributes judgement; the code holds the authority.

What happens if the decision model is down or unsure? #

The action is proposed instead of run. Errors, timeouts and low confidence all fall back to “ask the human”. The lane can fail useless, but it is built never to fail dangerous.

Which actions can your agent take without asking? #

Today, one class: restarting a single named deployment, in unattended runs, after all checks pass. With me present, it can do much more, with my approval for every change and a code on my phone for anything critical.

Is this safe enough to copy? #

The pattern is: a deny list in code, an allow list in code, cooldowns, dry runs, a decider that cannot invent answers, a threshold fitted on labelled cases, an audit log and a human notified every time. If you copy that, start with one reversible action, and expect your own mistakes to be next to the design, not inside it.

The point #

Letting an agent act is not one decision. It is a long series of small ones, each with a test case, each reversible, each written down. On 11 October my agent restarted one service on its own and proposed a deletion it was not allowed to make. That is a small step for a machine that can call hundreds of tools.

It is the right size. Trust that arrives in one big step leaves in one big step. Trust that is earned one allowed action at a time can survive a mistake, including mine.