Why local LLMs could not drive an autonomous pentest agent

I wanted to add an autonomous security-testing agent to my home lab. I installed it, pointed it at a tiny app with three deliberately planted, textbook weaknesses as a self-check, and let my best local model drive it for 35 minutes. It made 23 model calls, read 1.44 million input tokens of context, and found none of the three planted bugs. It never even opened the app’s source file.

That negative result taught me more about running agents on local models than any success had. This post is why a frontier-grade autonomous agent could not be driven by my local LLMs, and how to tell a configuration problem apart from a model that is simply not capable enough.

Contents · 9 sections

TL;DR #

• Autonomous multi-agent tools are built and tuned for frontier models. On my local 27B and 35B models the mechanics could be fixed; the long-horizon reasoning could not.
• Check the boring blockers first: tool-calling mode, prompt size and prompt-processing speed. A 70,000-token prompt at 250 tokens per second is about 280 seconds per model call.
• Forcing the model to always return a tool call fixed the empty-turn loops but did not produce useful work: it looped on bookkeeping tools instead of reading the target.
• Always evaluate an agent on a task with a known answer key before trusting it on anything real.
• The decision to shelve a tool is a result worth writing down, with the evidence, so the next person does not repeat the test.

What the tool is #

The agent I tried (an open-source project, Apache-2.0) is structured as a graph of agents: a root agent studies a target and spawns specialised sub-agents, each running in its own Docker sandbox, each driving an LLM through OpenAI-style tool calls. The model chooses an action, the sandbox runs it, the result is fed back, and the loop repeats until it produces a report.

Two facts shaped where it could run in my lab:

• It needs a real Docker daemon for its sandboxes. My cluster is k3s on containerd, so the agent runs on the host, which does have Docker, rather than as a cluster workload.
• It is designed for the reasoning depth of frontier models. That is the part local models could not supply.

My rule for it, written down before any run: internal targets only, a spend cap on every run that uses a paid model, and never any prohibited use. An autonomous tool that acts on a target is treated as traffic from my own address, with the authorisation recorded per run. This post is about the engineering, not about how to attack anything.

The boring blockers, in order #

Before concluding anything about intelligence, I worked through the mechanical reasons an agent stalls on a local model.

1. Does the model return tool calls at all? #

My local runtime is a llama.cpp server with its chat-template mode on. It does return proper OpenAI-style tool calls, but how you ask matters:

• tool_choice: "required" works reliably: the model must emit a tool call.
• tool_choice: "auto" is unreliable here: the model often declines to call a tool and writes an answer (or a hallucination) instead.
• The named-function form of tool_choice (an object naming one function) is not supported by this runtime.

With auto, the agent kept ending a turn without a lifecycle tool call, which its loop treats as an error, so it span in place. Setting the “always force a tool call” option removed those empty turns entirely. That is a real, reusable fix for local-model agents, and it is worth trying before you blame the model.

2. How big is the prompt, and how fast is prompt processing? #

The agent’s prompts are large: around 70,000 tokens once the system instructions, tool specs and history are included. Prompt-processing speed then dominates everything:

• the 35B model processed prompts at roughly 250 tokens per second, so a single call took about 280 seconds before it generated a word, on top of the empty-turn looping. Unusable.
• the 27B model processed at roughly 786 tokens per second, about three times faster, with no empty turns once tool calls were forced.

So I tuned for the 27B: forced tool calls, reasoning effort low (these models run with thinking disabled), and a cap on tool-output tokens to keep prompts small and processing fast. Mechanically, it now ran cleanly.

Where it failed anyway #

With the plumbing fixed, the 27B still produced nothing useful on the three-weakness self-check. The failure was specific and telling:

• it looped on orchestration and meta tools: listing agents, viewing the agent graph, waiting on sub-agents, stopping agents;
• it never read the target’s code and never ran a single sandbox tool against it;
• over 23 calls it generated only about 3,000 output tokens, and recorded 0 findings on the three-bug test app;
• it even asserted the weaknesses were “clear from the source” without opening the source.

I exhausted the meaningful knobs: model swap, forced tool choice, reasoning and output caps. The conclusion was not “misconfigured”. It was a model-competence ceiling: the long-horizon, self-directed reasoning that a multi-agent exploit loop needs is exactly what these local models lack. They can take one bounded instruction and follow it; they cannot run a 30-step investigation that decides its own next move each time.

Two contrasts made that precise:

• My other local security work is fine. A reconnaissance and intelligence pipeline that runs fixed, mostly deterministic steps on the same local models is solid and in daily use. It never asks the model to plan an open-ended campaign; the pipeline holds the plan.
• The same agent is reported to work well on frontier models. The design is sound; the brain was the variable.

The decision, and why I wrote it down #

I shelved the tool rather than deleting it: fully installed, with a one-page document recording the exact state, the tuned settings, the evidence above, and the conditions under which it is worth reviving (a frontier model via API for an authorised engagement, a stronger local agentic model, or faster prompt-processing hardware). Then a short entry in my lab’s shared record points to that page.

That is the actual deliverable of a failed experiment. Six weeks from now, “we tried the autonomous agent on local models and it scored zero on a known-answer test, here are the numbers and the settings” saves a whole day of re-testing. A shelved result is only wasted if it is not written down.

How this generalises #

The same pattern decides whether any agent can run on a local model:

1. Confirm tool-calling works and pick the mode that your runtime supports; force a tool call if the loop needs one every turn.
2. Measure prompt size and prompt-processing speed, and multiply. If each step costs minutes, a many-step agent is finished before it starts.
3. Shrink the prompt: cap tool output, trim history, disable reasoning if the model’s quality does not depend on it.
4. Run a known-answer evaluation. If a capable brain would obviously pass and yours loops on bookkeeping with zero results, that is the ceiling, not the config.
5. Prefer a fixed pipeline when the task is repeatable. Let code hold the plan and the model fill bounded steps; that is where local models earn their keep.

Checklist #

• Separate mechanics (tool-calling, prompt size, speed) from competence before judging a model.
• A 70,000-token prompt makes prompt-processing throughput the whole cost; measure it.
• Force a tool call per turn when the agent loop requires one, to stop empty-turn loops.
• Evaluate on a target with a known answer key, timed, with a token and spend budget.
• When a tool does not fit, shelve it with evidence and a revive condition instead of deleting it.
• Keep autonomous action tools internal, authorised and budget-capped.

Frequently asked questions #

Why can’t a local 27B or 35B model drive an autonomous agent? #

The mechanics can be made to work: these models return tool calls and, when forced, never skip a turn. What they lack is sustained, self-directed reasoning over many steps. On a known three-weakness test mine looped on bookkeeping tools, never examined the target, and reported nothing.

Is the problem the model or the configuration? #

Both, in order. Fix the configuration first: tool-calling mode, prompt size, processing speed. If a capable model would clearly pass the same task and yours still produces zero after those fixes, it is a competence ceiling.

Why do large prompts matter so much on local hardware? #

Because prompt-processing time scales with prompt length. At 250 tokens per second a 70,000-token prompt costs about 280 seconds before the model writes anything, and a multi-step agent pays that on every step.

What do local models do well, then? #

Bounded, well-specified steps inside a plan that something else holds. My deterministic reconnaissance and research pipelines run on the same models and are reliable, because the pipeline, not the model, owns the sequence.

The lesson #

Not every agent belongs on local models, and finding that out cheaply is a skill. Rule out the mechanical blockers first, because they are real and fixable; then run one honest test with a known answer. If the model loops on its own bookkeeping and never touches the task, you have hit a reasoning ceiling, not a bug. Write down the result, the numbers and the condition for a second try, and move the work to a fixed pipeline that plays to what local models actually do well.