AI Automation vs AI Assisted: How to Tell the Difference
I ran 1,000 invoices through an agent at 94% accuracy. That missing 6% is why a human still approves every run. Here is the real line between AI automation and AI assistance, what closing the loop costs, and how to tell which one you actually bought.
Most tools that say "AI-powered" are AI-assisted: a human still approves every run. Automation closes the loop without you watching. Last month my invoice agent parsed 1,000 documents at 94% accuracy on its own, and that missing 6% is the entire reason a person still has to sit there. That is the gap between assisted and automated.
What is the difference between AI automation and AI assisted?
AI automation is a system that finishes a task and commits the result with no human in the loop. AI assistance is a system that produces a draft and waits for a person to approve, edit, or retry it.
The test is one question: if everyone logs off, does the work still complete correctly? If the answer needs a person at a keyboard, you have assistance. The marketing page can say "automated workflow" all it wants. The approval button is the tell.
Most of what ships as "AI automation" is a fast assistant with a confirmation dialog. That is genuinely useful. It is also not the same product, and the difference shows up in your headcount and your nights.
Why do most AI tools stop at assisted?
The error rate. That is the whole answer, and it is worth being specific about why.
My invoice agent runs on Claude Sonnet 4.6. On clean PDFs it hits about 98%. On the real mix of scanned faxes, rotated images, and three-page statements stapled into one file, it settles around 94%. Six percent of 1,000 invoices is 60 wrong results. If each wrong result is a payment, you cannot ship that unattended. So the vendor adds an approval queue, calls the queue a "review step," and the human is back in the loop.
Here is the trap. The demo runs on the inputs that work. Your production traffic is the inputs nobody tested. A tool looks automated in a 5-minute demo because the demo never hit the 6%.
Automation is not a model capability. It is a decision about what you do with the outputs you cannot trust. Until a tool answers that, it is an assistant with good PR.
The other reason is liability. At 94%, someone owns the 6%. Vendors would rather that someone be you, clicking approve, than them.
How can you tell if a tool is actually automated?
Run the outputs through this. If any answer lands in the assisted column, the tool is assisted, whatever the homepage says.
| Trait | AI-assisted | Automated |
|---|---|---|
| Human action per run | required | none |
| Handles its own errors | surfaces them to you | retries or escalates by rule |
| Runs unattended on a schedule | no | yes |
| Who catches a bad output | you do | a rule does |
| Throughput ceiling | your attention | rate limit or budget |
| What happens at 3am | it waits | it runs |
The single sharpest question: what does the tool do when it is not sure? An assistant shows you and stops. An automated system has a written rule for the uncertain case, commits the confident ones, and routes the rest. The rule is the product.
What does it cost to actually close the loop?
More than the assisted version. This surprises people.
The line between the two modes is one function: a confidence gate that decides what gets committed without you.
// the whole difference between assisted and automated lives here
function route(result: AgentResult): Action {
if (result.confidence >= 0.97) return "commit"; // closes the loop
if (result.confidence >= 0.80) return "queue_review"; // human stays in
return "reject_and_retry"; // retry, then escalate
}
The catch is that a model's self-reported confidence is unreliable, so you usually pay for a second model to check the first before you let anything auto-commit. That verification pass is the real cost of automation.
Run the numbers on my invoice job. The primary parse on Sonnet 4.6 ($3 per million input, $15 per million output) runs about 2,000 input and 500 output tokens per document, so roughly $0.014 each, near $14 per 1,000 docs. To trust auto-commit I add a check pass on Claude Opus 4.8 ($5 / $25 per million) that re-reads the parse and votes. That adds about $0.02 a document. Closing the loop roughly doubled my per-run cost, to near $34 per 1,000.
For the record, the run that produced this post plus the pricing fact-check cost about 30 cents. The invoice job that started the whole thing costs more every morning, and it earns it by not needing me.
That is the trade. Automation costs more per run because you now pay the model to do the checking the human used to do for free. You buy it back in volume and in not staffing a review queue.
What to check before you call anything automated
Pick one task your tool claims to automate. Turn off the approval step for a single day, log every output it would have committed, and grade them by hand at the end. Two numbers fall out: the real accuracy on your data, and the cost of the worst wrong output. If the worst case is cheap and the accuracy clears your bar, you can close the loop. If not, you own an assistant, and you should price your own time accordingly, because you are the automation.
Is AI-assisted worse than automated?
No. For low-volume, high-stakes work, a person on every run is the correct design. You automate the task you run 500 times a day. The task you run twice a month is not worth a confidence gate. The mistake is paying for automation and receiving assistance without noticing.
Does "human in the loop" mean a tool is not automated?
For that step, yes. A human approval inside the run is the definition of assisted. The honest version puts a human on the loop instead of in it: the system runs and commits on its own, and a person audits a sample after the fact. That keeps oversight without making a person the bottleneck.
How accurate does an agent need to be before you can automate it?
It depends on the cost of a wrong output, not a fixed number. For reversible, low-cost actions like tagging an email or drafting a reply for later edit, 90% can be fine. For payments or anything you cannot undo, even 99% may need a verification pass or a confidence gate that routes the bottom slice to a person. Grade the cost of being wrong first, then set the threshold.
Tired of re-keying the same data between tools? Pylonworks builds custom automation and internal tools for businesses without a developer, on a fixed quote you approve up front. Tell us what's eating your time