The agent did exactly what we wrote: why AI automations fail quietly

AI automation rarely fails loudly. It throws no error, it stops no import, it wakes nobody at night. It finishes successfully and hands back output that looks fine. The catch is that the model followed the instruction literally, and the instruction meant something other than what its author had in mind.

People keep saying that models "understand". They do not understand, they execute. That difference stays invisible while an instruction is unambiguous and turns expensive the moment it is not. Below: three defects we measured in our own work, and the same mechanism applied to catalogs, attributes and feeds in a store.

It helps to separate two kinds of failure, because you defend against them in completely different ways:

• A loud failure: the script exits with an error, the import stops, the queue grows. You see it the same day and somebody fixes it.
• A quiet failure: the job succeeds, the data looks reasonable, and yet every output carries the same defect. You see it only in aggregate, and only if somebody bothers to aggregate.

Defect 1: one ambiguous sentence in the brief

We write guide chapters with the help of agents. The editorial brief contained one line: "one table or one list". The intent was a proportion, as in do not clutter the text with bullet points. The agents read it as a hard cap and did exactly what they read.

The scale of it only surfaced once we counted with a script:

• 10 chapters out of 12 contained not a single list.
• 31 paragraphs ran past 700 characters.
• One chapter had 20 paragraphs above 600 characters, out of 24 in total.

The interesting part is why nobody flagged it for a week. Each text on its own looked fine: coherent, factual, nothing wrong with it. The defect existed only in the aggregate, and nobody aggregates until there is a reason to. An ambiguous line did not spoil one chapter. It spoiled the whole batch, in the same direction, for the same reason.

Defect 2: one name, three meanings

The second defect is quieter still, because it leaves no trace in the output, only in behaviour. In our open set of skills for agents the argument --profile meant three different things in three places: a set of validation rules, a type of assortment, and a path to a config file.

A human in that situation asks. An agent does not ask: it takes the most likely reading and moves on. The result is formally correct and different from what was intended, and no error message ever appears, because from the point of view of the tool nothing bad happened.

The fix was boring and effective: one name, one meaning across the whole set, and separate names for separate concepts. A name collision is the cheapest kind of quiet defect to remove and the one most often missed, because names get picked in passing and then live in several files at once.

Defect 3: without measurement you have an opinion, not data

The first defect came out of no alert. It came out because a human read the text and said it was hard going. Only then did we count paragraph lengths, the number of lists and the number of subheadings. The impression turned into a task the moment it got numbers attached.

While we were at it, a single command found 57 unclosed quotation marks across three documents: an opening mark with no closing pair. The eye does not catch that while reading, and it repeats in every file produced by the same process. The check took a minute, and it simply did not exist before, because nobody had thought to run it.

The same thing happens in store operations

A guide chapter is a low-stakes case. In a store the same mechanism hits descriptions, attributes and feeds, which is the data a customer sees and buys on. An imprecise rule does not spoil one product. It spoils all of them at once, in every channel, until somebody rewrites the rule.

Product descriptions: "concise" is a wish, not an instruction

"Descriptions should be concise and sell well" sounds like an order and is a wish: you cannot check it with a script or settle an argument with it. An agent handed an adjective sets its value on its own, and sets it differently for a power drill than for a pair of earrings.

It becomes an instruction only once four things are in place:

• Range: 400 to 600 characters, three paragraphs, a concrete benefit in the first sentence.
• Required fields: material, dimensions, country of origin, care instructions, wherever the product record holds them.
• Banned claims: health, medical, "best on the market", and anything absent from the source data.
• Two examples: one good description and one bad one, plus a single line on why the bad one is bad.

That last point is the cheapest and the most often skipped. A model hits the intent far better from one negative example than from three paragraphs of guidance, because the example settles edge cases the guidance never anticipated. What a full rule set looks like, we laid out in the piece on AI product descriptions done right.

Attributes: black, BLK and jet black are three values

Take an ERP export: a few thousand rows and a column called colour holding black, Black, BLK, jet black and dark. To a human that is one colour. To the store filter, to the feed and to the agent those are five separate values, until somebody writes down which one is canonical and what happens to the rest.

A value dictionary settles three things at once:

• Canonical value: the catalog gets "black" and only that form.
• Synonyms: BLK, jet black and Black map onto "black" instead of spawning another entry in the filter.
• Missing data: an empty cell stays empty and goes onto a list to fill in, instead of getting a colour inferred from the product name.

Without a dictionary the agent guesses, and in most rows it guesses sensibly. That is worse than guessing wrong everywhere, because a defect scattered across a catalog is harder to spot than a consistent one. Why clean data is a precondition for any automation, we cover in the guide chapter on data as a prerequisite.

Feeds: spell out what to do when data is missing

A feed specification is where a quiet failure costs fastest, because the data goes straight into a sales channel. Three things have to be written down explicitly: the title length limit, the fields each channel requires, and the procedure when data is missing.

The last one matters most and gets skipped most. An agent told to "fill in the missing data" will fill it with a value that sounds plausible, because that is precisely what it was asked for. An agent told to "mark gaps as MISSING and never guess" hands back a catalog with holes and a list to work through. The second version looks worse in a report and is the only one that does not mislead a customer.

The difference between a wish and an instruction is clearest row by row:

An adjective leaves the decision with the model. A number and a dictionary keep it with you.

Scale cuts both ways

Manual work produces single, random errors: a person slips irregularly and slightly differently each time. Automation produces systemic errors: the same mistake in every row, consistently and without exception. That is at once the greatest advantage of automation and its greatest risk.

Two sides of the same property:

• The advantage: one fix to the rule repairs the entire set, because the error has a single source. Manual slips cannot be fixed in bulk, you have to hunt them down one by one.
• The risk: until that fix lands, the same error is visible to every customer, on every product page and in every channel that already pulled the feed.

So with automation it pays to move attention from the individual output to the rule that produced it, and to the sample somebody reviews before the full run goes out. How to do the arithmetic on savings from such a process, we work through in the anatomy of automation savings, and the approval step itself we describe as the human in the loop.

An instruction for an agent is an interface, not a note

A note is read with goodwill and the rest is inferred. An interface executes literally and returns exactly what was written into it. An instruction for an agent belongs to the second category, so you write it like a contract, not like a message to a colleague who already knows the context.

Three things remove most quiet defects:

• A defined vocabulary: one name, one meaning across the whole process, with values coming from a dictionary rather than from the model.
• A checkable criterion instead of an adjective: not "make it readable", but a number a script can count: length, number of fields, allowed values.
• A measured sample before you believe it works: a hundred rows or a dozen texts is enough to expose a systemic defect. Without it you do not know whether you have quality or merely no complaints.

We described the same mechanism from the code side, where verification, not writing, is the bottleneck. In product data it works identically, only the defect is quieter, because data carries no types and no tests until somebody writes them. On our side that layer, meaning the dictionaries, the rules and the checks on a sample, is part of the AI Automation stage and ships together with the automation it guards. It guarantees no clean run. It shortens the time between a mistake and somebody noticing it, and that is what the craft comes down to.