An AI’s reply is barely as up-to-date because the documentation it makes use of

Earlier this yr, a service provider taking in-person funds requested our AI assist assistant how you can shortly settle for a cost with their card reader. It checked our documentation and confidently replied: “faucet ‘Easy Cost.’”

However there was no Easy Cost button. We had renamed it to “Add Customized Quantity” months earlier than, however the documentation hadn’t been up to date. The service provider couldn’t discover what the AI described and finally gave up.

Assist reported the difficulty, we up to date the short begin information and the product web page, and the documentation was fastened. However this incident caught with me as a result of the assistant did precisely what it was imagined to do. It learn the documentation and shared what it discovered. The issue was that the documentation described a product that had already modified.

That is the primary level I would like you to recollect from this: once you ask an AI how you can do one thing in WooCommerce — like altering a setting, discovering a characteristic, or fixing an error — it solutions primarily based on the documentation, not on real-time information of the product. The product adjustments with each launch, however the docs are written by individuals. Any hole between them can result in a mistaken reply that sounds fully assured.

So we created one thing to shut that hole.

It’s a WordPress plugin, constructed as an inside Automattic mission, and it really works like a documentation author who’s at all times expecting adjustments.

It retains observe of WooCommerce releases and any incoming requests to repair documentation. When one thing adjustments, like a renamed button, a moved setting, or a brand new characteristic, an AI mannequin drafts an replace. The draft then has to fulfill sure requirements earlier than it will get printed:

1. We rating it from 1 to five utilizing our high quality rubric, which seems at accuracy, voice and tone, construction, grammar and elegance, findability, and visuals.
2. It additionally has to move automated security checks. If a draft deletes greater than about 30% of a web page, breaks the web page’s formatting, leaves a placeholder, or provides a brand new picture hyperlink, it will get flagged for a human to assessment.
3. A author from our staff evaluations the draft in a preview and approves it earlier than it goes reside on woocommerce.com.

My favourite a part of the rubric is within the findability part: the primary paragraph of each doc should work as a standalone reply that an AI can quote. We’re not simply writing documentation for individuals scrolling by way of a web page anymore. We’re creating the supply materials for each AI reply about WooCommerce, and our rubric displays that.

We don’t simply rating new drafts. We additionally rating the reside documentation utilizing the identical rubric.

By late June, 39 out of the 117 WooCommerce docs we reviewed had been under our high quality requirements. That’s why we constructed this method. WooCommerce updates on a regular basis, and even the most effective staff of writers can’t re-read each web page after every launch. However the plugin can. It checks pages as a author would with limitless time, and it flags those that want consideration.

One a part of the pipeline robotically applies our terminology checklist to maintain product names constant, like utilizing “WooCommerce” as a substitute of “woocommerce.” It does this with a direct find-and-replace. 

Some entries within the checklist had been written as writing recommendation as a substitute of tangible phrase swaps. The system couldn’t inform the distinction, so it pasted the recommendation into reside pages. For a short while, a broadcast doc advised readers “(descriptive hyperlink textual content describing the vacation spot)” the place a hyperlink ought to have been.

I seen this whereas engaged on one thing associated. We audited all 258 WooCommerce docs, discovered 9 that had been affected, fastened each, then eliminated the problematic entries and added a safeguard to forestall this sort of mistake from occurring once more.

I’m sharing this as a result of it’s the identical lesson from a distinct angle. Automation follows guidelines precisely as they’re written, shortly and with out judgment. That’s what makes it each useful and dangerous. Our system is constructed with this in thoughts: strict checks that fail safely, a human reviewing something necessary, and audits for the automated steps.

What this implies once you’re troubleshooting

For those who ask an AI assistant a WooCommerce query and take the reply at face worth — and most of the people do, since that’s the purpose — the standard of the reply was set earlier than you even typed your query. It relies on whether or not the documentation matches the product you’re utilizing.

Our job is to maintain the documentation correct, and the system above is how we try this. There’s a smaller half for you, however it’s nonetheless necessary:

• If an AI’s directions don’t match what you see in your display, the documentation is perhaps outdated. Verify the doc it’s utilizing earlier than pondering you’ve made a mistake.
• Tell us when you spot a problem. When utilizing our AI assistant, price a reply thumbs-down or faucet ‘Contact Assist’ to inform us what’s mistaken. For those who’re on a doc that doesn’t appear present, inform our staff through the ‘Was this information useful?’ thumb on the backside of the web page.

The service provider with the cardboard reader didn’t do something mistaken. Neither did the AI. The accountability was ours, and constructing a system that makes fixes shortly — and catches adjustments earlier than you ever discover them — was the easiest way we might study from that have.

David is an AI Engineer at Automattic who makes a speciality of transport manufacturing LLM infrastructure for buyer assist and documentation programs throughout WordPress.com and WooCommerce.