/ / 2 min read

Measurement before optimization

Deepanshu Pal

Why agent readiness should be measured before it is optimized: a plain method for testing whether a machine can find, compare and buy from your store.

There is a familiar order of operations in online retail: ship the change, watch the conversion rate, repeat. It works when the customer is a person whose behavior shows up in your analytics. It breaks when the customer is a machine that leaves no trace except an abandoned task somewhere else.

You cannot optimize what you cannot see

If an agent fails to buy from your store, there may be no session, no cart and no bounce. It simply never recommends you. The first job is therefore not optimization. It is visibility: a way to see what an agent sees and where it gives up.

A simple method

  • Pick a small set of real tasks, such as find this product, compare it to two others and complete a test purchase.
  • Run them against your store with an agent, the same way each time.
  • Record where each run succeeds, stalls or fails, and why.
  • Score the store on the tasks it completes, not on how it looks.

The scores matter less than the repeatability. A benchmark that gives the same answer for the same store tomorrow is a benchmark you can improve against.

Fig. 1 · Measure, change, re-measureIllustrative data

Where the runs end

Share of runs reaching each stage

Readiness score

Tasks completed end to end

Same store, five runs

SucceedsStallsFails
Tap a cell to see where and why a run ended.

Score across benchmark runs

Move across the chart to read a run.
Illustrative figures to show the method. They are not measurements of any store.

Then optimize

Once you can measure, optimization is ordinary engineering. Fix the page where price is missing. Simplify the form that traps agents. Re-run the tasks. Each change either moves the result or it does not.

Measured, not marketed

Plenty of vendors will say a store is agent-ready. A readiness claim without a test behind it is marketing. We publish open research, conformance tooling and public benchmarks so the claim can be checked by anyone, including the merchant.

The next read / Manifesto

The machine customer.

Commerce has a new reader. What changes when that reader can buy?

Read the manifesto