Visa’s New AI Shopping Benchmark: What Small Sellers Can Learn Before the Holidays

What Visa tested

On November 5, Visa launched the Visa Agentic Commerce Benchmark, a test of how well AI models handle realistic shopping. The premise will sound familiar to anyone who has worked a shop floor: customers often don’t know exactly what they want. Someone may know they want a TV without knowing which specifications they need.

So the benchmark makes the AI do the salesperson’s job. Through conversation, it has to uncover the shopper’s requirements — in one scenario, a budget, format compatibility, HDR support for family videos, and gaming features — and then make the right call. If nothing fits, the correct answer is not to buy. To rule out lucky guesses, a run only passes if the AI uncovered enough to justify its decision.

The first round put 22 AI models from 12 labs through 15 shopping challenges, including buying a TV, a laptop, and a personal care appliance. Each model got three attempts at each scenario.

The results

Visa Agentic Commerce Benchmark, first round (November 5, 2026).

Two numbers matter most for sellers. The gap between 70.2% and 44.4% says that finding out what the buyer needs is the difference between a good and a bad AI shopping experience — and the information an AI can find about your product is part of that. And the 54% figure means a failed run more often than not didn’t end in “no sale”; it ended in a purchase that didn’t fit what the customer needed.

Visa says plainly that it isn’t claiming AI agents aren’t ready for commerce: its FAQ points to the best models passing 82.2% of attempts, and names consistency and reliability as the open problems. Anyone can take the test themselves or watch replays of AI attempts on Visa’s Payments Frontier site.

How shoppers plan to use AI this holiday season

Recent surveys show where shoppers actually are — mostly using AI to research, not to pay:

The takeaway: this season, AI will shape which products shoppers find and choose far more than it will complete checkouts. And the 49% figure is good news for small brands — an AI recommendation can put an unfamiliar seller in front of a buyer, if the AI can find enough about the product to recommend it.

What small sellers can do before the holidays

1. Write listings that answer a salesperson’s questions

Visa’s results reward AI that uncovers requirements, and an AI can only match a requirement to your product if your listing states it. Cover the questions a good salesperson would ask: exact size and dimensions, compatibility, materials, what’s in the box, who it’s for, and how it’s used. If your platform supports structured product data — the schema.org Product format many store platforms add behind the scenes — fill in every field you can. Shopify sellers can also review the facts AI agents receive in Shopify’s new knowledge base for agents.

2. Say who a product isn’t for

The benchmark counts “don’t buy” as a correct answer when nothing fits. Clear limits — “not compatible with …,” “fits up to …,” “not dishwasher safe” — help an AI rule your product out for the wrong customer, which means fewer returns.

3. Make returns and exchanges specific and easy to find

If an AI-assisted purchase misses, you want the fix to be painless for both sides. State the return window, condition rules, and who pays shipping in plain words on every product page, not just in a footer link.

4. Tell customers when they’re talking to AI

If you use a chatbot for questions or order help this season, say so up front. Talkdesk found 77% of shoppers want to know. Our guide to AI chatbots for business covers disclosure and handoffs.

5. Don’t accidentally hide from AI search

If you want AI assistants to recommend your products, make sure your site isn’t blocking the crawlers AI search tools use. Blocking AI training is a separate choice — our explainer on the new AI training opt-out shows how to do one without the other.

I sell [product] online. Act as an AI shopping assistant helping a customer decide whether to buy it.

1. List the questions you'd need answered to recommend this product with confidence — size, compatibility, materials, use cases, what's included, shipping, and returns.
2. Here is my product listing: [paste the title, description, specs, and return policy]
3. For each question, say whether my listing answers it clearly, vaguely, or not at all.
4. Based only on my listing, describe the customers this product would be the WRONG fit for.
5. Suggest specific additions to the listing. Don't invent specs — mark anything I need to fill in with [brackets].

What the benchmark doesn’t show

• It’s a lab test. Fifteen scenarios in a few product categories, three attempts each — not real shopping traffic.
• Checkout isn’t tested yet. The benchmark covers discovery and the buy-or-don’t-buy decision. Visa says it will extend testing to checkout, payment, and order placement, and add more scenarios and product categories.
• Consistency is the open problem. All three attempts at a scenario passed only 34.2% of the time, so one good result doesn’t mean the next one will match.
• The surveys come from companies in the business. Visa runs agentic-commerce payment products and Talkdesk sells customer-service AI. Talkdesk’s retailer figures come from U.S. retailers with 50 or more employees that already use AI; both consumer surveys were fielded in September 2026.

The trend is the same one showing up across retail: Walmart is setting spending limits and rules for AI shopping agents, and its marketplace team has said listing quality matters more as agents shop.

Common questions

Business Prompted guides are written for practical use and checked against the sources listed above. Read our editorial policy.