Scraping works... until it doesn’t.

As AI agents take on real-world tasks like discovering products, comparing attributes, and making purchases, they’re encountering a reality long known to human operators: commerce is messy. And for agents to be effective, the data they rely on must be reliable, interpretable, and actionable.

Yet many teams entering the agentic commerce space have defaulted to a common shortcut: scraping webpages. It’s fast. It doesn’t require permission. It gets you off the ground.

But in commerce, those shortcuts don’t scale.

The Two Paths: Scraping vs Structured Access

There are two primary ways to get commerce data into an agent:

• Scraping: Parse HTML pages designed for humans. Rely on CSS selectors, DOM trees, and visual structure.

• Structured Access (API or MCP): Access structured JSON responses from merchant systems, via authenticated or consented channels.

At first glance, both paths seem to get you the same thing: a product name, a price, an image. But under the surface, the differences are dramatic—and consequential.

Scraped pages are often missing key metadata. They don’t expose inventory counts, SKU-level pricing, shipping methods, or tax logic. They change frequently, break silently, and lack any guaranteed uptime. Scrapers have to guess what matters.

By contrast, structured access exposes the schema behind the store. Agents receive typed data, including:

• SKU-level availability

• Variant options with price and inventory

• Tax-inclusive pricing

• Shipping timelines and options

• Return and cancellation policies

These differences define whether an agent can act—or just speculate.

A Concrete Example: Variants and Inventory

Scraping Example

Imagine an agent is helping a user buy a black running shoe. It loads the product page and sees:

<select id="size">
  <option>Small</option>
  <option>Medium</option>
  <option>Large</option>
</select>

At best, the agent knows the product comes in three sizes. But it doesn’t know:

• Which ones are in stock

• If different sizes have different prices

• What SKUs they map to

• When the item can be delivered

Structured Access Example

Now imagine the same agent receives this instead:

{
  "variants": [
    {"sku": "123-S", "size": "Small", "price": 19.99, "in_stock": true},
    {"sku": "123-M", "size": "Medium", "price": 19.99, "in_stock": false},
    {"sku": "123-L", "size": "Large", "price": 21.99, "in_stock": true}
  ]
}

Now the agent can:

• Recommend in-stock variants

• Reflect accurate pricing

• Skip over unavailable items

• Build a cart that actually works

One of these is a guess. The other is a system.

The Cost of Scraping at Scale

Beyond the functional gaps, scraping imposes hidden costs:

• 🤨 Brittle structure: Layout changes break scraping logic

• 🔄 Outdated data: Scraped content is often cached, stale, or misaligned

• ⚠️ Silent failures: No clear error when checkout breaks or stock is unavailable

• 📬 No post-purchase control: No refunds, updates, tracking, or cancellations

• 👻 No merchant visibility: Agents operate in the dark with no attribution or partnership

• 💸 Server strain: Repeated page loads spike merchant hosting costs

• 🚧 Growing defenses: Cloudflare and others are beginning to block agents

• 💥 Request explosion: Agents make 10–20× more requests per query than typical scrapers due to multiple page loads per variant, cart, and checkout step (costly and inefficient)

Even platforms that welcome innovation will become hostile once agents start driving cost without coordination.

System Design: What Actually Happens in a User Flow

Let’s say the user prompts: “Find me a pair of black running shoes under $150 that ships by this weekend.”

Scraping-Based Agent

• Crawls known store URLs or product feeds

• Scrapes product names and prices from search results

• Cannot determine exact size availability

• Doesn’t know if the product ships to user’s zip code

• Cannot calculate tax or delivery date

• Tries checkout, but:

  • Item is out of stock

  • Variant wasn’t selectable

  • Page changed structure

  • Checkout silently fails

The result? A poor experience that looks like it worked—until it doesn’t.

Structured: API/MCP-Based Agent

• Sends structured query to merchant endpoint

• Filters by color, price, and delivery date

• Gets SKU-level stock and shipping info

• Selects in-stock variant that matches criteria

• Provides real-time checkout flow with known delivery timeline

• Reflects taxes and shipping based on user context

The scraped agent answers a question. The API agent delivers a result.

Merchant Trust and Monetization

Scraped flows are invisible.

Merchants can’t see where traffic comes from, can’t attribute conversions, and can’t monetize or build partnerships with agentic platforms.

Worse, they pay for the privilege:

• Increased hosting costs from aggressive bot traffic

• Elevated infrastructure bills for bandwidth and CPU

• Need for new protections against scraping

Platforms like Cloudflare are already rolling out pay-to-access models for AI agents, signaling a shift from tolerance to monetization.

Without merchant participation, there is no shared upside. No commissions, no attribution, no support.

Structured access via APIs or protocols is the only path to scalable, aligned execution.

Closing Argument: Access Is Architecture

Agentic commerce is not just about what an agent can do. It’s about how it gets the data to do it.

• Scraping is fragile, expensive, and invisible

• Structured access is durable, trusted, and monetizable

Building agents on scraped HTML is like relying on computer vision using a paper map—technically feasible, but fragile, indirect, and destined to fail under pressure.

In commerce, how you access the data is the most important part.