Scraping Products: Prices, Stock and Listings

A product listing is the most common object of collection, and it has quirks that regularly make the collected data wrong while the scraper appears to work.

What a listing contains

Identifiers. SKU, barcode, the identifier in the page URL. The most important field: without it, collected data can neither be matched nor updated.

Name and brand. For humans, not for matching.

Price. The most sought-after and least reliable field - see below.

Stock. Often more important than price.

Specifications, usually as a set of pairs whose structure differs per source.

Images and description. Careful: these are copyrighted works, and collecting them for use on your own site is a different matter from collecting figures for analysis.

Why price is unreliable

Five causes, all of them routine:

It depends on region. One product, different cities, different prices. Collect using an address in the target region and label the result - see the proxy guide.

It depends on sign-in. Wholesale prices are visible only to authenticated users.

It depends on quantity. A per-unit price differs from a per-pack price, and either may be the one on the page.

Conditional promotions. “With a loyalty card”, “on orders over”, “until Friday” - on the page that is several numbers, and the scraper must know which one it is taking.

Personal offers. Some platforms show different prices to different visitors.

The practical conclusion: store the price together with its collection conditions - region, sign-in state, date, price type. Without that, a discrepancy with reality cannot be explained, and you will be asked to explain one.

Matching across sources

The main difficulty when collecting from several sites.

Reliable: barcode, manufacturer part number, exact brand and model match.

Unreliable: the name. “Smartphone X 128 GB black” on two sites can be different bundles, model years and warranties.

What to do:

• Collect every available identifier, not just one.
• Match on the stable one and fall back to the name last.
• Record match confidence: exact, probable, manual.
• Spot check by hand. Twenty items a week show match quality better than any internal metric.

Stock and change

Stock is expressed differently across platforms: an exact number, “low”, “in stock”, “to order”, a lead time. Normalise to your own scale at collection time rather than at analysis time.

Do not delete disappeared products - mark them unavailable. A product may vanish for a week, and the history of items appearing and disappearing is often more valuable than the current snapshot.

Price history is worth more than the current price: movement shows a competitor’s strategy while a snapshot shows a moment.

What to validate

Automatic checks that catch most problems:

• Price within a plausible range. Zero, negative, or a hundred times yesterday’s value means parsing broke.
• Mandatory fields populated. An empty name is a failure, not a nameless product.
• The item count is comparable with last time. A sharp drop means breakage, not a vanished catalogue.
• No duplicates by key - see deduplication.
• Correct types. A price arriving as a string with spaces and a currency symbol needs converting, and that belongs at collection time.

The general rule: a scraper should report the suspicious rather than silently write to the database. A thousand items at zero price is an incident, and you should learn about it immediately rather than from a colleague a week later.

How this works technically: scraping in Python. Marketplace specifics: their own article. The overview is in the scraping guide.