Skip to content
PD
EN RU
Browser Automation Studio 8 min read

Your Proxy Bill Is a Bandwidth Problem: How I Cut Bytes Per Record in Scrapers

Residential proxies are billed per gigabyte, and most scrapers waste most of it on images, fonts, trackers and retries. Here is how I measure bytes per record and cut traffic by 80 to 95 percent in BAS and HTTP scrapers.

PD

Pavel Duglas

AI Automation & MVP Architect

Most people think their scraping costs come from the number of pages they parse. They don’t. If you pay for residential or mobile proxies, you pay per gigabyte, and the gigabytes are mostly spent on things you never wanted: hero images, web fonts, analytics scripts, video previews, and the same CSS bundle downloaded ten thousand times. I’ve audited scrapers where less than 3 percent of the paid traffic was the HTML or JSON that actually contained the data.

This article is the process I use to bring that number down. It works for Browser Automation Studio scripts, for plain HTTP parsers, and for anything in between.

The only metric that matters: bytes per record

Forget “pages per hour” for a moment. The number I put on the dashboard of every parsing project is bytes per successful record. Total proxy traffic for a run, divided by the number of records that passed your quality checks.

Why per record and not per page:

  • A page that fails and gets retried three times costs you four times the traffic and produces one record.
  • A listing page that yields 40 records is far cheaper per record than a detail page that yields one.
  • Blocked requests, captchas and redirects to login all burn bytes and produce nothing.

Once you track bytes per record, every optimization becomes measurable. Without it, you are guessing.

How to measure it

You have three options, from easiest to most precise:

  1. Provider dashboard. Run a fixed batch of, say, 500 records on a fresh proxy sub-user and read the traffic counter. Crude but honest, because it is what you get billed for.
  2. Response size logging. In an HTTP client, log the compressed body size plus headers for each request. In a browser, collect the transfer sizes of all network requests per page.
  3. A local recording proxy. Route a test run through a local intercepting proxy and look at every request by host and content type. This is the one that shows you where the waste is.

I always do option 3 once at the start of a project. It takes 20 minutes and it usually tells you exactly what to cut.

Where the bytes actually go

On a typical modern e-commerce or classifieds page opened in a full browser, a breakdown looks something like this (illustrative, but close to what I see regularly):

  • Images and image previews: 40 to 60 percent
  • JavaScript bundles, including third-party: 20 to 30 percent
  • Fonts: 5 to 10 percent
  • Analytics, ads, tag managers, chat widgets: 5 to 15 percent
  • The HTML document or the JSON API response with your data: 1 to 5 percent

A page like that can cost 2 to 5 MB through a browser. The data you need might be 40 KB. At residential proxy prices, that difference is the gap between a profitable parser and one that loses money on every client.

Step 1: Block what you don’t render

If you must use a real browser, the first win is blocking resource types you don’t need. In BAS, the Network module has request mask actions that let you deny URLs by pattern before they ever leave the browser. I set these at the start of every thread, right after the proxy is applied.

My default deny list:

  • Images: *.jpg, *.jpeg, *.png, *.webp, *.gif, *.avif, *.svg
  • Fonts: *.woff, *.woff2, *.ttf, *.otf
  • Media: *.mp4, *.webm, *.m3u8
  • Known third-party hosts: analytics, tag managers, ad networks, session replay, chat widgets

Two warnings from experience:

  • Don’t block blindly on protected sites. Some anti-bot systems load their challenge script from a CDN host that looks like a tracker. Block it, and you get an infinite challenge loop that costs more traffic than the images ever did. Test the deny list against the target before rolling it out.
  • Some sites need certain images. Captcha images, product photos you actually extract, or lazy-load triggers that only fire after an image element loads. Use an allow list for those specific paths instead of turning images back on globally.

This step alone usually cuts browser traffic by 50 to 70 percent.

Step 2: Stop using the browser for data

The bigger win is noticing that the browser is often only needed for one thing: getting a valid session. After that, the data usually comes from a JSON endpoint that the page calls itself.

My pattern:

  1. Open the site in a browser once per session. Solve whatever challenge is there, log in if needed.
  2. Watch the network tab and find the XHR or fetch request that returns the actual data. Search the responses for a product name or price you see on screen.
  3. Copy cookies and the relevant headers from the browser into the HTTP client.
  4. Do all further data collection through the HTTP client, hitting the JSON endpoint directly.

BAS makes this convenient because the browser and the HTTP client live in the same script, so you can hand cookies over without exporting anything. A detail page that cost 3 MB in the browser often becomes a 15 KB JSON response. That’s a 200x reduction per record, and it’s also much faster.

When the session expires or you get a challenge response, you fall back to the browser to refresh it, then go back to HTTP. Treat the browser as a session factory, not as a reader.

Step 3: Make every request smaller

Once you are on plain HTTP, there’s still room to squeeze.

Always request compression

Send Accept-Encoding: gzip, deflate, br. HTML and JSON compress extremely well, often to 10 to 20 percent of their raw size. Some HTTP libraries don’t send this header by default, or send it but you then decompress manually. Check that your client actually advertises it and that you measure compressed size, because that’s what crosses the proxy.

Ask the API for less

Many internal APIs accept parameters the frontend never uses. Look for fields, include, limit, per_page, or GraphQL queries you can trim. Requesting 100 items per page instead of 20 cuts header overhead and request count by 5x. Removing nested objects you don’t store can halve the payload.

Use conditional requests

If you re-scrape the same URLs on a schedule, most of them haven’t changed. If the server returns an ETag or Last-Modified header, store it with the record. On the next run, send If-None-Match or If-Modified-Since. A 304 Not Modified response is a few hundred bytes instead of the full body.

Not every site supports this, but static listing pages, sitemaps and many CDN-backed APIs do. Where it works, it’s the cheapest optimization you’ll ever ship.

Use sitemaps and change feeds for discovery

Crawling category pages to find new items is expensive. Sitemaps with lastmod dates, RSS feeds and “sort by newest” endpoints let you discover only what changed. I diff the sitemap against my database and only fetch URLs that are new or have a newer lastmod.

Step 4: Route traffic by proxy type

Not all requests need the same proxy. Residential traffic is expensive; datacenter traffic is cheap or flat-rate. I split them:

  • Residential or mobile: only for requests that the target actually checks, like the session bootstrap, the protected API, login.
  • Datacenter or direct: public static files you genuinely need, images you extract (often served from a CDN that doesn’t care about your IP), sitemaps on unprotected hosts.

A product image hosted on a public CDN does not need to go through a $5 per GB residential IP. Download the URL list through the protected path, then fetch the images through a cheap one. On image-heavy projects, this single routing decision has saved my clients more than every other optimization combined.

Step 5: Stop paying for failures

Retries are invisible traffic. Every failed attempt that downloaded a full challenge page or a 500 KB error template is billed.

What I do:

  • Detect failure early. Check status code and the first bytes of the response before reading the full body. If it’s a challenge page, abort.
  • Cap retries per URL, and back off the whole proxy pool when the block rate spikes, instead of hammering harder.
  • Track bytes per record per proxy provider and per geo. One bad subnet with a 40 percent block rate can double your cost per record without anyone noticing.
  • Reuse sessions. Every fresh browser session re-downloads all the bootstrap scripts. Keep a session alive for as many requests as the site tolerates.

A real-world shaped example

Here’s the kind of before and after I see when I apply this to a client’s parser, simplified:

  • Before: full browser, nothing blocked, every detail page opened, retries unlimited. Roughly 3.1 MB per record.
  • After request masks: about 1.1 MB per record.
  • After moving detail data to the JSON endpoint: about 40 KB per record, including amortized browser sessions.
  • After compression, bigger page size and conditional requests on re-scrapes: about 9 KB per record on recurring runs.

That’s not a tuning exercise. That’s a different business. A job that cost hundreds of dollars a month in proxies now costs less than a lunch.

The checklist I run on every new parser

  1. Record one session through a local proxy and break traffic down by host and type.
  2. Add bytes per successful record to the run report.
  3. Block images, fonts, media and third-party trackers, then verify the anti-bot flow still passes.
  4. Find the data endpoint and move extraction to the HTTP client.
  5. Confirm compression is on and measured correctly.
  6. Maximize page size, trim fields, add conditional requests.
  7. Route non-protected assets through cheap proxies.
  8. Abort failed responses early and cap retries.

Do this before you scale threads. Scaling a wasteful parser just makes the waste bigger, faster.

FAQ

Will blocking images and scripts get my scraper detected?

It can on heavily protected sites, because some anti-bot checks look at whether resources were loaded or rely on scripts served from CDN hosts. Start with a deny list for images, fonts and obvious trackers, then run a test batch and compare block rates with and without it. If the block rate rises, allow the specific challenge and fingerprinting hosts while keeping the rest blocked.

Is it worth switching from a browser to an HTTP client if the site has strong protection?

Usually yes, using the hybrid approach. Let the browser handle the challenge and login, then pass cookies and headers to the HTTP client for data requests. When the session expires or you hit a challenge response, refresh it in the browser. You keep the protection-passing ability of the browser while paying HTTP-level traffic for most requests.

How do I know my bytes per record target is good enough?

Work backwards from economics. Take your proxy price per GB, your number of records per month, and how much the data is worth to you or your client. If proxy cost is more than 10 to 15 percent of the value of the data, keep optimizing. As a rough reference, a well-tuned HTTP parser on JSON endpoints often lands under 50 KB per record, while an untuned browser scraper can easily be above 2 MB.

Related articles

Done for you

Rather not build it yourself? I will build a BAS bot for your task

Multithreading, proxies, antidetect profiles and scheduled runs. The whole project and every access stay with you.

from $300 · 3 to 7 days

Similar caseiHerb + PochtaGlobal - Business Process AutomationA bot for a reseller of US goods: it collects iHerb orders on its own, creates purchases and parcels on PochtaGlobal, follows tracking numbers and reports every step to Telegram and the accounting system.

"Pavel is a great specialist, he delivered the parser in a few hours. I will keep ordering from him. Huge thanks for the professionalism and the speed."

volodymyrlemchenko · KworkTranslated from Russian