Using the HTTP Client in BAS

BAS has two ways to talk to a site: a real browser, and the HTTP-client — direct requests with no rendering. The HTTP-client is how you scale, and the module gives you everything a normal HTTP request needs: methods, headers, cookies, proxies and downloads.

What the HTTP-client is for

The HTTP-client sends requests straight to the web server without opening a page in a browser and without drawing any visuals. You get back the raw response — HTML, JSON, a file — and nothing else. Because there’s no rendering engine eating RAM and CPU, the same hardware that handles 100–200 browser threads can run a thousand or more HTTP-client threads. It’s the right tool whenever the data you need is already in the response and throughput matters.

The core request actions

GET request. Performs requests with the GET, HEAD, DELETE, TRACE and OPTIONS methods. Use it to pull a page’s HTML directly from the web server, or to fetch data without a browser. An extra field lets you specify the headers the server will see on send.

POST request. Performs requests with the POST, PUT and PATCH methods — used to send changes to a site (for example, run a product search or submit data) and read the result. Unlike GET, a POST carries a Post Data field (the parameters you send), a content type (application/x-www-form-urlencoded, multipart/form-data, or application/json) and a request encoding. It too has a field for the request headers.

Download. Downloads a file by the URL in the request — an image, a captcha, a video or an audio file. An additional parameter sets the path where the file is saved.

Controlling the request

Set header. Adds or sets a request header the site and server will see — essential for requests that expect specific headers (auth tokens, content negotiation, a realistic user-agent).

HTTP-client proxy. Routes the client’s requests through a proxy. It’s the equivalent of setting a proxy in the browser, but it applies only to the HTTP-client.

Load cookie. Loads cookies into the client so authenticated requests carry the right session.

Reset. Clears all headers and cookies back to a zero state — a full reset of the client’s settings. Use it between identities so one request’s headers and cookies don’t leak into the next.

Reading the result

Response content. Returns the body of the response so you can parse it — feed it into JSON parsing, XPath or regular expressions to extract what you need.

Error check. Tells you whether the request failed, so your flow can branch and retry instead of silently continuing on a bad response.

A common pattern

The HTTP-client shines in combination with the browser, not instead of it. A typical scaled bot logs in with a real browser (because that step needs one), exports the session cookies, then fans out across thousands of HTTP-client threads — loading cookies, sending GET/POST requests through rotating proxies, and parsing the raw responses. Browser for the parts that require a browser; HTTP-client for the heavy lifting that doesn’t.