Parsing Pages with XPath in BAS

When a bot needs to pull data out of a page, the tool of choice in BAS is XPath. It’s one of the most-used modules in the whole product, because parsing is at the heart of so many bots — and XPath is the fastest way to do it.

What XPath is

XPath is a query language for HTML and XML pages. You write an expression that points at the part of the document you want, and XPath returns it. It’s purpose-built for parsing: instead of hunting through raw HTML, you address elements directly and pull their contents.

The actions

• XPath get HTML — return the HTML of the matched element.
• XPath get each HTML — return the HTML of every matched element, for looping over a set.
• XPath get text — return the text content of the matched element.
• XPath get each text — return the text of every match, ideal for a list of rows.
• XPath get attribute — read an attribute such as href or src.
• XPath check existence — test whether a match exists, so you can branch when a field is missing.

A concrete example

Building a classifieds parser — an Avito-style scraper — is the classic case. You open the listings in a browser, then for each product card you use XPath to pull the characteristics, description and photo. XPath get each text walks the cards, XPath get attribute grabs image URLs, and XPath check existence keeps the flow alive when a card is missing a field.

XPath in the parsing toolkit

XPath is for rendered HTML/XML — the page as the browser sees it. It pairs naturally with the rest of the data toolkit: locate and click elements with selectors, parse API responses with JSON, and clean the extracted text with string operations. For most scraping jobs, though, XPath is where the real extraction happens — learn it well and parsing stops being the hard part.