Skip to content

How to parse dynamic CSS classes when web scraping

Class names that change between page loads break selectors that depend on them.

OUTPUT FORMATSHTMLJSONMarkdown

A more durable approach is to anchor on stable visible text and then move to the element that holds the value. This guide shows a text-based XPath pattern, a worked stock-count example, and a short checklist for deciding when the pattern is safe to use.

Why class-based selectors fail on dynamic markup

When a site generates class names per render or per session, a selector such as div.la may work once and then miss. The class attribute is still present, but its value is not a reliable identity. Text labels and structural relationships are often more stable: a label like "Remaining Stock" tends to persist even when the class around it changes. The pattern below uses that label as the anchor and then selects the next sibling element that carries the value.

A worked example: stock count next to a label

Consider a product block where the label and the value sit in adjacent div elements. The class names in this example are short and arbitrary, which is exactly the situation where a class selector is fragile. The label text is the dependable part.

The XPath pattern

The selector below finds the div whose text is exactly "Remaining Stock" and then takes the first following sibling div. That sibling holds the count. The pattern is intentionally narrow: it assumes the value is the immediate next sibling and that the label text is exact.

Indexing note

In Selenium, XPath indexing starts at 1, which is why the selector uses div[1] rather than div[0]. If you move the same expression to a tool with different indexing conventions, check that behavior before assuming the selector will return the same node.

Decision checklist before using a text-anchor selector

Use this checklist to decide whether the pattern fits the page you are scraping.

  • The label text is stable across renders and locales you care about.
  • The value is in the immediate next sibling, not nested deeper or placed before the label.
  • The label appears once in the relevant scope; if it repeats, add a parent scope or a more specific path.
  • Whitespace around the label is predictable, because text() matching is exact.
  • Your tool's XPath index base is known (Selenium uses 1).
  • You have a fallback if the sibling is missing, such as returning null rather than failing the whole run.

Field and control map for the sample block

The sample HTML contains four class-bearing elements. The table below maps each one to its role and whether the selector depends on it.

Failure distinctions

Three failures look similar but need different fixes. First, the label text is present but no following sibling exists: the value may be nested inside the label element or placed before it, so the path needs restructuring. Second, the label text is absent or differs by whitespace or case: the anchor itself is wrong, and a contains() or normalized comparison may be needed. Third, the sibling exists but the index is off: in Selenium the first sibling is div[1], so div[0] returns nothing. Separating these cases prevents chasing the wrong part of the selector.

Sample block field and control map
ElementClass in sampleRoleUsed by selector
div (outer)ddContainer for the product blockNo
h1aaProduct headingNo
div (inner)ffaGroups label and valueNo
div (label)laHolds the text "Remaining Stock"Yes — anchor
div (value)adHolds the stock countYes — following sibling

Class values shown are from the illustrative sample only and are not stable identifiers.

Frequently asked questions

Can I use contains() instead of exact text matching?

Yes, when the label may include extra whitespace or surrounding text. Exact text() matching is simpler but brittle if the label changes. contains() is more forgiving but can match multiple elements, so scope it to a parent or add a position filter.

What if the value is not the immediate next sibling?

Adjust the path. You can select a later sibling with following-sibling::div[2], or move up to a common parent and then down to the value element. The right choice depends on the actual structure, not on the class names.

Does this pattern work outside Selenium?

The XPath expression itself is standard, but indexing conventions and text normalization can differ between tools. Verify the index base and whitespace handling in your environment before reusing the selector.