No-Code Competitor Monitoring: Build a Scrape-and-Compare Workflow
Competitor monitoring is repetitive by nature: load a page, find the same element, compare it with what you saw last time, and decide whether anyone needs to act. That loop is a good fit for automation, and it does not require a custom scraper. This guide shows how to combine a no-code scenario builder with a scraping API so that a stored list of competitor URLs drives the whole process. The example tracks a product documentation page, but the same structure works for prices, stock status, feature lists, ratings, reviews, and specification changes.
Why pair a scenario builder with a scraping API
A scenario builder connects modules: one module reads records, another sends each record to a scraping service, a filter checks the result, and a final module performs an action. The scraping API handles the part that is hardest to do reliably by hand: fetching the target page and returning only the elements you asked for. In the scenario, you fill in form fields rather than writing a program. The URL and the extraction instructions can come from stored variables, so adding a competitor means adding a row, not editing the scenario. The extraction rules are the part worth learning carefully, because they determine whether you get a clean value or a noisy one.
The four-step monitoring loop
Every run follows the same sequence. First, load the current records: the pages to check, the selector for each page, and the last value you stored. Second, scrape each page using those instructions. Third, compare the fresh result with the stored value. Fourth, run an action only when the comparison shows a difference. In a scenario builder, the first step is usually a data store search followed by an iterator, so each record is processed separately. The comparison step is a filter placed on the connection between modules. The action step can be anything the builder supports: create a task, append a spreadsheet row, send a message, update a document, or open a link on a phone.
What to store for each competitor page
A minimal record has three fields. URL holds the competitor page link. Selector holds the extraction instructions sent to the scraping API. Current Value holds the last result, which becomes the baseline for the next comparison. You can add competitor name, date changed, or an active/inactive flag later; the three-field version is enough to run the loop. In the scenario builder, create a data structure with those fields, then create a data store table using that structure. The first module searches the table; filtering on URL exists returns every record. A later refinement is to filter on an active flag so only selected competitors are checked.
Choose the right extraction shape before writing a selector
The extraction rule format is a JSON object that maps a key name to a selector, as shown in the extract-rule syntax reference. The shape you choose depends on what you need to detect. Use a single-element selector when one visible value changes meaningfully, such as a last-modified date. Use an attribute selector when the value lives in an attribute rather than between tags, such as an image src or a meta content field. Use a list selector when you need a whole table row or a set of repeated items. Use a whole-body text selector when you only need to know that something changed and do not need to identify what. The checklist below summarizes the trade-offs.
Selector choice checklist
Ask these questions in order. Does the value sit between an opening and closing tag? A plain CSS selector returns the tag contents with HTML stripped. Is the value stored in an attribute, such as src or content? Add an output argument beginning with @ followed by the attribute name. Do you need every match rather than the first one? Switch the type from item to list. Do you only need a change signal and not the changed value? Return the body as text instead of crafting a selector. If you return full HTML, expect false positives from CDN cache strings and script URLs that change without any meaningful content change.
Turning a copied XPath into a CSS selector
Browser developer tools usually offer Copy > XPath rather than a reliable CSS path. The XPath is close to a CSS selector but needs two substitutions. Replace each slash with a greater-than sign surrounded by spaces. Replace each bracketed position such as [2] with :nth-of-type(2), keeping a space after it and no space before it. The XPath /html/body/h3[2]/em therefore becomes html > body > h3:nth-of-type(2) > em. Spacing matters: inserting a space before :nth-of-type changes which element is selected. A converter tool can perform the same substitution, but checking the result against the two rules is faster for simple paths.
Worked example: a last-modified value and a table row
On the demo page, the Last Modified text sits inside an em element under the second h3. The converted selector is html > body > h3:nth-of-type(2) > em. Wrapped in an extract rule under the key date, the API returns a JSON object with that key and the date string as its value. The demo response shows a placeholder date, which is enough to confirm the selector works before you store it. For a repeated table, copy the XPath for one cell, convert it, then remove the final :nth-of-type segment so the selector matches the whole row. A selector ending at td with type list returns an array of cell values for the 1920 model row, as the table-row response reference shows. Test the rule in the scraping API first, then paste it into the data store; do not hard-code it in the scenario module.
Attributes, meta tags, and hidden values
Self-closing tags such as img have no inner text, so a plain selector returns nothing useful. Supply an output argument instead: @src for an image URL, or @content for a meta tag. A meta tag may not be visible on the page, so open developer tools from any element, scroll to the head section, and copy the XPath for the meta element you need. The same technique reaches values hidden by CSS: if the information is present in the markup, its XPath can be copied and scraped. This is useful for modified-time metadata that pages expose for crawlers but do not display to readers.
Compare, then act only on a real difference
Place a filter on the connection after the scraping module. The condition is that the stored value does not match the live result. If the values match, the run stops for that record. If they differ, the next module runs and the new value is written back to the data store so the next run compares against the latest baseline. For multiple extracted fields, insert a JSON parser after the scraping module before comparing. A router can send different cases down different paths, for example a high-profile competitor versus a routine one, or a changed page versus an unchanged one. Typical actions include creating a review task, appending a row with URL, change, and date, emailing a colleague, updating internal documentation, or opening the competitor URL on a phone.
Two failures that look similar but need different fixes
A 400 error from the scraping API usually means the selector string was malformed, often because of stray spaces or tabs inside the JSON. Re-send the rule with no extra whitespace and the request should proceed. A false positive, by contrast, is a successful scrape that reports a change when nothing meaningful changed. This commonly happens when you compare full HTML: CDN cache strings and script URLs vary between requests, so the page looks different even when the visible content is identical. The fix is to narrow the selector to the element that matters, or to return the body as text rather than raw HTML. Treating a 400 as a content change, or a cache-driven difference as a real update, will both waste review time.
What the finished workflow gives you
Once the scenario is scheduled, it checks every stored page on a recurring basis, compares each result with the last stored value, and acts only when something differs. Adding a competitor means adding a record with a URL and a tested selector. Changing what you watch means editing that selector, not rebuilding the scenario. The same structure covers prices, stock, features, ratings, comments, and specification changes; the selector and the action are the only parts that change.
Frequently asked questions
Should I scrape one element or the whole page body?
Scrape one element when you need to know what changed and want a clean value to compare. Scrape the body as text when you only need a change signal and do not want to maintain a selector. Avoid comparing raw HTML, because CDN cache strings and script URLs create differences that are not content changes.
Why does the scraping request return a 400 error?
The most common cause is whitespace inside the selector string. Send the extract rule with no extra spaces or tabs and retry. A 400 is a request-format problem, not evidence that the page content changed.
When should the extraction type be list instead of item?
Item returns only the first matching element and is the default. List returns every match, which is what you want for a table row, a set of product cards, or any repeated block. To broaden a copied selector, remove the final :nth-of-type or :nth-child segment before switching the type to list.
How do I capture an image URL or a meta tag value?
Use an output argument beginning with @ followed by the attribute name, such as @src for an image or @content for a meta tag. For meta tags that are not visible on the page, open developer tools, scroll to the head section, and copy the XPath for the meta element.
Why must the new value be written back after a change?
The stored value is the baseline for the next comparison. If you do not update it after a detected change, the following run will compare against the old value and report the same change again. Writing the fresh result back keeps each run's comparison meaningful.
{ "key_name" : "css_selector" }
{
"date":"html > body > h3:nth-of-type(2) > em"
}
{
"title": "01/01/1920"
}
{
"image":"html img"
}
{
"image" : {
"selector": "html img",
"output": "text",
"type": "item"
}
}
{
"image" : {
"selector": "html > body > img",
"output": "@src",
}
}
{
"meta" : {
"selector": "html > head > meta:nth-of-type(2)",
"output": "@content",
}
}
{
"1920_model" : {
"selector": "html > body > table > tbody > tr:nth-of-type(2) > td",
"type": "list",
}
}
{
"1920_model": [
"1920",
"800",
"600"
]
}
{
"content": "A short example update."
}