Skip to content

Scraping Yelp Restaurant Data with Python

Yelp is one of the largest crowd-sourced review sites, and its restaurant listings are a practical target for competitor research, neighborhood analysis and lead lists. This guide walks through a two-part Python scraper: one part reads Yelp search results from the JSON endpoint the page already calls, and the other parses individual business pages with BeautifulSoup. You will see exactly which keys and selectors produce each field, where optional data can be missing, and how to keep the scraper running when Yelp starts blocking requests.

Set up the project and libraries

Create a working folder and a single script file. The example uses three Python libraries: re for regular expressions, Requests for downloading pages, and BeautifulSoup for parsing HTML. re ships with Python; install the other two with pip.

The exact shell commands and pip line are preserved in the accompanying code modules. The search URL used throughout targets restaurants in Seattle, WA, but the same pattern works for any city or cuisine you put in find_desc and find_loc.

Command example

pip install beautifulsoup4 requests

Read search results from the JSON endpoint

On Yelp, running a new search leaves the surrounding page in place and only the results area changes. That behavior points to a background data call, and for Yelp the call returns JSON rather than a fresh HTML document. The endpoint is /search/snippet, and requesting it directly yields the same records the browser uses to fill the list.

The endpoint URL has only a few variable parts. The parent_request_id parameter can be dropped and the request still works, so the city and search term are the only values you need to change. The exact URL shape is preserved in the accompanying code module.

Inside the JSON, results live under searchPageProps → mainContentComponentsListProps. That list also contains non-result entries, so filter to items whose searchResultLayoutType equals iaResult. Each matching item exposes the business data under searchResultBusiness.

Field map for the search results

The following table shows which JSON key supplies each output field. Use it as a quick reference when you extend the scraper beyond the five fields covered here.

Worked example: Seattle restaurants

The complete search script loops over the filtered results and prints the name, neighborhoods, review count, rating and a full business URL. Running it against the Seattle restaurant search produced output like this:

  • The Pink Door — Downtown — 6139 reviews — 4.4
  • Biang Biang Noodles — First Hill — 658 reviews — 4.3
  • Tilikum Place Cafe — Belltown — 2167 reviews — 4.3
  • Nue — Capitol Hill — 1269 reviews — 4.2
  • Toulouse Petit Kitchen & Lounge — Lower Queen Anne — 4663 reviews — 4.0

Notice that the businessUrl value is a relative path, so the script prepends https://www.yelp.com to make it absolute. The same JSON also carries images, price ranges and business tags if you need them. The full script and its exact output are preserved in the accompanying code modules.

Parse individual business pages with BeautifulSoup

Business pages do not use the JSON endpoint. Their data is embedded in the HTML response, so BeautifulSoup is the right tool here. The target fields are name, rating, website, phone number and address.

Because Yelp randomizes CSS class names, class-based selectors are unreliable. Target stable structural features instead. The table below maps each output field to its selector; the exact selector strings and code modules are unchanged.

Output fieldSelector
Namethe single h1
Ratinga div whose aria-label contains star rating
Review counta span whose text contains reviews
Websitethe sibling of the p whose text is Business website
Phone numberthe sibling of the p whose text is Phone number
Addressthe sibling of the parent of the a tag whose text is Get Directions

Each of these selectors is preserved exactly in the accompanying code modules, along with the complete combined script and its sample output.

Python example

from bs4 import BeautifulSoup
import requests

html = requests.get("https://www.yelp.com/biz/the-pink-door-seattle-4?osq=Restaurants")
soup = BeautifulSoup(html.text)

name = soup.find('h1').text
print(name)

Missing fields versus blocked requests

Two different failures can look similar in a scraper log, so treat them separately.

A missing optional field is normal. Not every business lists a website or phone number, so soup.find returns None. The example wraps those lookups in if checks so the script skips the field instead of crashing. If a business page has no website, that is a data gap, not a scraping failure.

A blocked request is different. If Yelp decides your traffic is automated, it may return a captcha or an error page instead of the expected HTML or JSON. In that case the selectors will not match and you will see empty or malformed output across many records at once. Setting a User-Agent string alone is usually not enough; rotating proxies and captcha handling are typically required. Whether to build that yourself or evaluate a managed scraping API is a decision for your own project.

When to move to a managed scraping API

If you run the scraper occasionally, the direct Requests approach is fine. If you need scheduled runs across many cities, block rates rise and maintenance grows. A managed scraping API is one option to evaluate. If you choose a provider such as {{OWNER_MANAGED_API_NAME}}, confirm in its own documentation whether it handles proxy rotation and captcha solving and on what billing terms before relying on it. The client workflow for any such service is owner-supplied ({{OWNER_MANAGED_API_CLIENT_WORKFLOW}}) and is not reproduced here; the public contract modules cover the direct Requests and BeautifulSoup path.

Pre-flight checklist

  • Confirm the search URL returns JSON at /search/snippet before writing parsing code.
  • Filter mainContentComponentsListProps to iaResult entries so non-result items do not break the loop.
  • Prepend the Yelp origin to businessUrl because it is stored as a relative path.
  • Wrap optional field lookups (website, phone, address) in if checks.
  • Test selectors against a live page before scheduling runs, since Yelp can change markup.
  • Decide early whether you need a managed API for proxy and captcha handling.
Search result field map
Output fieldJSON key path
Nameresult['searchResultBusiness']['name']
Neighborhoods (list)result['searchResultBusiness']['neighborhoods']
Review countresult['searchResultBusiness']['reviewCount']
Ratingresult['searchResultBusiness']['rating']
Business URLresult['searchResultBusiness']['businessUrl']

Key paths reflect the JSON structure captured at the time of the original walkthrough; Yelp may change its response shape.

Frequently asked questions

Why does the search page use JSON but the business page uses HTML?

Yelp refreshes only the results panel on a new search, which means the results come from a background JSON request to /search/snippet. Business pages are rendered as full HTML, so their data has to be parsed with BeautifulSoup. The two page types need different extraction approaches.

Why does the business URL need the Yelp origin prepended?

The businessUrl value in the JSON is a relative path such as /biz/the-pink-door-seattle-4?osq=Restaurants. Prepending https://www.yelp.com turns it into a full URL you can request directly.

What happens if a business has no website or phone number listed?

The soup.find call returns None, and without a guard the script would raise an error. The example wraps those lookups in if checks so the field is simply skipped when it is absent.

How do I know if I am being blocked rather than hitting a markup change?

A markup change usually breaks one selector; a block affects many records at once and often returns a captcha or error page instead of the expected content. If most records come back empty, suspect blocking first. Rotating proxies and captcha handling are the usual remedies.