Getting started with a Python SDK for a web scraping API
This guide shows how to connect a Python project to a web scraping API through its software development kit (SDK), install the package, build a first request, and inspect or save what comes back.
It is written as a short task sequence you can follow in order, with a worked screenshot example and a checklist for checking your first result.
What this guide covers
You will cover two things: installing the Python SDK, and creating your first API request. That order matters. Installation gives your project the client class, and the first request proves that your key, target URL, and response handling all work together before you add options such as screenshots, headers, proxies, JavaScript scenarios, or waiting.
1. Install the SDK
Install the package from the command line before importing it in Python. Use the standard pip command shown in the source for a single Python installation. If your machine has several Python versions and the default pip points to the wrong one, use the pip3 variant instead, which the source gives for that situation. Once installation succeeds, you can import the client class in your script.
2. Create your first API request
Every request follows the same three-part pattern. First, import the client class from the SDK. Second, create a client instance and pass your API key to it. Third, call the get method with the URL you want to scrape and a params dictionary for any request options. The call returns a response object that you can inspect, print, parse, or save. The exact client construction and get call are preserved in protected module code0001.
With the documented client installed, set API_KEY and TARGET_URL in your environment before running this request.
import os
from scrapingbee import ScrapingBeeClient
service = ScrapingBeeClient(api_key=os.environ["API_KEY"])
result = service.get(os.environ["TARGET_URL"], params={})
print(result.status_code)
print(result.content)Check the response
The response object exposes the fields you need for a quick sanity check. Its status_code field tells you the HTTP status of the response, and its content field holds the response body. Printing both, as shown in protected module code0002, is the fastest way to confirm that a request succeeded and to see what data came back before you build parsing or storage logic on top of it.
Worked example: save a screenshot
A useful first project is capturing a page as an image. In the worked example, the client is created with the same API key, then get is called on the blog URL with a screenshot option set to true in params. If the response reports success through its ok field, the example writes the response content to a local file named screenshot.png in binary mode. If the request was not successful, it prints the response content instead so you can see the error body. The full example is preserved in protected module code0003.
This self-contained request captures the configured site’s /blog/ page and saves screenshot bytes only when the response is successful. Keep the client credentials outside the script.
import os
from pathlib import Path
from scrapingbee import ScrapingBeeClient
service = ScrapingBeeClient(api_key=os.environ["API_KEY"])
result = service.get("https://finescrape.com/blog/", params={"screenshot": True})
if result.ok:
Path("screenshot.png").write_bytes(result.content)
else:
print(result.status_code, result.content)Add request options
The params dictionary is where you add behaviour beyond a plain fetch. The source names forward_headers, premium_proxy, js_scenario, and wait as examples of additional parameters, and points to the full parameter list with explanations in the documentation. Add one option at a time and re-check status_code and content after each change, so you can tell whether a new option caused a failure.
Field and control map
Use this field and control map when you build or debug a request:
- api_key: passed when the client is created; identifies your account for the request.
- URL argument to get: the page address you want the API to scrape.
- params: a dictionary of request options, such as screenshot set to true for an image capture.
- response.status_code: the HTTP status returned for the request.
- response.content: the response body, which you can print, parse, or write to a file.
- response.ok: a success check to use before saving response content as a file.
First-request checklist
Before you move on, confirm each item:
- The SDK installs without error using pip, or pip3 when you have multiple Python versions.
- Your script imports the client class and creates a client with your API key.
- A get call with a target URL and an empty or minimal params dictionary returns a response.
- You printed status_code and content and confirmed the response looks correct.
- For the screenshot example, you checked ok before writing screenshot.png, and printed content when the request failed.
- You added any extra params one at a time and re-checked the response after each change.