Skip to content

Scrapy Cloud Deployment Guide: Deploy, Schedule and Monitor Spiders

Scrapy Cloud is Zyte's managed hosting service for Scrapy projects. Instead of keeping a crawler on a laptop or a hand-built server, you upload the project and let the platform run each job in an isolated container with its own resource allocation. This guide walks through the deploy workflow end to end, then covers scheduling, monitoring, exports and the operational choices that separate a demo from a spider you can leave running.

What Scrapy Cloud does, and when it fits

Scrapy Cloud sits between your spider code and the infrastructure it needs. You keep writing standard Scrapy spiders; the platform supplies containers, job queuing, a dashboard, scheduling and log storage. Each unit is described as 1GB RAM with 2.5GB disk space, and jobs run in isolation from one another.

The value is clearest when more than one person or project is involved. Locally, every developer has to match dependency versions and coordinate who runs what. In the cloud, deployment is a single command and the dashboard becomes the shared place to launch, schedule and inspect runs.

It is a weaker fit if you only need occasional one-off crawls, or if you want full control of the host operating system. In those cases the open-source framework on your own machine or server may be simpler.

Local deployment vs managed hosting: a decision checklist

Use this checklist before migrating an existing spider. Each row describes a responsibility that shifts when you move to managed hosting.

  • Uptime: locally, a sleeping laptop or dropped connection stops the crawl; managed hosting runs independently of your machine.
  • Scheduling: cron or a task scheduler works, but multiple spiders on different cadences get hard to track; the dashboard offers a visual scheduler with timezone handling.
  • Monitoring: local runs need logging and alerting built by hand; the platform shows job status, runtime stats and error reports, and can notify you when something fails.
  • Scaling: adding concurrency locally means provisioning servers and load balancing; on the platform you adjust the number of units.
  • Team access: local setups require matching environments; the dashboard lets team members deploy and monitor regardless of their own setup.

If most of those rows are currently your responsibility and causing friction, managed hosting is worth evaluating. If only one row matters, a smaller fix may be enough.

Setup in five steps

The workflow is short: create an account and project, install and authenticate the command-line client, prepare the spider, deploy, then run from the dashboard.

  1. Create a Zyte account and a project. Registration on the free tier does not require a credit card. A project is a container for related spiders, and it receives a unique identifier you will use when deploying.
  2. Install the shub client with pip (see the command module below), then open your profile in the dashboard, copy an API key and run shub login, pasting the key when prompted. Authentication persists per machine, so each team member should use their own key.
  3. Prepare a spider. The walkthrough uses the booksbot demo repository and a small spider that reads titles, prices and ratings from books.toscrape.com and follows pagination links.
  4. Deploy with shub deploy [project_id], taking the project ID from the dashboard URL. The upload usually takes under a minute and creates a snapshot, so local edits do not affect the deployed version until you deploy again.
  5. Open the project in the dashboard, select the spider and press Run. You can pass arguments, set priority and choose resources before starting; logs stream while the job runs.

A useful field map for the deploy step: the project ID identifies the destination, the API key authorizes the upload, and each shub deploy creates a new versioned snapshot rather than editing the previous one in place.

Scheduling, priority and monitoring

Periodic jobs are configured in the dashboard. You can run a spider hourly, daily, weekly or on a custom interval, and each schedule can carry its own arguments, priority and resource allocation. That makes it practical to give a full-site crawl more units than a frequent small update check. The scheduler is timezone-aware and handles daylight saving transitions.

Priority levels run from lowest to highest, and the queue processes higher-priority jobs first when capacity frees up. The practical use is to keep a long bulk crawl from delaying a time-sensitive refresh.

Logs are viewable in real time with search and filtering, and runtime statistics cover memory use, request rates, response times and success ratios. Historical metrics are useful for spotting slowdowns and planning capacity. Failed periodic jobs can retry automatically, with limits, backoff and notification thresholds you configure.

Exporting results

Completed jobs offer a download button in the dashboard. Items can be exported as CSV, JSON or XML, and large result sets can be downloaded compressed. Encoding is handled for you and formatting stays consistent across data types. For pipelines that need proxy rotation or anti-bot handling alongside extraction, a separate scraping API can sit in front of the same downstream processing.

Making a spider production-ready

Three areas decide whether a deployed spider keeps running without intervention: error handling, proxy configuration and scaling design.

Proxies. Zyte Smart Proxy Manager can be wired into spider settings for IP rotation and geo-targeting with minimal code changes; it handles proxy selection, rotation timing and failure recovery. External proxy services work through Scrapy's standard proxy settings, and the container supports username/password and IP whitelisting. Test the configuration on the free tier before scaling.

Monitoring. Spidermon adds automated alerting: notify when error rates pass a threshold, data quality drops or a job fails outright. Validation rules can check items for completeness and accuracy, and can pause a spider when quality issues appear so bad records do not reach downstream systems.

Scaling. Prefer running several spider instances over trying to make one spider faster. Scrapy's duplicate filtering lets concurrent instances cover different parts of a large site without overlapping extraction.

Deployment pitfalls. Dependency mismatches are the most common failure; pin exact versions in requirements.txt and test in the free tier first, since the container may differ from your local environment. Keep scrapy.cfg current and make sure custom modules are inside the project directory — imports that resolve locally can fail in the cloud because of different Python path settings.

Three deployment failures and how to tell them apart

Not every failed deploy has the same cause, and the fix differs:

  • Package version mismatch: the job starts but errors on import or behaves differently than locally. Pin versions in requirements.txt and redeploy.
  • Missing module: an import that works on your machine fails in the container. Confirm the file sits inside the project directory that gets uploaded.
  • Stale project configuration: the deploy targets the wrong project or misses settings because scrapy.cfg is out of date. Check the project ID against the dashboard URL and update the config.

Testing each change on the free tier before promoting it to a paid plan keeps these failures cheap to find.

Free plan limits and paid tiers (dated reference)

The figures below reflect the captured reference and should be checked against current Zyte terms before you commit.

Free tier: unlimited projects and team members, but a one-hour limit per job, seven-day data retention, one concurrent job and no priority queue management. That is enough to validate that a spider deploys and runs correctly.

Professional tier: from $9 per unit per month, with each unit providing 1GB RAM and 2.5GB disk space. It adds unlimited crawl time, 120-day retention, concurrent jobs and priority queues. Units can be added or removed as demand changes.

Enterprise: custom pricing with dedicated resources, SLA guarantees and premium support, negotiated per customer.

A pay-as-you-go structure means capacity can follow demand rather than a fixed annual commitment.

Frequently asked questions

Is Scrapy Cloud suitable for someone new to web scraping?

It is workable if you already know basic Scrapy. The deployment steps are simple, but writing an effective spider still requires Python and Scrapy knowledge.

How does Scrapy Cloud differ from running spiders locally?

The managed platform supplies infrastructure, scheduling, monitoring and shared team access. Local runs leave server upkeep, scheduling and alerting to you.

What does the free plan restrict?

Jobs are limited to one hour of runtime, data is retained for seven days, only one job runs at a time and priority queues are unavailable. Paid plans lift those limits.

Can spiders run on a schedule automatically?

Yes. Periodic jobs are configured in the dashboard with timezone support, retry logic and failure notifications.

What should I focus on to make a spider production-ready?

Error handling, proxy configuration, monitoring and horizontal scaling. Spidermon covers automated alerts, and spiders should be designed to run concurrently.