Python Requests Timeouts: Set Connect/Read Limits, Catch Errors, Retry Wisely
A Python script that hangs on a network call is usually missing one thing: an explicit timeout. Requests does not impose one for you, so a stalled server, a slow proxy, or a dropped route can leave a worker waiting far longer than intended. This guide shows how to set connect and read limits, tell the two failure stages apart, catch the right exceptions, bake defaults into a session, and add retries that back off instead of hammering the host.
What a timeout actually controls
A timeout is the client's instruction for how long to wait before giving up. In Requests, the value you pass is not a single wall-clock stopwatch for the whole download. It is split by stage, and understanding that split is the difference between a fast-fail connect and a patient read.
When you pass one number, such as timeout=5, Requests applies it to both stages: up to five seconds to establish the connection, and up to five seconds waiting for data. The read portion covers the first byte and the gaps between chunks, so a slow-dripping response can still trip it even after the connection succeeds.
A tuple gives you separate control. timeout=(2, 15) means two seconds to connect and fifteen seconds of read patience. That is the recommended shape for production because an unreachable host should fail quickly while a busy server gets room to start responding.
The default is no timeout
Requests does not ship a hidden five-second or thirty-second safety net. If you call requests.get(url) without timeout=..., the call can wait until the underlying socket gives up, which depends on the OS, DNS resolver, proxy setup, and network stack. In a worker pool that means one stuck request can pin a thread, jam a connection pool, and slow the whole service. In a scraping loop it can freeze an entire batch.
Treat explicit timeouts as part of your standard, not an optional tweak. Predictable failure is easier to debug than a job that is still running at 3 AM with no output.
Connect timeout vs read timeout
The connect timeout covers opening the socket. If the host or proxy is unreachable, this is where you want a fast fail. Two practical caveats: the connect timeout can apply per IP attempt, so a hostname resolving to both IPv4 and IPv6 can feel slower than the number suggests; and slow DNS resolution is not cleanly governed by the Requests timeout in the way many people assume.
The read timeout starts after the connection is established. It measures the gap between bytes coming back, which in most cases feels like time-to-first-byte. Requests does not enforce a total-download timer; it times idle gaps between chunks. That is why a large but steadily streaming response can finish even with a modest read timeout, while a server that connects and then goes silent will trip it.
A short connect value (around three seconds is a reasonable starting point for internet-facing calls) plus a larger read value handles the common case: fail fast on a dead route, stay patient with a server that is thinking.
Choosing values: a starting checklist
These ranges are starting points, not rules. Tune them against logs.
- Fast internal APIs: 1–3 seconds. If your own infrastructure is slow, you want to know quickly.
- Public REST APIs: 5–10 seconds. The public internet adds variance.
- Heavy or batch-style endpoints: 10–20 seconds. Some services legitimately need time before the first byte.
- Slow proxies or JS-heavy pages: give the read side more room, but cap it.
If you hit timeouts constantly, the endpoint may be slow by design or the site may be pushing back. If you never hit them, your workers may be stalling longer than necessary. Either signal is worth a look at the logs.
Catching timeout exceptions cleanly
Requests raises specific exceptions rather than a generic stack trace. requests.exceptions.ConnectTimeout means the connection stage failed. requests.exceptions.ReadTimeout means the connection succeeded but the server went silent too long. Both sit under requests.exceptions.Timeout, and broader network failures sit under requests.exceptions.RequestException.
Wrap the call in try/except, catch the specific variants first, then the umbrella, then the broader request exception. From there you can log, retry, skip the item, or bail out gracefully. The goal is that one slow endpoint does not drag the whole run down.
A practical pattern: catch ConnectTimeout and ReadTimeout separately so your logs tell you which stage failed, then fall back to Timeout and RequestException for anything else. That distinction is what makes later tuning possible.
Centralizing defaults with a session
Passing timeout=... into every call gets old fast. A requests.Session() gives you connection reuse and cleaner code, but it does not apply a timeout automatically. The grown-up move is a small subclass that bakes in a default timeout and mounts an HTTPAdapter with urllib3.Retry for transient failures.
A few rules for that session:
- Keep retries on idempotent methods (GET, HEAD, OPTIONS). Retrying POST or PUT can double-create or double-charge unless the API supports idempotency keys.
- Decide which HTTP codes count as temporary in status_forcelist; 429, 500, 502, 503, and 504 are common choices.
- Respect Retry-After when the server sends it.
- Passing timeout=None disables timeouts entirely, so do that only when you mean it.
Timeouts stop hangs; retries handle transient flakiness. You usually want both, configured in one place so the codebase cannot forget them.
Backoff, jitter, and retry budgets
Instant retries against a failing server are not robustness; they are a synchronized stampede. Exponential backoff gives the system room to recover, and jitter spreads retry timings so workers do not all hit the host at once.
The usual shape is a base delay multiplied by 2 raised to the retry number, plus a small random offset. Bound the attempts so a scraper cannot drift into a multi-minute retry coma.
A retry budget is the decision you make up front about how much time or how many attempts you will spend before moving on. Two budgets matter: an attempts budget (for example, three tries then bail) and a time budget (for example, twelve seconds total including backoff). Without a budget, one stubborn endpoint can stall a pipeline, backoff can balloon into long delays, workers pile up, and paid proxies or APIs cost real money on every retry.
What to retry and what not to retry
Not every failure deserves another attempt. The rule of thumb: retry what is unstable, do not retry what is intentional. If the server is saying "I can't," retry. If it is saying "I won't," stop.
Generally safe to retry:
- Timeouts and connection errors (slow server, DNS stumble, dropped socket).
- 429 rate limits, where the server is asking you to slow down.
- 503 temporary load issues, where backoff usually helps.
- Flaky proxies, which are normal in scraping.
Generally not worth retrying:
- 401 and 403, which usually mean you are not allowed. If your IP is blocked, retrying the same IP is pointless; switching proxies is a different strategy, not a retry.
- 404, where the page is not there.
- Consistent 500s on static assets, where the file is broken.
- Business-logic responses such as "out of stock" or "invalid input."
Be careful with non-idempotent writes. Retrying GET and HEAD is usually fine. Retrying POST, PUT, or PATCH can double-charge or double-create if the first attempt reached the server but your client timed out. If you must retry writes, use an idempotency key when the API supports it, or make the operation idempotent server-side.
Smart retries for scraping
Plain retries repeat the same request and hope for a different outcome. Smart retries change something between attempts:
- User-agent, because some sites punish repeated client signatures.
- Proxy, because switching the exit IP often fixes what looked like a timeout or a block.
- Headers or cookies, because small changes can slip past basic filters or caching layers.
- Timeouts, with a small read bump for heavy pages, but not endless inflation.
- Fetch strategy, starting with a normal GET and moving to JS rendering or a full browser mode if it keeps failing.
That is the retry logic that adapts to what the site is actually doing. Static, identical retries keep dying on pages where an evolving attempt survives.
When the website is the cause, not your code
Sometimes a timeout is not a code weakness. Bot filters delay or drop responses, IP blocking kicks in after a few hits, CAPTCHAs freeze the response until a human solves them, geo-based throttling slows certain regions, heavy JavaScript pages do not produce HTML until a browser executes code, and giant content loads choke slow connections.
When those are in play, you will see repeated timeout failures even with sane values. Bigger timeouts and more retries usually will not help, because the site wants a real browser, clean IP rotation, or full JS rendering. That is the point where a managed scraping layer absorbs the blocking, rendering, and IP pool work so your scraper stays simple.
Worked example: a bounded fetch with backoff
Suppose you are fetching a page that sometimes stalls behind a proxy. A reasonable shape is: a connect/read tuple of (2, 8), three attempts, exponential backoff with jitter, and a small read bump only after a timeout.
On attempt one, the client gets two seconds to connect and eight seconds of read patience. If it times out, wait roughly one second plus jitter, then retry with a slightly larger read value. If it times out again, wait roughly two seconds plus jitter. After the third failure, log the URL as failed and move on. The exact numbers are yours to tune; the structure is what matters: explicit limits, bounded attempts, and a clear exit.
If you use a retry library such as tenacity, the same shape applies: Requests handles the timeout, the library handles the retry policy. Add a result-based retry for 429 or 503 when you want to honor Retry-After, and cap the wait so a server cannot ask you to sleep for an hour.
Requests vs a managed scraping layer
Plain Requests is the right tool when the target is simple: internal dashboards, public sites with no JavaScript and no blocking, and fast API-like endpoints. You control the timeout, the retries, and the infrastructure, and there is nothing extra to pay for.
A managed layer earns its place when the site is defensive: CAPTCHAs, IP bans, slow JavaScript, inconsistent latency, or regional throttling. At that point you are juggling proxy lists, rotating user agents, retry loops, cookie jars, and constant timeout tweaks, and a server that deliberately delays you turns your timeout strategy into guesswork. A managed service handles proxy rotation, JS rendering, anti-bot countermeasures, geo routing, and built-in retries behind the same Python call shape.
The two work together. You do not have to rewrite the scraper to benefit from the managed layer; you route the hard pages through it and keep the simple ones on plain Requests.
| Control | What it governs | Typical starting point | Failure signal |
|---|---|---|---|
| timeout=5 (single number) | Both connect and read stages | Simple API calls | ConnectTimeout or ReadTimeout |
| timeout=(2, 8) (tuple) | Connect separately from read | Production default | Specific exception type tells you which stage |
| Connect value | Opening the socket, per IP attempt | ~3 seconds internet-facing | requests.exceptions.ConnectTimeout |
| Read value | Gap between bytes after connect | 5–10s public, 10–20s heavy | requests.exceptions.ReadTimeout |
| Attempts budget | Maximum retries | 3 tries then bail | Logged failure, item skipped |
| Time budget | Total wall time including backoff | e.g. 12 seconds | Job moves on instead of stalling |
Starting points only; tune against logs.
| Outcome | Retry? | Why |
|---|---|---|
| Timeout | Yes | Server slow or network hiccup |
| Connection error | Yes | DNS stumble, dropped socket |
| 429 | Yes | Server asks you to slow down |
| 503 | Yes | Temporary load; backoff helps |
| Flaky proxy | Yes | Normal in scraping |
| 401 / 403 | No (switch proxy instead) | Not allowed; same IP retry is pointless |
| 404 | No | Page is not there |
| Consistent 500 on static asset | No | File is broken |
| Business-logic response | No | Not an error, it is reality |
Idempotency matters: retrying POST/PUT/PATCH can double-create or double-charge.
Frequently asked questions
How do I set a timeout for requests.get in Python?
Supply either a single number or a two-value tuple: requests.get(url, timeout=5) or requests.get(url, timeout=(2, 10)). A lone number is applied to both the connect and read stages, while a tuple assigns each stage its own limit.
What is the difference between connect timeout and read timeout in Python Requests?
Connect timeout is how long the client waits to open the connection to the server or proxy. Read timeout is how long it waits between bytes while receiving the response, including the first byte and any mid-download stalls.
Does Python Requests have a default timeout if I don't set one?
No. Without an explicit timeout, the call can wait indefinitely.
How should I handle a Python Requests timeout exception without crashing my script?
Wrap the request in try/except and catch requests.exceptions.Timeout, plus the specific ConnectTimeout and ReadTimeout variants. Log it, retry if needed, or skip the item without stopping the whole script.
What are good timeout values for Python web scraping and external APIs?
Fast internal APIs usually work with 2–3 seconds, public APIs with 5–10 seconds, and heavier or slow pages with around 10–20 seconds. Treat these as starting points and tune against logs.
When should I switch from plain Python Requests to a managed scraping layer?
When pages rely on JavaScript, use bot protection, throttle regions, or block your IPs. A managed layer handles rendering, proxies, and retries, which reduces timeout failures that timeout settings alone cannot fix.