The urllib.error module contains the exceptions used by urllib.request when a network operation fails. Distinguishing an HTTP error from DNS failure, timeout, certificate verification, connection refusal, or an incomplete download helps you build reliable clients, useful logs, and retry policies that do not amplify incidents.
This guide focuses on exception handling in the standard-library client. To create requests, headers, proxy handlers, and bounded downloads, read Python urllib.request. To split and validate URL components, see Python urllib.parse.
The basic hierarchy
URLError is the base exception and inherits from OSError. HTTPError inherits from URLError. Consequently, the order of your except blocks matters: catch HTTPError before URLError, otherwise the broader handler hides the specific one.
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
try:
with urlopen("https://example.com/resource", timeout=10) as response:
data = response.read(100_000)
except HTTPError as error:
print("HTTP status:", error.code)
except URLError as error:
print("Transport failure:", error.reason)
A 404 or 500 status means that a valid HTTP response arrived, even though it reports failure. A plain URLError can represent name resolution, connection refusal, timeout, TLS, or another step before an HTTP response exists.
Understanding URLError
The reason attribute may be a string or another exception. Avoid making operational decisions by comparing message text alone. Inspect the underlying type when the distinction matters.
import socket
from urllib.error import URLError
from urllib.request import urlopen
try:
urlopen("https://example.invalid", timeout=5)
except URLError as error:
if isinstance(error.reason, socket.timeout):
print("The connection timed out")
else:
print(type(error.reason).__name__, error.reason)
Depending on the platform and call path, a timeout may appear as TimeoutError, socket.timeout, or a nested exception. Tests should model the deployment environment instead of turning unstable error strings into a strict API.
HTTPError is both exception and response
An HTTPError exposes url, code, reason, headers, and a file-like object in fp. The error itself can also be read like the response. This is useful when a server sends a JSON or text explanation.
import json
from urllib.error import HTTPError
from urllib.request import Request, urlopen
request = Request("https://api.example.com/items/999")
try:
with urlopen(request, timeout=10) as response:
payload = json.load(response)
except HTTPError as error:
body = error.read(64_000)
media_type = error.headers.get_content_type()
if media_type == "application/json":
detail = json.loads(body.decode("utf-8"))
else:
detail = body.decode("utf-8", errors="replace")
print(error.code, detail)
Always bound the error body. An untrusted remote server can return megabytes or keep the connection busy. An error status does not make the body safe for logs or HTML output.
Classify status codes
A practical policy often groups responses:
- 400–499 usually indicate a request, authentication, authorization, or resource issue.
- 500–599 often represent a temporary or internal server failure.
- 429 requests lower traffic and may include
Retry-After. - 401 and 403 should not trigger endless retries with the same credentials.
Do not retry every status automatically. Repeating a non-idempotent POST can duplicate orders, charges, messages, or database records.
Selective retries with backoff
import time
from urllib.error import HTTPError, URLError
from urllib.request import urlopen
RETRYABLE = {429, 500, 502, 503, 504}
def download(url: str, attempts: int = 3) -> bytes:
for index in range(attempts):
try:
with urlopen(url, timeout=10) as response:
return response.read(1_000_000)
except HTTPError as error:
if error.code not in RETRYABLE or index == attempts - 1:
raise
except URLError:
if index == attempts - 1:
raise
time.sleep(2 ** index)
raise RuntimeError("unreachable")
Production code should add random jitter so many clients do not retry at the same instant. Respect Retry-After when appropriate and set a total time budget. A series of attempts should not each consume the full user-facing deadline.
A timeout is not a complete limit
The timeout argument limits blocking operations, but robust clients also bound body size, redirect count, and overall duration. Reading until end-of-file without a size limit can take a long time after a connection has succeeded.
When a user controls the destination URL, add SSRF protections: allowed schemes and ports, DNS and IP checks, redirect validation, and strict response limits. Exception handling does not validate the destination.
TLS failures
Certificate problems commonly arrive as a URLError whose reason is an SSL exception. Do not fix them by disabling verification.
import ssl
from urllib.error import URLError
try:
# HTTPS call
pass
except URLError as error:
if isinstance(error.reason, ssl.SSLCertVerificationError):
print("The certificate could not be verified")
raise
Correct the clock, trust chain, hostname, or CA bundle. The guide to Python ssl explains secure contexts and certificate validation.
ContentTooShortError
ContentTooShortError is associated with urlretrieve() when the downloaded content is shorter than the expected Content-Length. Its content attribute preserves the received data, but that data must be treated as incomplete.
from urllib.error import ContentTooShortError
from urllib.request import urlretrieve
try:
path, headers = urlretrieve(
"https://example.com/archive.zip",
"archive.zip",
)
except ContentTooShortError as error:
print("Truncated download:", len(error.content))
raise
Do not process a partial archive silently. Remove or quarantine the temporary target, retry according to policy, and verify a hash or signature when one is available. For secure temporary locations, use Python tempfile.
Error bodies with unknown encoding
A server can declare a charset, omit it, or provide incorrect metadata. For diagnostics, use the declared charset when suitable and a replacement fallback. Strict UTF-8 decoding inside an error handler may raise a second exception and hide the original failure.
def read_error(error: HTTPError, limit: int = 64_000) -> str:
data = error.read(limit)
charset = error.headers.get_content_charset() or "utf-8"
return data.decode(charset, errors="replace")
Logging without leaking secrets
Record the method, host, normalized path, status, duration, and a correlation identifier. Do not log bearer tokens, Authorization headers, cookies, URL passwords, or complete bodies containing personal data.
The url attribute may include a sensitive query string. Redact known parameters before logging. Also bound the text from reason, because externally influenced content can reach operational systems.
Expected failures versus programming bugs
Catch only exceptions you can handle. An except Exception around an entire workflow can turn a programming defect into a misleading network warning. Keep the try region narrow.
try:
response = urlopen(request, timeout=10)
except (HTTPError, URLError) as error:
handle_network_failure(error)
else:
with response:
process(response)
Errors from process() are no longer mislabeled as transport failures.
Command-line and library APIs
A command-line program can convert known failures into concise messages and stable exit codes, while preserving details in verbose mode. A reusable library should usually propagate the original exception or wrap it with raise DomainError(...) from error so callers retain the cause.
Testing without the public internet
Run a local HTTP server that deliberately returns 404, 429, 500, redirects, slow responses, and truncated bodies. The guide to Python socketserver can support controlled integration tests. Mocks are fast, but a local protocol test reveals framing and body-handling mistakes.
Test handler order, bounded reads, retry only for eligible operations, preservation of exception causes, and redaction of secrets.
Common mistakes
Frequent mistakes include catching URLError before HTTPError, retrying every response, disabling TLS verification, reading unlimited bodies, comparing unstable error messages, logging credentials, accepting truncated files, retrying POST without idempotency protection, and catching exceptions too broadly.
Conclusion
urllib.error separates HTTP responses, transport failures, and incomplete downloads. That distinction improves user messages, metrics, and recovery decisions. Use HTTPError to inspect status, headers, and body; inspect URLError.reason for the underlying transport cause; and treat ContentTooShortError data as untrusted and incomplete.
Read the official urllib.error documentation and HTTP Semantics. A safe client combines timeouts, limits, destination validation, selective retries, and secret-free logging.
