The urllib.request module opens URLs and sends HTTP requests using only the Python standard library. It provides urlopen(), configurable Request objects, and an extensible handler system for redirects, authentication, cookies, proxies, and HTTPS.
Higher-level clients such as Requests or HTTPX are more convenient for large applications, but urllib.request is valuable in portable scripts, installers, administration tools, and constrained environments. This guide covers GET, POST, JSON, bounded downloads, TLS, redirects, proxies, errors, retries, and protection against untrusted URLs.
A basic GET request
from urllib.request import urlopen
with urlopen("https://www.python.org/", timeout=10) as response:
print(response.status)
print(response.headers.get_content_type())
data = response.read(4096)
The response is a context manager and exposes status, headers, and url. The body is bytes because the client cannot reliably determine the text encoding automatically.
charset = response.headers.get_content_charset() or "utf-8"
text = data.decode(charset, errors="replace")
The Python codecs guide explains the boundary between text and bytes.
Always set a timeout
Without a timeout, DNS resolution, connection establishment, TLS, or reads may block for a long time.
with urlopen(url, timeout=10) as response:
body = response.read(1_000_000)
A timeout is not a response-size limit. A server can continuously send data while remaining active. Count bytes and stop when the configured maximum is exceeded.
Using Request
from urllib.request import Request, urlopen
request = Request(
"https://api.example.com/items",
headers={
"Accept": "application/json",
"User-Agent": "MyTool/1.0",
},
method="GET",
)
with urlopen(request, timeout=10) as response:
body = response.read(500_000)
Identify automation honestly. Do not impersonate a browser to bypass access policies. Respect rate limits, terms, and robots rules where relevant.
Correct query strings
Use urlencode() rather than manual concatenation.
from urllib.parse import urlencode
params = urlencode({"q": "secure python", "page": 2})
url = f"https://example.com/search?{params}"
The urllib.parse guide covers quoting, URL decomposition, and security validation.
Form POST requests
from urllib.parse import urlencode
from urllib.request import Request, urlopen
form = urlencode({"name": "Ana", "active": "1"}).encode("ascii")
request = Request(
"https://example.com/form",
data=form,
headers={"Content-Type": "application/x-www-form-urlencoded"},
method="POST",
)
with urlopen(request, timeout=10) as response:
result = response.read(100_000)
If data is supplied and no method is declared, POST becomes the default. Explicit methods make behavior easier to review.
Sending JSON
import json
from urllib.request import Request, urlopen
payload = json.dumps({"title": "Example"}).encode("utf-8")
request = Request(
"https://api.example.com/items",
data=payload,
headers={
"Content-Type": "application/json",
"Accept": "application/json",
},
method="POST",
)
with urlopen(request, timeout=10) as response:
raw = response.read(500_000)
result = json.loads(raw.decode("utf-8"))
Validate Content-Type, status, and size before parsing. The REST API guide adds response validation, authentication, and retry strategies.
HTTPError and URLError
from urllib.error import HTTPError, URLError
try:
with urlopen(request, timeout=10) as response:
body = response.read(100_000)
except HTTPError as error:
error_body = error.read(20_000)
print(error.code, error.reason)
except URLError as error:
print(f"Network failure: {error.reason}")
except TimeoutError:
print("Request timed out")
HTTPError represents an HTTP error response and remains readable as a response object. URLError covers DNS, connection, TLS, and protocol failures.
Bounded downloads
from pathlib import Path
MAX_BYTES = 50 * 1024 * 1024
with urlopen(url, timeout=20) as response:
declared = response.headers.get("Content-Length")
if declared and int(declared) > MAX_BYTES:
raise ValueError("Declared file is too large")
total = 0
with Path("download.tmp").open("wb") as output:
while chunk := response.read(64 * 1024):
total += len(chunk)
if total > MAX_BYTES:
raise ValueError("Download exceeded limit")
output.write(chunk)
Write to a temporary file, validate its format or digest, and atomically move it into place. The hashlib guide shows SHA-256 verification.
Compressed responses
urllib.request does not transparently handle every content encoding. If you request gzip, inspect the header and limit compressed and expanded data.
import gzip
encoding = response.headers.get("Content-Encoding", "").lower()
raw = response.read(2_000_000)
data = gzip.decompress(raw) if encoding == "gzip" else raw
For large or untrusted content, use incremental decompression with an output cap. See the Python gzip guide.
TLS and private CAs
import ssl
context = ssl.create_default_context(cafile="company-ca.pem")
with urlopen(request, timeout=10, context=context) as response:
body = response.read(100_000)
HTTPS certificate and hostname validation are secure by default. Do not disable them. Fix the trust chain instead. The Python ssl guide explains TLS contexts.
Redirect behavior
The default opener follows HTTP redirects. Inspect response.url to learn the final destination. Sensitive headers should not be forwarded to an unexpected host. Use add_unredirected_header() or a restrictive redirect handler for credentials.
Responses 301 and 302 may change POST to GET, matching common browser behavior. Codes 307 and 308 preserve the method. Review this policy for state-changing requests.
Disabling automatic redirects
from urllib.request import build_opener, HTTPRedirectHandler
class NoRedirect(HTTPRedirectHandler):
def redirect_request(self, req, fp, code, msg, headers, newurl):
return None
opener = build_opener(NoRedirect())
If redirects are allowed, cap their count and validate the scheme, hostname, port, and resolved IP at every hop.
Environment proxies
The default opener can read http_proxy, https_proxy, and system settings. Sensitive services should not depend on uncontrolled environment state.
from urllib.request import ProxyHandler, build_opener
opener = build_opener(ProxyHandler({}))
An explicit empty mapping disables auto-detected proxies. Configure approved proxies directly and keep credentials out of source code.
Basic and Digest authentication
HTTPBasicAuthHandler and HTTPDigestAuthHandler integrate credentials into an opener. Basic authentication only encodes credentials and therefore requires HTTPS. Scope each credential to the intended URI. Python 3.14 added SHA-256 support to Digest authentication.
SSRF protection
Never pass an attacker-provided URL directly to urlopen(). The library supports more than HTTPS, including file:, data:, and FTP. An application can accidentally read local files or contact internal services.
Parse the URL, allow only HTTPS, normalize the hostname, resolve DNS, block private, loopback, link-local, and cloud-metadata addresses, and repeat validation after each redirect. Account for DNS rebinding and differences between validated and connected addresses.
Safe retries
The module does not provide a complete retry policy. Retry only transient failures and idempotent operations, using exponential backoff and a maximum attempt count. A POST may have succeeded even when its response was lost. Use an idempotency key when the API supports one.
Common mistakes
Common problems include missing timeouts, unlimited reads, assuming an encoding, disabling TLS verification, following redirects without destination checks, leaking Authorization headers, accepting arbitrary schemes, inheriting untrusted proxy variables, and blindly retrying POST.
Best practices
Create explicit Request objects, set timeouts, cap response and decompression sizes, validate status and media type, use secure HTTPS contexts, control redirects and proxies, and close responses with with. For complex applications, prefer a maintained HTTP client with connection pooling and richer policies.
Conclusion
urllib.request is a capable dependency-free HTTP client. It works well for controlled scripts when timeouts, limits, TLS, redirects, and external URLs are handled explicitly. Its handler architecture is powerful, but safe usage requires understanding each stage.
Read the official urllib.request documentation and RFC 9110 for HTTP semantics.
