The Python urllib.parse module splits URLs into components, rebuilds addresses, resolves relative references, and encodes query-string data. It replaces fragile manual string concatenation with well-defined functions for schemes, authorities, paths, queries, and fragments.
The library is practical and backward compatible, but it is not a complete URL validator. urlsplit() and urlparse() may accept unusual input and return empty components rather than raising an error. When a URL affects security, validate the scheme, host, port, credentials, path, and eventual destination after parsing.
Split a URL
from urllib.parse import urlsplit
result = urlsplit(
"https://user:secret@api.example.com:8443/v1/items"
"?page=2&sort=name#details"
)
print(result.scheme)
print(result.hostname)
print(result.port)
print(result.path)
print(result.query)
print(result.fragment)urlsplit() returns scheme, netloc, path, query, and fragment. The structured result also exposes username, password, hostname, and port.
Prefer urlsplit for modern URLs
urlparse() adds a historical params field. Most current HTTP code is simpler with urlsplit(). Use urlparse() only when path-parameter separation is part of the format your application genuinely needs.
Double slashes change the interpretation
from urllib.parse import urlsplit
print(urlsplit("example.com/page"))
print(urlsplit("//example.com/page"))
print(urlsplit("https://example.com/page"))Without //, the first value is a path, not a hostname. Do not prepend a scheme blindly, because a local path, custom identifier, or dangerous scheme may be transformed into an unintended destination.
Validate scheme, hostname, and port
from urllib.parse import urlsplit
ALLOWED_SCHEMES = {"https"}
ALLOWED_HOSTS = {"api.example.com", "cdn.example.com"}
def validate_url(text: str):
parts = urlsplit(text)
if parts.scheme.lower() not in ALLOWED_SCHEMES:
raise ValueError("scheme is not allowed")
if parts.hostname not in ALLOWED_HOSTS:
raise ValueError("host is not allowed")
if parts.username is not None or parts.password is not None:
raise ValueError("credentials in URLs are rejected")
try:
port = parts.port
except ValueError as exc:
raise ValueError("invalid port") from exc
if port not in (None, 443):
raise ValueError("port is not allowed")
return partshostname is lowercased, while netloc preserves more of the original text. Reading port may raise ValueError for an out-of-range or malformed value.
Parsing is not validation
The official documentation explicitly warns that parsing functions favor practical behavior. They are not strict validators for either RFC 3986 or the browser-oriented WHATWG standard. Your application must define what is acceptable.
A policy may require HTTPS, a known hostname, default ports, no credentials, an absolute path, a maximum length, and no control characters. Another system may intentionally allow relative references. Validation must follow that contract.
URLs containing IP addresses
When a hostname may be an IP literal, validate it with Python ipaddress. This helps detect loopback, private, and link-local targets, but it does not solve SSRF alone. Hostnames may resolve to several IPs, change after validation, or redirect to a different destination.
Remove fragments
from urllib.parse import urldefrag
result = urldefrag("https://example.com/manual#install")
print(result.url)
print(result.fragment)A fragment is normally not sent to an HTTP server. Removing it can help cache keys and comparisons, although equivalent URLs may still differ in default ports, percent encoding, or path normalization.
Rebuild structured results
from urllib.parse import urlsplit
parts = urlsplit("HTTP://Example.com/page?#")
clean = parts._replace(fragment="").geturl()
print(clean)_replace() creates a new result. geturl() may lowercase the scheme and remove empty delimiters. Do not use it when byte-for-byte preservation of the original text is required.
Resolve relative URLs with urljoin
from urllib.parse import urljoin
base = "https://docs.example.com/guides/python/"
print(urljoin(base, "install.html"))
print(urljoin(base, "../reference/api.html"))urljoin() implements relative-reference resolution and is useful in crawlers, feeds, and documentation tools.
urljoin can replace the domain
from urllib.parse import urljoin
base = "https://site.example.com/users/"
print(urljoin(base, "https://attacker.example/steal"))An absolute second argument replaces the base scheme and hostname. Do not call urljoin(base, attacker_controlled_value) expecting the result to stay on the original site.
from urllib.parse import urljoin, urlsplit
def join_internal(base: str, relative: str) -> str:
candidate = urlsplit(relative)
if candidate.scheme or candidate.netloc:
raise ValueError("absolute URL is not allowed")
result = urljoin(base, relative)
if urlsplit(result).hostname != urlsplit(base).hostname:
raise ValueError("destination left the allowed host")
return resultValidate the final URL again because path normalization and redirects can introduce additional policy decisions.
Quote path components
from urllib.parse import quote
name = "Sales report/2026"
print(quote(name))
print(quote(name, safe=""))quote() uses percent encoding. A slash is safe by default because the function is designed for paths. To encode one path segment, set safe="" so a slash cannot create another segment.
Do not quote an entire URL blindly
Schemes, hostnames, paths, queries, and fragments follow different rules. Applying quote() to the full URL can encode structural separators such as :, /, and ?. Build and encode each component separately.
quote versus quote_plus
from urllib.parse import quote, quote_plus
text = "advanced python"
print(quote(text)) # advanced%20python
print(quote_plus(text)) # advanced+pythonquote_plus() is intended for HTML form and query data, where spaces become plus signs. Use quote() for paths. A literal plus sign must be encoded to avoid becoming a space during unquote_plus().
Build query strings with urlencode
from urllib.parse import urlencode
parameters = {
"search": "python web",
"page": "2",
"language": "en",
}
query = urlencode(parameters)
url = f"https://api.example.com/search?{query}"
print(url)urlencode() handles special characters correctly. In Python 3.14, passing some false-valued objects other than empty strings, bytes-like values, and None is deprecated. Convert domain values and numbers explicitly.
Repeated parameters
from urllib.parse import urlencode
query = urlencode(
{"tag": ["python", "web"], "page": "1"},
doseq=True,
)
print(query)Without doseq=True, a list may be encoded as its Python representation instead of multiple key-value pairs.
Preserve order with pairs
from urllib.parse import urlencode
pairs = [
("sort", "name"),
("sort", "date"),
("page", "1"),
]
print(urlencode(pairs))A sequence of pairs retains order and duplicate keys. This can matter for request signatures or APIs with ordered parameters, although such a rule should be documented.
Parse query strings safely
from urllib.parse import parse_qs, parse_qsl
query = "tag=python&tag=web&empty="
print(parse_qs(query, keep_blank_values=True))
print(parse_qsl(query, keep_blank_values=True))parse_qs() groups values into lists; parse_qsl() preserves the sequence. Set max_num_fields for external data to avoid excessive resource consumption.
values = parse_qs(
incoming_query,
keep_blank_values=True,
strict_parsing=True,
max_num_fields=200,
)Decoding errors
unquote() uses UTF-8 and replaces invalid sequences by default. Critical parsers may choose errors="strict" to reject them. unquote_to_bytes() returns original octets when the application must decide the encoding itself.
Avoid double decoding
%252e%252e%252f becomes %2e%2e%2f after one decoding and ../ after two. Decode exactly once in a clearly defined layer. When decoded values become local paths, validate them against traversal attacks; the same class of risk appears in Python tarfile.
Text and bytes
Parsing functions accept either str or ASCII bytes. Do not mix them in one call. Non-ASCII bytes raise UnicodeDecodeError. Web applications usually benefit from decoding transport data explicitly and then using text consistently.
IPv6 literals in URLs
An IPv6 hostname must be enclosed in brackets, such as https://[2001:db8::1]:8443/. Unmatched brackets raise an error. Validate the parsed hostname with ipaddress and define how scope IDs are handled.
Protect logs from secrets
URLs can contain passwords, tokens, personal data, and signatures. Do not log the full value by default. Remove user information and redact sensitive query keys.
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
SENSITIVE = {"token", "password", "key", "signature"}
def url_for_log(url: str) -> str:
parts = urlsplit(url)
query = urlencode([
(key, "***" if key.lower() in SENSITIVE else value)
for key, value in parse_qsl(parts.query, keep_blank_values=True)
])
host = parts.hostname or ""
if parts.port:
host = f"{host}:{parts.port}"
return urlunsplit((parts.scheme, host, parts.path, query, ""))Configuration and concurrency
Load allowed schemes, hosts, and ports through a validated configuration layer such as Python configparser. Use Python trace to inspect redirect and parsing decisions. Crawlers can distribute URLs through Python queue while applying per-host limits.
Testing strategy
Test absolute and relative URLs, missing hosts, embedded credentials, invalid ports, IPv6, control characters, Unicode, fragments, duplicate query values, too many fields, invalid percent encoding, double encoding, absolute input passed to urljoin(), and redirects to private networks.
Best practices
- Prefer
urlsplit()for modern URLs. - Validate every component after parsing.
- Use allowlists for critical schemes and destinations.
- Do not trust
urljoin()with external input. - Encode each component separately.
- Limit query fields.
- Avoid double decoding.
- Redact secrets in logs.
- Combine URL, DNS, and IP validation against SSRF.
Conclusion
Python urllib.parse provides dependable primitives for URL decomposition, resolution, quoting, and query strings. Safe use requires an additional validation layer based on the exact contract of the application.
Read the official urllib.parse documentation and RFC 3986. Parsing returns components; deciding whether to trust the destination is a separate step.
