The urllib.robotparser module parses robots.txt files and answers whether a user agent may fetch a specific URL. It is an important component of responsible crawlers, indexers, monitors, and automation because it handles Allow, Disallow, crawl delays, suggested request rates, and sitemap declarations.
Respecting robots.txt is not the only responsibility of a crawler. The file does not grant permission, replace terms of service, secure private content, or guarantee that collection is ethical or lawful. This guide covers RobotFileParser, bounded fetching, updates, shared rate limits, failure policy, and origin-aware caching.
What is robots.txt?
A site can publish /robots.txt at the root of an origin, such as https://example.com/robots.txt. The file groups rules by user-agent token.
User-agent: MyCrawler
Disallow: /admin/
Allow: /admin/public-docs/
Crawl-delay: 5
Sitemap: https://example.com/sitemap.xml
Rules apply to a specific scheme, hostname, and port. One subdomain’s file does not automatically control another. Core behavior is standardized by RFC 9309.
Basic RobotFileParser usage
from urllib.robotparser import RobotFileParser
parser = RobotFileParser("https://example.com/robots.txt")
parser.read()
allowed = parser.can_fetch(
"MyCrawler",
"https://example.com/articles/python",
)
print(allowed)
read() downloads and parses the file. For explicit timeout, size, status, and TLS handling, fetch the content with an HTTP client and call parse(lines).
The urllib.request guide demonstrates bounded HTTP downloads.
Manual parsing with limits
from urllib.request import Request, urlopen
from urllib.robotparser import RobotFileParser
MAX_ROBOTS_BYTES = 512 * 1024
USER_AGENT = "MyCrawler/1.0 (+https://example.org/bot)"
request = Request(
"https://example.com/robots.txt",
headers={"User-Agent": USER_AGENT},
)
with urlopen(request, timeout=10) as response:
raw = response.read(MAX_ROBOTS_BYTES + 1)
if len(raw) > MAX_ROBOTS_BYTES:
raise ValueError("robots.txt exceeded the size limit")
text = raw.decode("utf-8", errors="replace")
parser = RobotFileParser()
parser.set_url(request.full_url)
parser.parse(text.splitlines())
parser.modified()
Full crawlers should implement the protocol’s exact limits and encoding rules. A conservative application cap prevents unexpected resource consumption.
Use one consistent user agent
Use the same token when downloading robots.txt, calling can_fetch(), and sending real requests. A descriptive HTTP user agent helps administrators understand traffic and contact the operator.
USER_AGENT_TOKEN = "MyCrawler"
HTTP_USER_AGENT = "MyCrawler/1.0 (+mailto:bot@example.org)"
if not parser.can_fetch(USER_AGENT_TOKEN, target_url):
raise PermissionError("URL is blocked by robots.txt")
Do not impersonate a browser or a well-known crawler. Do not use * merely because it produces a more permissive decision when a specific group applies.
Validating target URLs
from urllib.parse import urljoin, urlsplit
base = "https://example.com/"
target = urljoin(base, "/products?page=2")
parts = urlsplit(target)
if parts.scheme != "https" or parts.hostname != "example.com":
raise ValueError("Unexpected target")
if parser.can_fetch(USER_AGENT_TOKEN, target):
print("Allowed")
The urllib.parse guide explains secure URL joining and validation.
Crawl delay
crawl_delay() returns the suggested number of seconds between requests or None.
delay = parser.crawl_delay(USER_AGENT_TOKEN)
if delay is None:
delay = 2.0
delay = max(delay, 1.0)
Coordinate the delay across every worker accessing the same origin. Sleeping independently in ten threads can still generate ten times the intended rate.
Request rate
request_rate() returns a tuple containing request count and seconds.
rate = parser.request_rate(USER_AGENT_TOKEN)
if rate:
interval = rate.seconds / rate.requests
print(f"average minimum interval: {interval:.2f}s")
Use a shared token bucket or another limiter. Combine request-rate, crawl-delay, HTTP 429, Retry-After, and your own conservative ceiling.
Sitemaps
site_maps() returns sitemap URLs declared in the file.
for sitemap_url in parser.site_maps() or []:
print(sitemap_url)
Sitemaps can reduce crawling by listing public resources directly. Validate each URL, limit response size and decompression, and apply the same origin and request policies.
Caching and refresh
mtime() reports when rules were fetched, and modified() updates that time.
import time
REFRESH_SECONDS = 6 * 60 * 60
if time.time() - parser.mtime() > REFRESH_SECONDS:
refresh_robots(parser)
Maintain one cached parser per origin, refresh it periodically, and use synchronization so many workers do not refresh simultaneously.
Failure policy
Define behavior for 404, 401, 403, timeout, malformed content, and 5xx responses. A responsible crawler should usually pause or reduce collection on temporary failure rather than converting every exception into permission.
Use bounded retries with backoff and record the status. A network outage is not evidence that all paths are allowed.
robots.txt is not access control
The rules are public and voluntary. They do not stop a malicious client. Private data requires authentication and authorization on the server. Publishing sensitive paths in robots.txt may even reveal them.
Redirects and origin changes
Whenever a link or redirect changes scheme, hostname, or port, load a separate parser and rate limiter. Never apply rules from example.com to cdn.example.net.
Identification and logging
Use a user agent containing a name, version, and contact URL or email. Record the target URL, robots decision, applied delay, status, byte count, and duration. Avoid logging collected personal data unnecessarily.
Integrating with a crawler
def may_fetch(url: str, cache: dict[str, RobotFileParser]) -> bool:
origin = origin_from_url(url)
parser = cache.get(origin)
if parser is None or is_stale(parser):
parser = load_robots(origin)
cache[origin] = parser
return parser.can_fetch(USER_AGENT_TOKEN, url)
After the decision, the crawler still needs shared concurrency limits, 429 handling, cycle prevention, canonicalization, deduplication, and SSRF protection.
Responsible web scraping
The web scraping guide covers extraction, pagination, and responsible practices. Prefer an official API, feed, or export when one exists.
Common mistakes
Typical errors include downloading robots.txt for every page, using different user agents for parsing and requests, ignoring delays, treating errors as permission, applying one parser to another origin, assuming robots.txt provides authorization, allowing an unlimited file, and running many workers without a shared limiter.
Best practices
Cache one parser per origin, refresh it, identify the crawler truthfully, apply the most conservative rate, validate sitemaps and redirects, cap responses, and pause on uncertainty. Respect privacy, copyright, terms, and administrator requests even when a path is technically allowed.
Conclusion
urllib.robotparser provides a simple interface to robots.txt, but responsible crawling requires origin-aware caching, shared rate limits, conservative failure handling, honest identification, and URL validation. Treat the module as an operational-respect layer, not authorization or legal approval.
Read the official urllib.robotparser documentation and RFC 9309.
