Web scraping is the automated collection of information from web pages. Instead of copying prices, headlines, links, or table rows manually, a Python program can download the HTML, locate the relevant elements, clean the values, and save the results.
This beginner guide explains web scraping with BeautifulSoup and Requests. You will learn how to fetch pages safely, inspect HTML, select elements, extract text and attributes, follow pagination, use sessions and retries, export results, and respect website rules.
Before You Scrape a Website
Publicly visible information is not automatically free of restrictions. Review the site’s terms of service, privacy policy, copyright conditions, and robots.txt file. Avoid personal or sensitive data, respect rate limits, and never bypass authentication, access controls, CAPTCHAs, or technical protections.
Prefer an official API when one exists. APIs usually provide structured, stable data and explicit usage rules. Web scraping should be polite, limited, and designed not to disrupt the site.
Install Requests and Beautiful Soup
Create a virtual environment and install the required packages:
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS or Linux
source .venv/bin/activate
python -m pip install requests beautifulsoup4The Python virtual environment guide explains project isolation. The primary references are the Requests documentation and the Beautiful Soup documentation.
Download a Page with Requests
Use requests.get() with a timeout. Then call raise_for_status() so HTTP errors become exceptions instead of being silently processed as valid pages.
import requests
url = "https://example.com"
response = requests.get(url, timeout=15)
response.raise_for_status()
print(response.status_code)
print(response.text[:200])The response object contains the status code, headers, detected encoding, final URL, and body. A timeout prevents the program from waiting forever.
Send an Honest User-Agent
Some servers reject requests without a recognizable user agent. Identify the script responsibly instead of pretending to be a human browser.
headers = {
"User-Agent": "LearningScraper/1.0 (contact: [email protected])"
}
response = requests.get(
"https://example.com/products",
headers=headers,
timeout=15,
)
response.raise_for_status()Do not use headers to evade rules or restrictions. A clear identifier can help a site owner contact you if the script causes a problem.
Parse HTML with Beautiful Soup
from bs4 import BeautifulSoup
soup = BeautifulSoup(response.text, "html.parser")
print(soup.title)
print(soup.title.get_text(strip=True))html.parser is included with Python. Other parsers may be faster or more tolerant, but they require additional packages and can produce slightly different trees.
Inspect the Page Structure
Open the page in a browser, right-click the desired element, and choose Inspect. Look for stable semantic elements, IDs, classes, or data attributes. Do not rely on a selector merely because it works once; auto-generated class names may change frequently.
Consider this simplified HTML:
<article class="product-card" data-id="42">
<h2 class="product-name">Mechanical Keyboard</h2>
<span class="price">$89.90</span>
<a class="details" href="/products/42">View product</a>
</article>Find Elements
Beautiful Soup provides several approaches:
card = soup.find("article", class_="product-card")
name = card.find("h2", class_="product-name")
print(name.get_text(strip=True))CSS selectors are often concise:
card = soup.select_one("article.product-card")
name = card.select_one(".product-name")
price = card.select_one(".price")
print(name.get_text(strip=True))
print(price.get_text(strip=True))select_one() returns the first match or None. select() returns a list of all matches.
Extract Text and Attributes
from urllib.parse import urljoin
card = soup.select_one("article.product-card")
name = card.select_one(".product-name").get_text(strip=True)
price_text = card.select_one(".price").get_text(strip=True)
link_element = card.select_one("a.details")
product_url = urljoin(response.url, link_element.get("href"))
product_id = card.get("data-id")
print(name)
print(price_text)
print(product_url)
print(product_id)urljoin() converts a relative link into an absolute URL. Use get() for optional attributes because direct indexing raises an error when the attribute is missing.
Extract Multiple Product Cards
products = []
for card in soup.select("article.product-card"):
name_element = card.select_one(".product-name")
price_element = card.select_one(".price")
link_element = card.select_one("a.details")
if not name_element or not price_element:
continue
product = {
"name": name_element.get_text(" ", strip=True),
"price_text": price_element.get_text(" ", strip=True),
"url": urljoin(
response.url,
link_element.get("href", "") if link_element else "",
),
}
products.append(product)
print(products)Checking optional elements prevents one incomplete card from terminating the entire collection.
Clean Prices
Extracted text often contains currency symbols, spaces, or regional separators. Create a dedicated cleaning function rather than mixing parsing rules with page navigation.
from decimal import Decimal, InvalidOperation
def parse_price(text: str) -> Decimal:
cleaned = (
text.replace("$", "")
.replace(",", "")
.strip()
)
try:
return Decimal(cleaned)
except InvalidOperation as exc:
raise ValueError(f"Invalid price: {text!r}") from exc
price = parse_price("$1,299.90")
print(price)Adapt the function to the site’s documented locale. Do not assume that commas and periods have the same meaning everywhere.
Use a Session
A Session reuses connections and shared headers. It can also retain cookies when that behavior is allowed and required.
import requests
session = requests.Session()
session.headers.update({
"User-Agent": "LearningScraper/1.0 (contact: [email protected])"
})
response = session.get("https://example.com/products", timeout=15)
response.raise_for_status()Close the session when finished, or use a context manager:
with requests.Session() as session:
session.headers.update(headers)
response = session.get(url, timeout=15)
response.raise_for_status()Add Limited Retries
Temporary server failures may justify a small number of delayed retries. Do not aggressively retry blocked or forbidden requests.
from requests import Session
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
retry_policy = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504],
allowed_methods=["GET"],
respect_retry_after_header=True,
)
session = Session()
session.mount("https://", HTTPAdapter(max_retries=retry_policy))
session.mount("http://", HTTPAdapter(max_retries=retry_policy))A 429 response means the client is sending too many requests. Respect the server’s Retry-After header and reduce the request rate.
Follow Pagination
Many sites divide results into pages. The safest pattern is to follow the explicit next-page link until it disappears.
from time import sleep
from urllib.parse import urljoin
next_url = "https://example.com/products"
all_products = []
while next_url:
response = session.get(next_url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("article.product-card"):
name = card.select_one(".product-name")
price = card.select_one(".price")
if name and price:
all_products.append({
"name": name.get_text(" ", strip=True),
"price": price.get_text(" ", strip=True),
})
next_link = soup.select_one("a.next-page")
next_url = (
urljoin(response.url, next_link.get("href"))
if next_link and next_link.get("href")
else None
)
if next_url:
sleep(2)Add a page limit while developing so a selector mistake cannot trigger an endless crawl.
Export to CSV
import csv
with open("products.csv", "w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(
file,
fieldnames=["name", "price", "url"],
)
writer.writeheader()
writer.writerows(products)For table-shaped analysis, Pandas can simplify cleaning and export:
import pandas as pd
frame = pd.DataFrame(products)
frame.to_csv("products.csv", index=False)
frame.to_excel("products.xlsx", index=False)The Pandas beginner guide covers DataFrames, cleaning, filtering, and export.
Export to JSON
import json
with open("products.json", "w", encoding="utf-8") as file:
json.dump(products, file, ensure_ascii=False, indent=2)JSON preserves nested structures better than CSV and is convenient for APIs and application data.
Handle Errors
import requests
from bs4 import BeautifulSoup
def fetch_soup(session, url):
try:
response = session.get(url, timeout=15)
response.raise_for_status()
except requests.Timeout as exc:
raise RuntimeError(f"Request timed out: {url}") from exc
except requests.HTTPError as exc:
raise RuntimeError(
f"HTTP error {exc.response.status_code}: {url}"
) from exc
except requests.RequestException as exc:
raise RuntimeError(f"Request failed: {url}") from exc
return BeautifulSoup(response.text, "html.parser")Catch specific network exceptions and preserve their context. The Python exception-handling guide explains chaining and cleanup.
A Complete Small Scraper
import csv
from time import sleep
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup
BASE_URL = "https://example.com"
START_URL = f"{BASE_URL}/products"
MAX_PAGES = 5
def parse_card(card, page_url):
name_element = card.select_one(".product-name")
price_element = card.select_one(".price")
link_element = card.select_one("a.details")
if not name_element or not price_element:
return None
return {
"name": name_element.get_text(" ", strip=True),
"price": price_element.get_text(" ", strip=True),
"url": urljoin(
page_url,
link_element.get("href", "") if link_element else "",
),
}
def scrape_products():
products = []
next_url = START_URL
with requests.Session() as session:
session.headers.update({
"User-Agent": "LearningScraper/1.0 (contact: [email protected])"
})
for _ in range(MAX_PAGES):
if not next_url:
break
response = session.get(next_url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("article.product-card"):
product = parse_card(card, response.url)
if product:
products.append(product)
next_link = soup.select_one("a.next-page")
next_url = (
urljoin(response.url, next_link.get("href"))
if next_link and next_link.get("href")
else None
)
if next_url:
sleep(2)
return products
def save_csv(products, filename="products.csv"):
with open(filename, "w", newline="", encoding="utf-8") as file:
writer = csv.DictWriter(
file,
fieldnames=["name", "price", "url"],
)
writer.writeheader()
writer.writerows(products)
if __name__ == "__main__":
try:
products = scrape_products()
save_csv(products)
print(f"Saved {len(products)} products.")
except requests.RequestException as exc:
print(f"Network error: {exc}")Replace the demonstration selectors and URL only after confirming that collection is permitted.
Why Some Pages Return No Data
Requests downloads the server response, but some sites build content later with JavaScript. The desired elements may not exist in the original HTML.
Check the response text and the browser’s page source. If the information comes from a documented public API, use that API. For permitted browser automation and testing, the Selenium with Python guide explains dynamic pages, waits, and selectors. Browser automation is heavier and does not remove legal or contractual restrictions.
Selectors Can Break
A scraper depends on another site’s structure, which can change without notice. Reduce fragility by:
- Choosing semantic, stable selectors.
- Validating required fields.
- Logging the source URL and missing selector.
- Saving small permitted HTML fixtures for parser tests.
- Separating download logic from parsing logic.
- Writing automated tests for known samples.
The Pytest guide can help test parsing functions without repeatedly contacting a live site.
Common Mistakes
- Sending requests without a timeout.
- Ignoring HTTP status codes.
- Scraping too quickly or in parallel without permission.
- Assuming every element exists.
- Using fragile copied selectors without understanding them.
- Saving unclean text directly as numbers.
- Collecting personal data without a lawful purpose.
- Trying to bypass blocks, authentication, or CAPTCHAs.
- Depending on a scraper when an official API exists.
Frequently Asked Questions
What is the difference between Requests and Beautiful Soup?
Requests downloads web resources. Beautiful Soup parses HTML or XML and helps locate elements inside the document.
Can Beautiful Soup execute JavaScript?
No. It parses the content it receives. Use an official API or an appropriate permitted browser-automation tool when data is rendered later.
Is web scraping legal?
It depends on the jurisdiction, data, access method, contracts, and purpose. Review the site’s rules and obtain professional legal advice for high-risk or commercial collection.
How fast should a scraper run?
Use a conservative rate, follow published limits and Retry-After, and stop when the server signals throttling or denial.
Should I use regular expressions to parse HTML?
Use an HTML parser for document structure. Regular expressions may help clean extracted text, but they are not a reliable general HTML parser.
Conclusion
A responsible scraper has three separate jobs: download pages carefully, parse and validate data, and save structured results. Use timeouts, status checks, sessions, limited retries, stable selectors, clear cleaning functions, and respectful delays. Prefer official APIs, follow site rules, and design the parser so changes can be detected and repaired quickly.
