Python Data Analysis with Pandas and NumPy

Python data analysis becomes much easier when you combine NumPy and Pandas. NumPy provides fast numerical arrays and mathematical operations. Pandas adds labeled tables, missing-value tools, grouping, filtering, and file import features.

These libraries are often used together. NumPy handles efficient numerical computation, while Pandas turns raw information into rows and columns that are convenient to explore. This guide walks through a complete beginner workflow, from installation to a small sales analysis.

What is data analysis?

Data analysis is the process of collecting, cleaning, transforming, exploring, and interpreting information. A typical project asks questions such as:

• What happened?
• Which categories performed best?
• Are values missing or inconsistent?
• How do results change over time?
• What patterns deserve further investigation?

Python is popular for this work because it combines readable code with libraries for tables, statistics, charts, machine learning, and automation. The broader Pandas guide and NumPy introduction explore each library separately.

Installing Pandas and NumPy

Create a virtual environment before installing project dependencies:

python -m venv .venv

Activate it and install both libraries:

python -m pip install numpy pandas

The Python virtual environment tutorial explains why isolated environments prevent dependency conflicts.

Import the libraries with their conventional aliases:

import numpy as np
import pandas as pd

NumPy arrays

A NumPy array stores values of a consistent type in an efficient structure. It supports vectorized operations, meaning one expression can process an entire array without a manual loop.

import numpy as np

prices = np.array([12.50, 8.90, 15.00, 6.75])
quantities = np.array([4, 8, 3, 10])

totals = prices * quantities
print(totals)
print(totals.sum())

With regular Python lists, multiplying two lists does not perform element-by-element arithmetic. NumPy arrays are designed for exactly this kind of numerical work.

Useful NumPy operations

scores = np.array([72, 88, 91, 65, 84])

print(scores.mean())
print(scores.min())
print(scores.max())
print(scores.std())
print(scores[scores >= 80])

Boolean filtering is especially powerful. The expression scores >= 80 creates a Boolean mask, and the array returns only matching values.

The official NumPy beginner guide covers shapes, dimensions, indexing, and vectorization in greater depth.

Pandas Series and DataFrames

A Series is a one-dimensional labeled collection. A DataFrame is a two-dimensional table with rows and columns.

import pandas as pd

sales = pd.DataFrame({
    "product": ["Notebook", "Mouse", "Keyboard", "Mouse"],
    "quantity": [2, 5, 3, 4],
    "unit_price": [950.00, 25.00, 70.00, 25.00],
})

print(sales)

A DataFrame can contain different data types across columns. Pandas uses NumPy internally for many operations, but adds labels and high-level table tools.

Loading data from files

CSV is one of the most common formats:

data = pd.read_csv("sales.csv")

Excel files can also be loaded:

data = pd.read_excel("sales.xlsx")

For file-specific workflows, review the guides to CSV files in Python and importing Excel data.

Always inspect the result immediately:

print(data.head())
print(data.shape)
print(data.columns)
print(data.dtypes)
print(data.info())

Selecting rows and columns

Select one column with brackets:

products = sales["product"]

Select several columns with a list:

summary = sales[["product", "quantity"]]

Use loc for label-based selection and iloc for numeric positions:

print(sales.loc[0, "product"])
print(sales.iloc[0:2, 0:3])

Filtering data

Boolean conditions filter rows:

large_orders = sales[sales["quantity"] >= 4]
print(large_orders)

Combine conditions with & and |. Wrap each condition in parentheses:

filtered = sales[
    (sales["quantity"] >= 3)
    & (sales["unit_price"] < 100)
]

Creating calculated columns

Vectorized arithmetic creates a revenue column without a row-by-row loop:

sales["revenue"] = sales["quantity"] * sales["unit_price"]
print(sales)

You can also use NumPy conditions:

sales["order_size"] = np.where(
    sales["revenue"] >= 200,
    "large",
    "small",
)

Cleaning missing data

Real datasets often contain blank values. First, count them:

print(data.isna().sum())

Fill missing values when a sensible replacement exists:

data["discount"] = data["discount"].fillna(0)

Remove rows only when missing data makes them unusable:

data = data.dropna(subset=["product", "quantity"])

Do not fill every blank with zero automatically. A missing age, missing price, and missing category represent different situations. Cleaning decisions should reflect the meaning of each column.

Fixing data types

Text files may load numbers or dates as strings. Convert them explicitly:

data["quantity"] = pd.to_numeric(data["quantity"], errors="coerce")
data["date"] = pd.to_datetime(data["date"], errors="coerce")

The errors="coerce" option turns invalid values into missing values, making them easier to detect and clean.

Grouping and aggregation

groupby() answers questions such as total revenue per product:

product_summary = (
    sales.groupby("product", as_index=False)
    .agg(
        units=("quantity", "sum"),
        revenue=("revenue", "sum"),
        average_price=("unit_price", "mean"),
    )
    .sort_values("revenue", ascending=False)
)

print(product_summary)

This pattern is central to reporting and exploratory analysis.

Sorting and finding top results

top_orders = sales.sort_values("revenue", ascending=False).head(3)
print(top_orders)

You can also use nlargest() for a numeric column:

print(sales.nlargest(3, "revenue"))

Joining tables

Business data is often split across files. Use merge() to combine tables by a shared key:

products = pd.DataFrame({
    "product_id": [1, 2],
    "category": ["Computers", "Accessories"],
})

orders = pd.DataFrame({
    "product_id": [1, 2, 2],
    "quantity": [1, 3, 2],
})

combined = orders.merge(products, on="product_id", how="left")

Check whether keys are unique before joining, because duplicate keys can unexpectedly multiply rows.

Complete practical workflow

import numpy as np
import pandas as pd

sales = pd.read_csv("sales.csv")

sales.columns = sales.columns.str.strip().str.lower()
sales["quantity"] = pd.to_numeric(sales["quantity"], errors="coerce")
sales["unit_price"] = pd.to_numeric(sales["unit_price"], errors="coerce")
sales = sales.dropna(subset=["product", "quantity", "unit_price"])

sales["revenue"] = sales["quantity"] * sales["unit_price"]
sales["performance"] = np.where(
    sales["revenue"] >= sales["revenue"].median(),
    "above median",
    "below median",
)

report = (
    sales.groupby("product", as_index=False)
    .agg(
        units=("quantity", "sum"),
        revenue=("revenue", "sum"),
    )
    .sort_values("revenue", ascending=False)
)

report.to_csv("sales_report.csv", index=False)
print(report)

This script loads, standardizes, converts, cleans, calculates, groups, sorts, and exports data. It represents the basic structure of many real analysis projects.

Visualizing the result

Pandas integrates with Matplotlib:

report.plot(
    x="product",
    y="revenue",
    kind="bar",
    title="Revenue by Product",
)

The Matplotlib beginner guide explains labels, legends, chart types, and image export.

Common beginner mistakes

• Analyzing data before checking types and missing values.
• Using Python loops for operations that Pandas or NumPy can vectorize.
• Overwriting the original file before validating results.
• Assuming every column name is clean and consistent.
• Using chained assignment instead of explicit loc.
• Joining tables without checking duplicate keys.
• Drawing conclusions from a small or biased dataset.

Best practices

Keep raw data unchanged, save cleaned data separately, write reusable functions, document assumptions, and validate totals at each stage. For larger projects, place loading, cleaning, analysis, and reporting in separate modules. The guide to modules and packages shows how to organize this code.

The official Pandas introductory tutorials provide additional exercises with real tables.

Conclusion

NumPy and Pandas cover complementary parts of Python data analysis. NumPy provides efficient numerical arrays; Pandas provides labeled tables and practical cleaning, grouping, merging, and export tools.

Begin with a small CSV, inspect every column, clean obvious problems, create one or two calculated fields, summarize the data, and export a report. Repeating that workflow builds the foundation for visualization, statistics, and machine learning.