Inspecting Data¶

Once you've loaded a DataFrame, the first thing to do is look at it. These methods answer: "What's the shape? What types? How much is missing? What does it look like?"

Set up — a sample dataset¶

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)

df = pd.DataFrame({
    "name":     ["Alice", "Bob", "Carol", "Dave", "Eve", "Frank", "Grace"],
    "age":      [25, 30, 35, 40, 45, 50, np.nan],
    "city":     ["Mumbai", "Delhi", "Mumbai", "Pune", "Delhi", "Mumbai", "Pune"],
    "salary":   [50000, 60000, 75000, 90000, 100000, 110000, 65000],
    "joined":   pd.to_datetime(["2024-01-15", "2024-02-20", "2023-11-10",
                                 "2024-03-15", "2023-08-01", "2024-05-20", "2024-04-10"]),
    "active":   [True, True, False, True, True, False, True],
})
print(df)

We'll use this df for every example below.

`df.shape` — rows and columns¶

import pandas as pd
import numpy as np
rng = np.random.default_rng(0)
df = pd.DataFrame({
    "name": ["Alice","Bob","Carol","Dave","Eve"],
    "age": [25,30,35,40,45],
    "city": ["Mumbai","Delhi","Mumbai","Pune","Delhi"],
})

print(df.shape)          # (5, 3)
print(len(df))           # 5  — same as shape[0]
print(df.columns.tolist())

`df.head()` / `df.tail()` — first / last rows¶

import pandas as pd
import numpy as np
rng = np.random.default_rng(0)

df = pd.DataFrame({
    "x": range(20),
    "y": rng.integers(0, 100, size=20),
})

print(df.head())          # first 5 (default)
print()
print(df.head(3))         # first 3
print()
print(df.tail(2))         # last 2

df.sample(n) — pick n random rows (great for big datasets):

import pandas as pd
import numpy as np

df = pd.DataFrame({"x": range(100), "y": np.random.default_rng(0).random(100)})
print(df.sample(5, random_state=42))

`df.info()` — types, non-null counts, memory¶

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol"],
    "age":    [25, 30, np.nan],
    "salary": [50000, 60000, 75000],
    "active": [True, False, True],
})

df.info()

Reads like a quick-glance summary: - How many rows and columns - Column names + dtypes - How many non-null values per column - Memory usage

`df.describe()` — descriptive statistics¶

For numeric columns:

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "age":    [25, 30, 35, 40, 45, 50, np.nan],
    "salary": [50000, 60000, 75000, 90000, 100000, 110000, 65000],
})

print(df.describe())

Reports: count, mean, std, min, quartiles, max.

For all columns (including non-numeric):

import pandas as pd

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Carol", "Alice"],
    "age":  [25, 30, 35, 25],
    "city": ["Mumbai", "Delhi", "Mumbai", "Mumbai"],
})

print(df.describe(include="all"))

For non-numeric: count, unique, top (most frequent), freq.

`df.dtypes` — types of each column¶

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":   ["Alice", "Bob"],
    "age":    [25, 30],
    "salary": [50000.0, 60000.0],
    "joined": pd.to_datetime(["2024-01-15", "2024-02-20"]),
    "active": [True, False],
})

print(df.dtypes)

Common dtypes: - int64, int32 — integers - float64, float32 — decimals - bool — True/False - object — strings (and mixed) - datetime64[ns] — dates - category — efficient categorical data - timedelta64[ns] — time differences

`df.columns` and `df.index`¶

import pandas as pd

df = pd.DataFrame({
    "a": [1, 2, 3],
    "b": [4, 5, 6],
    "c": [7, 8, 9],
})

print("columns:", df.columns.tolist())
print("index  :", df.index.tolist())

# Rename columns
df = df.rename(columns={"a": "Alpha", "b": "Beta"})
print(df.columns.tolist())

# Rename ALL columns
df.columns = ["X", "Y", "Z"]
print(df)

`df.isna().sum()` — missing-value count per column¶

The most common first-look question — "where's the data missing?":

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol", None],
    "age":    [25, np.nan, 35, 40],
    "city":   ["Mumbai", "Delhi", np.nan, "Pune"],
    "salary": [50000, 60000, 75000, 90000],
})

print("Missing per column:")
print(df.isna().sum())
print()
print("Missing per column (%):")
print(df.isna().mean() * 100)

`df.duplicated().sum()` — duplicate rows¶

import pandas as pd

df = pd.DataFrame({
    "name": ["Alice", "Bob", "Alice", "Carol", "Bob"],
    "age":  [25, 30, 25, 35, 30],
})

print("duplicate rows:", df.duplicated().sum())
print("duplicates only:")
print(df[df.duplicated()])

# Drop them
print("\nafter drop_duplicates:")
print(df.drop_duplicates())

`df.value_counts()` — frequency table¶

import pandas as pd

df = pd.DataFrame({
    "city": ["Mumbai","Delhi","Mumbai","Pune","Delhi","Mumbai","Bangalore"],
    "role": ["dev","dev","manager","dev","manager","dev","dev"],
})

# Series.value_counts
print("by city:")
print(df["city"].value_counts())

print("\nby city (proportions):")
print(df["city"].value_counts(normalize=True))

# Multi-column
print("\nby (city, role):")
print(df.value_counts(["city", "role"]))

`df["col"].unique()` — distinct values¶

import pandas as pd

df = pd.DataFrame({"city": ["Mumbai","Delhi","Mumbai","Pune","Delhi"]})

print("unique:", df["city"].unique())
print("nunique:", df["city"].nunique())

A full first-look checklist¶

import pandas as pd
import numpy as np

rng = np.random.default_rng(0)
df = pd.DataFrame({
    "name":   ["Alice","Bob","Carol","Dave","Eve","Frank","Grace","Henry"],
    "age":    [25, 30, 35, 40, 45, 50, np.nan, 28],
    "city":   ["Mumbai","Delhi","Mumbai","Pune","Delhi","Mumbai","Pune","Mumbai"],
    "salary": rng.integers(40000, 120000, size=8),
})

print("=" * 50)
print("SHAPE:", df.shape)
print()
print("=" * 50)
print("DTYPES:")
print(df.dtypes)
print()
print("=" * 50)
print("FIRST 5 ROWS:")
print(df.head())
print()
print("=" * 50)
print("MISSING:")
print(df.isna().sum())
print()
print("=" * 50)
print("DUPLICATES:", df.duplicated().sum())
print()
print("=" * 50)
print("DESCRIBE (numeric):")
print(df.describe())
print()
print("=" * 50)
print("VALUE COUNTS:")
print(df["city"].value_counts())

This 6-step routine is what every data scientist runs first on a new dataset.

Cheatsheet¶

Want	Method
Dimensions	`df.shape`
Column names	`df.columns`
Index labels	`df.index`
First rows	`df.head(n)`
Last rows	`df.tail(n)`
Random sample	`df.sample(n)`
Types + non-null counts	`df.info()`
Numeric summary	`df.describe()`
Everything summary	`df.describe(include="all")`
Per-column dtype	`df.dtypes`
Memory	`df.memory_usage(deep=True)`
Missing per col	`df.isna().sum()`
Duplicate rows	`df.duplicated().sum()`
Frequency table	`df["col"].value_counts()`
Unique values	`df["col"].unique()`
Count of uniques	`df["col"].nunique()`
Correlation matrix	`df.corr(numeric_only=True)`

Common pitfalls¶

❗ df.info() prints, doesn't return — you can't capture it as a variable.
❗ describe() ignores non-numeric by default — pass include="all" for everything.
❗ isna() vs isnull() — they're identical aliases. Use isna() (matches NumPy).
❗ object dtype — usually means strings, but can also mean mixed-types. Worth converting to proper dtypes (string, category, datetime) for memory + speed.
❗ Counting rows: len(df) vs df.shape[0] — both work, both fast.

Practice¶

What does this print?

Expected: 2

import pandas as pd
import numpy as np
df = pd.DataFrame({"x": [1, np.nan, 3, np.nan, 5]})
print(df["x"].isna().sum())

Get a descriptive summary that INCLUDES the non-numeric 'city' column

Expected: count

import pandas as pd
df = pd.DataFrame({"city": ["Mumbai","Delhi","Mumbai"], "age": [25, 30, 35]})
print(df.describe().index[0])      # bug: default describe skips object columns

Quiz — Quick check¶

What you remember

Q1. What does df.isna().sum() return?

Total number of missing cells across the whole DataFrame
A Series with the count of missing values per column
Boolean indicating whether any value is missing
The mean of missing values

Why: df.isna() produces a same-shape boolean DataFrame. .sum() defaults to axis=0 (collapse rows → per-column counts). For a single total: df.isna().sum().sum().

Q2. Which method gives a 5-number numeric summary (count, mean, std, min, max, quartiles)?

df.info()
df.head()
df.describe()
df.dtypes

Why: describe() is the go-to numeric profile. Pass include="all" to also summarize non-numeric columns (gets unique, top, freq).

Q3. What does df.value_counts(["city", "role"]) produce?

The count of each city
The count of each unique (city, role) combination, sorted descending
An error
A pivot table

Why: Passing a list to value_counts does a multi-column frequency table. It's a fast way to see "what combinations are most common?"

Common doubts¶

Why does df.info() not return anything I can capture?

info() prints directly to stdout — it returns None. To capture, use io.StringIO:

import io
buf = io.StringIO(); df.info(buf=buf); s = buf.getvalue()

What's the difference between df.isna() and df.isnull()?

Identical aliases. isna() is preferred because it matches NumPy's naming (np.isnan). Same for notna()/notnull().

Why does df.describe() skip my 'date' column?

By default describe() only summarizes numeric columns. For datetime, it does summarize them — but with different stats (min/max/count). For object columns, pass include="all" or include=[object].

What's next¶

→ Selecting Columns & Rows

Inspecting Data¶

Set up — a sample dataset¶

df.shape — rows and columns¶

df.head() / df.tail() — first / last rows¶

df.info() — types, non-null counts, memory¶

df.describe() — descriptive statistics¶

df.dtypes — types of each column¶

df.columns and df.index¶

df.isna().sum() — missing-value count per column¶

df.duplicated().sum() — duplicate rows¶

df.value_counts() — frequency table¶

df["col"].unique() — distinct values¶