Adding, Removing, Modifying Columns¶

The most common transformation: derive new columns from existing ones.

Add a column — simple assignment¶

import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol"],
    "salary": [50000, 60000, 75000],
})

# Constant value for every row
df["country"] = "India"

# Computed from another column
df["bonus"] = df["salary"] * 0.10

# From a list (must match length)
df["age"] = [25, 30, 35]

print(df)

Multiple new columns at once¶

import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob", "Carol"],
    "salary": [50000, 60000, 75000],
})

df[["bonus", "total"]] = pd.DataFrame({
    "bonus": df["salary"] * 0.10,
    "total": df["salary"] * 1.10,
})
print(df)

`.assign()` — chainable, doesn't modify original¶

import pandas as pd

df = pd.DataFrame({"a": [1, 2, 3]})

# Returns a new DataFrame with extra columns
new = df.assign(
    b = lambda d: d["a"] * 2,
    c = lambda d: d["a"] ** 2,
)
print("Original (unchanged):")
print(df)
print("\nWith assigns:")
print(new)

Great for method chaining:

import pandas as pd

df = pd.DataFrame({"x": [1, 2, 3, 4, 5]})

result = (
    df
    .assign(square=lambda d: d["x"] ** 2)
    .assign(cube=lambda d: d["x"] ** 3)
    .assign(is_even=lambda d: d["x"] % 2 == 0)
)
print(result)

Renaming columns¶

import pandas as pd

df = pd.DataFrame({"a": [1, 2], "b": [3, 4], "c": [5, 6]})

# Rename specific
df = df.rename(columns={"a": "alpha", "b": "beta"})
print(df)

# All at once
df.columns = ["x", "y", "z"]
print(df)

# Programmatic — lowercase, strip
df = pd.DataFrame({" Name ": [1], "AGE": [2]})
df.columns = df.columns.str.strip().str.lower()
print(df)

Removing columns — `.drop()`¶

import pandas as pd

df = pd.DataFrame({
    "name": ["Alice","Bob"],
    "age":  [25, 30],
    "city": ["Mumbai","Delhi"],
    "tmp":  [None, None],
})

# Single
df = df.drop(columns="tmp")
print(df)
print()

# Multiple
df2 = df.drop(columns=["age", "city"])
print(df2)
print()

# Original unchanged (drop returns a copy by default)
# In place:
df.drop(columns=["age"], inplace=True)
print(df)

Deleting with `del` or `pop()`¶

import pandas as pd

df = pd.DataFrame({"a": [1, 2], "b": [3, 4], "c": [5, 6]})

del df["a"]                # modifies in place
print(df)

removed = df.pop("c")      # removes AND returns the column
print("removed:", removed.tolist())
print(df)

Reordering columns¶

import pandas as pd

df = pd.DataFrame({
    "salary": [50000, 60000],
    "name":   ["Alice", "Bob"],
    "age":    [25, 30],
})

# Specify the new order
df = df[["name", "age", "salary"]]
print(df)

Modify a column¶

Just assign back to the same name:

import pandas as pd

df = pd.DataFrame({
    "name":   ["alice", "bob", "carol"],
    "salary": [50000, 60000, 75000],
})

# Transform
df["name"]   = df["name"].str.title()
df["salary"] = df["salary"] * 1.10            # 10% raise

print(df)

Conditional column — `np.where`¶

import pandas as pd
import numpy as np

df = pd.DataFrame({
    "salary": [40000, 75000, 100000, 200000],
})

df["bracket"] = np.where(df["salary"] >= 100_000, "high", "low")
print(df)

Multi-condition — `np.select`¶

import pandas as pd
import numpy as np

df = pd.DataFrame({"score": [95, 85, 72, 60, 45]})

conditions = [
    df["score"] >= 90,
    df["score"] >= 80,
    df["score"] >= 70,
    df["score"] >= 60,
]
choices = ["A", "B", "C", "D"]

df["grade"] = np.select(conditions, choices, default="F")
print(df)

`.apply()` — custom function per row or column¶

import pandas as pd

df = pd.DataFrame({
    "first": ["Alice", "Bob", "Carol"],
    "last":  ["Smith", "Jones", "Davis"],
})

# Apply to a column (Series)
df["first_upper"] = df["first"].apply(str.upper)
print(df)
print()

# Apply across rows — axis=1
df["full_name"] = df.apply(lambda r: f"{r['first']} {r['last']}", axis=1)
print(df)

Heads up: .apply() is convenient but slow on big data. Prefer vectorized operations (df.a + df.b, np.where, string .str. methods) when possible.

`.map()` — value replacement using a dict¶

import pandas as pd

df = pd.DataFrame({
    "code": ["A", "B", "A", "C", "B"],
})

names = {"A": "Alpha", "B": "Beta", "C": "Gamma"}
df["name"] = df["code"].map(names)
print(df)

String operations — `.str.`¶

import pandas as pd

df = pd.DataFrame({
    "email": ["Alice@Gmail.COM", "  Bob@example.com  ", "carol@yahoo.com"],
})

df["email"]      = df["email"].str.strip().str.lower()
df["domain"]     = df["email"].str.split("@").str[1]
df["is_gmail"]   = df["email"].str.contains("gmail")

print(df)

Datetime ops — `.dt.`¶

import pandas as pd

df = pd.DataFrame({
    "joined": pd.to_datetime([
        "2024-01-15", "2024-02-20", "2023-11-10", "2024-03-05"
    ]),
})

df["year"]    = df["joined"].dt.year
df["month"]   = df["joined"].dt.month
df["weekday"] = df["joined"].dt.day_name()

# Time since
df["days_ago"] = (pd.Timestamp("2025-01-01") - df["joined"]).dt.days

print(df)

Bin a continuous column — `pd.cut`¶

import pandas as pd

df = pd.DataFrame({"age": [10, 22, 35, 48, 67, 75, 8, 30]})

df["age_group"] = pd.cut(
    df["age"],
    bins=[0, 12, 18, 35, 60, 120],
    labels=["child", "teen", "young", "adult", "senior"],
)
print(df)

pd.qcut does the same but uses quantiles (equal-sized bins by count):

import pandas as pd
import numpy as np
rng = np.random.default_rng(0)

df = pd.DataFrame({"score": rng.integers(0, 100, size=20)})
df["quartile"] = pd.qcut(df["score"], q=4, labels=["Q1","Q2","Q3","Q4"])
print(df)

One-hot encoding — `pd.get_dummies`¶

For ML — turn a categorical column into 0/1 columns:

import pandas as pd

df = pd.DataFrame({
    "name": ["Alice","Bob","Carol","Dave"],
    "city": ["Mumbai","Delhi","Mumbai","Pune"],
})

print(pd.get_dummies(df, columns=["city"]))

Change column dtype — `.astype()`¶

import pandas as pd

df = pd.DataFrame({
    "id":     ["1", "2", "3"],            # strings — should be int
    "rating": [4.7, 3.2, 5.0],
})

print("Before:", df.dtypes.tolist())

df["id"]     = df["id"].astype(int)
df["rating"] = df["rating"].astype("float32")    # save memory

print("After :", df.dtypes.tolist())
print(df)

Inserting a column at a specific position¶

import pandas as pd

df = pd.DataFrame({
    "name":   ["Alice", "Bob"],
    "salary": [50000, 60000],
})

df.insert(1, "age", [25, 30])      # insert at position 1
print(df)

Cheatsheet¶

Want	Code
Add column	`df["new"] = ...`
Add multiple (chainable)	`df.assign(a=..., b=...)`
Drop columns	`df.drop(columns=["a", "b"])`
Delete in place	`del df["col"]` or `df.pop("col")`
Rename	`df.rename(columns={"a": "x"})`
Reorder	`df = df[["c1", "c2", ...]]`
Modify	`df["c"] = transform(df["c"])`
Conditional value	`np.where(cond, x, y)`
Multi-condition	`np.select([c1, c2], [v1, v2], default=...)`
Map dict	`df["c"].map({...})`
Bin numeric	`pd.cut(df["c"], bins=[...])`
Quartile bin	`pd.qcut(df["c"], q=4)`
One-hot	`pd.get_dummies(df, columns=[...])`
String op	`df["c"].str.lower()` etc
Datetime op	`df["c"].dt.year` etc
Change dtype	`df["c"].astype(...)`
Apply function	`df["c"].apply(fn)` — slow!

Common pitfalls¶

❗ df["new"] = some_list of wrong length — raises ValueError. Make sure lengths match (or use a scalar).
❗ SettingWithCopyWarning — happens when you assign to a chained selection. Always use .loc[mask, "col"] = value.
❗ .apply() is slow — for 1M rows, prefer vectorized ops. .apply() is fine for small data or complex logic.
❗ map() doesn't have a default — missing keys → NaN. Pass .fillna(...) after if needed.
❗ get_dummies() creates many columns — for high-cardinality columns this can explode. See ML Encoding.

Practice¶

What does this print?

Expected: [55000.0, 66000.0, 82500.0]

import pandas as pd
df = pd.DataFrame({"salary": [50000, 60000, 75000]})
df["new"] = df["salary"] * 1.10
print(df["new"].tolist())

Add email_domain extracted from the email (just the part after @)

Expected: gmail.com

import pandas as pd
df = pd.DataFrame({"email": ["alice@gmail.com", "bob@yahoo.com"]})
df["domain"] = df["email"].apply(lambda e: e.split("@"))   # bug: returns the whole list, not the domain
print(df.loc[0, "domain"])

Quiz — Quick check¶

What you remember

Q1. Does df.assign(new_col=...) modify the original DataFrame?

Yes
No — it returns a new DataFrame with the extra column
Sometimes
Only if inplace=True

Why: .assign() is chainable and immutable. Use it for building pipelines without side effects. If you want in-place: df["new_col"] = ... (direct assignment).

Q2. What's the FASTEST way to apply a value-replacement dict like {"A": "Alpha", "B": "Beta"}?

df.apply(lambda r: dict[r.code])
df["code"].map({"A": "Alpha", "B": "Beta"})
A Python for loop
df.replace(dict)

Why: .map() is dedicated to value mapping — vectorized and fast. .apply() calls the function once per row in Python (slow). .replace() is general-purpose, slower for simple cases.

Q3. What does pd.get_dummies(df, columns=["city"]) produce?

A pivoted DataFrame
One-hot encoded columns: a new 0/1 column per unique value of city
A frequency count
A summary table

Why: get_dummies turns categorical into numeric — ML algorithms need numbers. For high-cardinality columns (1000s of unique values), this can explode. Consider target encoding or hashing instead.

Common doubts¶

When is .apply() actually OK to use?

For small DataFrames (under 100K rows) or when the logic is too complex for vectorized ops — like calling an external API per row, or applying logic that depends on multiple columns in a non-trivial way. For 1M+ rows, prefer vectorized alternatives even if they require more code.

Why does .map() give NaN for some values?

Because the dict doesn't have a key for them. .map({"A": "Alpha"}).fillna("Unknown") is the idiom for "map known values, fall back for the rest".

Should I use np.where or df["col"].mask()?

For setting a value based on a condition, np.where(cond, x, y) is the most common idiom (works on columns and arrays). df.mask(cond) keeps the original where the condition is FALSE — the opposite logic. Both work; np.where is more widely used.

What's next¶

→ Handling Missing Data

Adding, Removing, Modifying Columns¶

Add a column — simple assignment¶

Multiple new columns at once¶

.assign() — chainable, doesn't modify original¶

Renaming columns¶

Removing columns — .drop()¶

Deleting with del or pop()¶