Drop Rows with NaN Values in a Pandas DataFrame

You will learn how Pandas represents missing data with NaN, how to remove rows based on missing values in a particular column, and when boolean filtering is a useful alternative to dropna().

In Pandas, NaN means Not a Number and is commonly used to represent a missing value in numeric columns. A DataFrame can have missing values in some columns while still containing useful values in others.

To remove rows only when a particular column is missing, use DataFrame.dropna() with its subset argument:

df = df.dropna(subset=["EPS"])

subset=["EPS"] tells Pandas to inspect only the EPS column. A row is removed if EPS is missing, even if values in other columns such as cash are present or missing.

This distinction matters because calling df.dropna() without options is more aggressive: by default, it removes a row when any column in that row contains a missing value. In the example, both remaining rows have a missing cash value, so a plain dropna() would remove them too.

The DataFrame in the question uses a MultiIndex (STK_ID and RPT_Date). That does not change how dropna(subset=["EPS"]) works: EPS is still a regular data column.

Think of each DataFrame row as a form and each column as a field on that form.

You need forms that contain an EPS value. It does not matter whether the cash field is blank. Using:

df.dropna(subset=["EPS"])

means: “Discard a form only if its EPS field is blank.”

By contrast, plain df.dropna() means: “Discard a form if any field is blank.”

Use dropna() with subset to remove rows where specified columns have missing values.

cleaned = df.dropna(subset=["EPS"])

For the DataFrame in the question, cleaned contains rows whose EPS values are 4.3 and 2.5.

print(cleaned)

                 STK_ID  EPS  cash
STK_ID RPT_Date
600016 20111231  600016  4.3   NaN
601939 20111231  601939  2.5   NaN

You can check more than one required column:

complete_financials = df.dropna(subset=["EPS", "cash"])

This keeps a row only when both EPS and cash are present.

A boolean-filtering alternative is:

cleaned = df[df["EPS"].notna()]

df["EPS"].notna() produces True for present values and False for missing values. Pandas keeps the rows marked True.

Consider this small DataFrame:

import pandas as pd

sales = pd.DataFrame({
    "product": ["Notebook", "Pen", "Bag"],
    "price": [4.50, None, 25.00],
    "stock": [12, 30, None]
})

available_prices = sales.dropna(subset=["price"])

Step by step:

1. sales has three rows.
2. Pandas interprets None in the numeric price column as a missing value (NaN).
3. dropna(subset=["price"]) examines price, not every column.
4. The Pen row has a missing price, so Pandas removes it.
5. The Bag row remains because its price is present, even though stock is missing.

The result is:

    product  price  stock
0  Notebook    4.5   12.0
2       Bag   25.0    NaN

The original sales DataFrame is unchanged because dropna() returns a new DataFrame unless you explicitly assign the result or use inplace=True.

Removing rows based on a required field is common when cleaning data before analysis or application logic.

• Financial reports: Keep records that have an earnings-per-share value before calculating averages or rankings.
• CSV imports: Drop rows without a required customer ID, email address, or transaction amount.
• API data: Ignore records that do not contain a required timestamp or status field.
• Machine learning preparation: Remove training examples that do not have a target value.
• Data exports: Keep orders only when a valid order total is available.

Choose the subset columns based on what your next operation requires. A missing optional field should usually not cause an otherwise useful row to be discarded.

In real projects, developers usually avoid mutating raw imported data immediately. They often keep the raw DataFrame and create a clearly named cleaned version:

valid_eps_records = raw_financials.dropna(subset=["EPS"])

For a data pipeline, validate required columns before processing:

required_columns = ["STK_ID", "EPS"]
missing_columns = set(required_columns) - set(df.columns)

if missing_columns:
    raise ValueError(f"Missing columns: {sorted(missing_columns)}")

valid_records = df.dropna(subset=required_columns)

When missing rows should be investigated rather than silently discarded, split valid and invalid records:

invalid_eps_records = df[df["EPS"].isna()]
valid_eps_records = df[df["EPS"].notna()]

This pattern supports logging, error reports, and data-quality dashboards. Use dropna() when removal is truly the intended business rule; otherwise, consider filling missing values or flagging them for review.

Calling dropna() without subset

# Too broad for the stated goal
df.dropna()

This removes rows with a missing value in any column. In the original example, it would remove the rows with valid EPS values because cash is missing.

Use:

df.dropna(subset=["EPS"])

Comparing directly to NaN

# Incorrect
df[df["EPS"] != float("nan")]

NaN does not compare equal to itself, so direct equality and inequality checks are unreliable for detecting missing values.

Use Pandas methods instead:

df[df["EPS"].notna()]
df[df["EPS"].isna()]

Forgetting to save the result

df.dropna(subset=["EPS"])
print(df)  # df is unchanged

Assign the returned DataFrame:

df = df.dropna(subset=["EPS"])

Using an index level as though it were a column

In a MultiIndex DataFrame, STK_ID may be both an index level and a column in some datasets. subset accepts column labels, not index-level names. To filter on an index level, use index-based logic instead.

Using inplace=True by default

df.dropna(subset=["EPS"], inplace=True)

This can be valid, but it changes df directly and makes it harder to retain the original data. Prefer assignment unless mutation is intentional.

dropna(subset=["EPS"]) and df[df["EPS"].notna()] generally produce the same rows for this task. dropna() is especially convenient when checking one or several required columns; boolean filtering is flexible when conditions become more complex:

filtered = df[df["EPS"].notna() & (df["EPS"] > 0)]

# Remove rows where EPS is missing
df = df.dropna(subset=["EPS"])

# Keep rows where EPS is present
df = df[df["EPS"].notna()]

# Select rows where EPS is missing
eps_missing = df[df["EPS"].isna()]

# Require multiple columns to be present
df = df.dropna(subset=["EPS", "cash"])

# Remove a row only if all listed columns are missing
df = df.dropna(subset=["EPS", "cash"], how="all")

# Change the existing DataFrame directly
df.dropna(subset=["EPS"], inplace=True)

Key rules:

• subset contains column names.
• Default behavior is how="any": remove a row if any selected column is missing.
• NaN, None, and NaT are typically treated as missing by Pandas.
• Prefer .isna() and .notna() over comparisons with NaN.

How do I drop rows with NaN in only one Pandas column?

Use df.dropna(subset=["column_name"]). For example: df.dropna(subset=["EPS"]).

Does dropna(subset=["EPS"]) remove rows with NaN in other columns?

No. It checks only EPS. A row with a valid EPS and a missing cash value stays in the result.

Why does df.dropna() remove too many rows?

Without subset, Pandas checks every column. Any missing value in a row causes that row to be removed by default.

How do I keep NaN rows instead of removing them?

Filter with isna():

missing_eps = df[df["EPS"].isna()]

Can I remove rows where either of two columns is missing?

Yes:

df.dropna(subset=["EPS", "cash"])

This removes a row if EPS or cash is missing.

Can I remove rows only when both EPS and cash are missing?

Yes. Use how="all":

df.dropna(subset=["EPS", "cash"], how="all")

Does this work with a MultiIndex DataFrame?

Yes. dropna(subset=["EPS"]) works normally when EPS is a data column, regardless of whether the DataFrame has a MultiIndex.

• Pandas missing data — Learn how NaN, None, and NaT represent absent values.
• isna() and notna() — Build explicit boolean masks for missing-data filtering.
• Boolean indexing in Pandas — Select rows using one or more true/false conditions.
• fillna() — Replace missing values when removal is not appropriate.
• Pandas DataFrame indexing — Understand the difference between columns and index levels, including MultiIndex data.
• Data validation — Check required fields before analysis, exports, or database writes.