Pandera Validation Skill

Pandera Validation

Audience: Data engineers validating pandas DataFrames.

Goal: Provide pandera patterns for schema validation and type checking.

Scripts

Execute schema functions from scripts/schemas.py:

from scripts.schemas import (
    create_user_schema,
    create_nullable_schema,
    create_date_range_schema,
    UserSchema,
    validate_with_errors,
    infer_and_export_schema
)

Usage Examples

Basic Schema Validation

from scripts.schemas import create_user_schema

schema = create_user_schema()
validated_df = schema.validate(df)

Collect All Errors

from scripts.schemas import create_user_schema, validate_with_errors

schema = create_user_schema()
validated_df, errors = validate_with_errors(df, schema)

if errors:
    for err in errors:
        print(f"{err['column']}: {err['check']} - {err['failure_case']}")

Class-Based Schema

from scripts.schemas import UserSchema

# Validate with type hints
UserSchema.validate(df)

# Use as function type hint
def process_users(df: pa.typing.DataFrame[UserSchema]) -> pd.DataFrame:
    return df.query("status == 'active'")

Infer Schema from DataFrame

from scripts.schemas import infer_and_export_schema

schema_export = infer_and_export_schema(df)
print(schema_export['python_code'])  # Python schema definition
print(schema_export['yaml'])         # YAML schema

Built-in Checks Reference

| Check Type | Example | Description | |------------|---------|-------------| | Numeric | Check.gt(0), Check.in_range(0, 100) | Comparisons | | String | Check.str_matches(r'pattern') | Regex match | | Set membership | Check.isin(['A', 'B']) | Allowed values | | Uniqueness | unique=True on Column | No duplicates | | Nullable | nullable=True on Column | Allow nulls |

Decorator-Based Validation

import pandera as pa

@pa.check_output(schema)
def load_data(path: str) -> pd.DataFrame:
    return pd.read_csv(path)

@pa.check_input(schema, "df")
def process_data(df: pd.DataFrame) -> pd.DataFrame:
    return df.assign(processed=True)

@pa.check_io(df=input_schema, out=output_schema)
def transform_data(df: pd.DataFrame) -> pd.DataFrame:
    return df.transform(...)

When to Use Pandera

| Use Case | Pandera | Alternative | |----------|---------|-------------| | DataFrame validation | ✓ | - | | Type hints for DataFrames | ✓ | - | | ETL pipeline checks | ✓ | Great Expectations | | Record-level validation | - | Pydantic |

Dependencies

pandera>=0.18
pandas

Agent Skills: Pandera Validation

Install this agent skill to your local

Skill Files