Project Overview
A Python ETL pipeline that extracts Adidas sales data from Excel, applies business transformations, and loads the clean data into a PostgreSQL database — with full logging throughout.
Pipeline Steps
- Configuration: Reads a config file specifying which columns to extract, transformation rules, and database credentials
- Extract: Loads specific columns from an Excel file using openpyxl/pandas
- Transform: Rounds numeric values to 2 decimal places, standardizes date formats, handles missing values
- Load: Writes the transformed DataFrame to a PostgreSQL table using SQLAlchemy
- Logging: Structured logs track each pipeline stage — errors, row counts, and duration
Design Decisions
Config-driven design means the same pipeline can be reused for different data sources by changing a single file. Logging ensures the pipeline is observable and debuggable in production.
Outcome
A production-ready, config-driven ETL script that can be scheduled (e.g., via cron or Airflow) to run automatically on new data drops.