NYC Taxi Batch Pipeline
Planned
Overview
A planned batch pipeline for the public NYC TLC taxi trip records: monthly Parquet files cleaned with PySpark, modeled into a star schema, and scheduled with Apache Airflow.
Problem statement
The NYC Taxi & Limousine Commission publishes every yellow-taxi trip as monthly Parquet files: tens of millions of rows a year, with missing values, outliers and schema changes between years. The aim is a repeatable pipeline that turns them into tables an analyst can query directly.
Architecture
Planned design: an Airflow DAG downloads each new month's file, PySpark validates and cleans it (dropping impossible fares and trip times, joining the taxi zone lookup), and the result is written as a fact table of trips with date, zone and payment dimensions.
Technologies
Python
PySpark
Apache Airflow
- SQL