Skip to content
All projects

Batch pipeline

NYC Taxi Batch Pipeline

Planned

Overview

A planned batch pipeline for the public NYC TLC taxi trip records: monthly Parquet files cleaned with PySpark, modeled into a star schema, and scheduled with Apache Airflow.

Problem statement

The NYC Taxi & Limousine Commission publishes every yellow-taxi trip as monthly Parquet files: tens of millions of rows a year, with missing values, outliers and schema changes between years. The aim is a repeatable pipeline that turns them into tables an analyst can query directly.

Architecture

Planned design: an Airflow DAG downloads each new month's file, PySpark validates and cleans it (dropping impossible fares and trip times, joining the taxi zone lookup), and the result is written as a fact table of trips with date, zone and payment dimensions.

Technologies

  • Python
  • PySpark
  • Apache Airflow
  • SQL