From race data to queryable telemetry.
An end-to-end Formula 1 data pipeline: OpenF1 data enters Kafka, is persisted in PostgreSQL, and is processed with Spark into Cassandra for driver- and session-oriented lookups.
In progress: orchestration and publication
The problem
Build a platform, not just a chart.
The goal was to build the moving parts of a data platform: ingest a continuous feed, keep the raw stream, serve two different access patterns, and schedule the pipeline. Formula 1 offers realistic, time-ordered data in a domain I care about.
Built with OpenF1 data from the 2024 season. This is a streaming architecture; live-session latency has not been benchmarked.
Architecture
One stream, two serving paths.
Kafka decouples ingestion from the relational and streaming consumers.
OpenF1 APIRace data source
producer.pyKafka producer
Kafka · KRaftf1_lap_data topic
consumer.py → PostgreSQL · f1_db
Relational copy for ad hoc SQL queries and analysis.
Spark streaming → Cassandra · f1_data
Two query-shaped tables, lap_telemetry and race_gaps, for fast driver and session reads. Writes are batched in groups of 50.
Batch processing: Kafka decouples the producer from both consumers so either side can catch up independently. Airflow orchestration is in progress; Superset and Grafana are planned.
Key decisions
Designed around local constraints.
Kafka in KRaft mode
Removes ZooKeeper as a service, reducing the stack footprint on an 8 GB laptop.
Two sinks, two access patterns
PostgreSQL supports flexible SQL; Cassandra is shaped for fast key-based reads.
Batch Cassandra writes
Writes 50 rows per batch instead of making one round trip per row.
Docker Compose stack
Uses free, open-source tools and a custom Airflow image to include pipeline dependencies.
What is built
Working stages and open work.
Ingestion
OpenF1 API → Kafka → PostgreSQL.
Processing
Kafka → Spark → Cassandra.
Orchestration
Airflow DAG.
Dashboards
Superset and Grafana.
Publishing
Free-tier cloud deployment.
Results
What has been recorded.
The current stored dataset contains 30 sessions, 99 drivers, and more than 4,119 laps.
Throughput
Throughput benchmarking is planned.
End-to-end delay
Producer-to-Cassandra delay has not been benchmarked.
Cross-sink reconciliation
Cross-sink count and duplicate checks are planned.
Benchmarks are the next step: this page shows the architecture and the data it stores. It makes no throughput or latency claims.
What broke, and how it was fixed
Resource and runtime constraints shaped the pipeline.
Python 3.12 compatibility
kafka-python failed on Python 3.12; switched to kafka-python-ng==2.2.3.
Consumer polling stalled
Replaced a for-loop consumer with an explicit poll loop.
Cassandra memory pressure
Set a 1,500 MB container limit and a 768 MB heap to stabilize it.
Full-stack laptop memory
Raised the WSL2 limit to 6 GB and treated memory as a design constraint.
Airflow dependencies
The base image lacks pipeline packages; a custom image based on apache/airflow:2.9.3 adds them. A custom image is the planned fix.
Limitations and next steps
A local demonstration, not production resilience.
- Kafka and Cassandra are single-node deployments.
- Duplicate handling and replay behavior have not been tested; no delivery guarantee is claimed.
- Dashboards are not available yet.
- The stack runs on a laptop and is not deployed in the cloud.
Next
- Finish Airflow orchestration and schedule the pipeline.
- Add Superset and Grafana dashboards.
- Reconcile PostgreSQL against Cassandra, following the approach used in DataGuard.
- Add producer tests and data-quality checks.
- Publish the repository with an architecture diagram and screenshots.
Run it
Local Docker Compose workflow.
- Start Docker, then run:
docker compose -f docker/docker-compose.yml up -d
- Wait about 90 seconds for Cassandra, then check
docker ps. - Run
producer.py, thenconsumer.py, then submitprocessing/spark_streaming.pywith the Kafka package flags from the project notes.