Automating a recurring data job

The last step is turning the analysis into a script that runs on a schedule, logs what it did, and fails loudly.

Lesson 4.311m 30sBeginner13.6k students

Chapters

11:30
  1. 0:00Introduction to Airflow
  2. 2:49The Setup
  3. 3:40Script ETL pipeline: Extract
  4. 5:52Transform
  5. 7:39Load
  6. 8:00Define Direct Acyclic Graph (DAG
  7. 9:36Airflow UI: Dag enable & run
  8. 10:09DAG Overview
  9. 10:29Test ETL Pipeline

What this lesson covers

  • Turn a notebook into a parameterised script
  • Log progress and failures usefully
  • Schedule the job and alert when it breaks

This lesson sits in Reproducible Workflows, part of Python for Data Work. It assumes what came before it and leads directly into the next lesson in the module.

In this lesson you will

  • Turn a notebook into a parameterised script
  • Log progress and failures usefully
  • Schedule the job and alert when it breaks

Resources