Apache Airflow is an open-source workflow automation and orchestration platform that helps organizations schedule, monitor, and manage complex data workflows. It is a popular choice for organizations of all sizes to handle data pipelines, ETL processes, and task automation.
Features
Feature
Description
Directed acyclic graphs (DAGs)
Each job in Prophecy represents an Airflow DAG, and contains gems that represent various tasks.
Dynamic workflow scheduling
In the job settings, you can configure complex dependencies, retries, and time-based triggers for tasks.
Extensibility
Custom gems can be written and used within Airflow jobs. These gems can be shared via the Package Hub.
Monitoring and logging
Task progress, logging, and alerting can be monitored from the Airflow environment or the Prophecy UI.
Parallel execution and scalability
Airflow can execute multiple tasks in parallel and scale horizontally to accommodate high workloads.
Visual and code views
Visually-developed Airflow jobs are automatically compiled into high-quality and open-source Airflow code.
Airflow Gems
Prophecy provides a set of gems that you can use in your Airflow jobs. They are listed below and grouped by category.
Sensor
Gem
Description
EMRClusterSensor
Waits for a specific EMR cluster state
EMRPipelineSensor
Waits for a specific EMR pipeline state
HTTPSensor
Waits for a specific HTTP endpoint
S3FileSensor
Waits for a specific S3 file
SFTPSensor
Waits for a specific file on an SFTP path
Data Transfer
Gem
Description
SFTPToS3
Transfer files from an SFTP server to Amazon S3
SFTPToSnowflake
Copies data from SFTP to Snowflake and creates a table
TableauExtract
Generates Tableau Hyper files
Spark/SQL
Gem
Description
DatabricksPipeline
Runs a Databricks pipeline (Spark or SQL)
DataprocPipeline
Runs a pipeline on Dataproc
DataprocServerlessPipeline
Runs a pipeline on Dataproc serverless
DataprocTesting
Runs a pipeline on Dataproc
EMRCreateCluster
Creates an EMR cluster
EMRPipeline
Runs an pipeline on EMR
Model
Runs a SQL model
OnPremPipeline
Runs a pipeline on a Hadoop cluster
SnowflakeSQL
Runs a SQL query on Snowflake
Trigger/Notify
Gem
Description
Email
Sends a notification via email
Slack
Sends a notification via Slack
TriggerDag
Trigger another Airflow job via its name or DAG ID
Custom
Gem
Description
Branch
Insert a Python lambda or function
Python
Insert Python code
Script
Insert a Shell script
Subgraph
Gem
Description
ForEachLoop
Iterate through a set of gems depending on code, a JSON list, or gem output