Prophecy integrates with open-source Airflow deployments via an uploader service, which securely transfers pipeline artifacts, DAGs, and datasets into an Airflow instance for orchestration and execution. This setup lets you trigger and monitor Airflow DAGs as part of your data workflows.
You'll need two sets of information to proceed:
- Airflow credentials allow Prophecy to access the Airflow instance itself (so that Prophecy can connect, trigger, and monitor DAGs).
- Uploader service credentials identify and authorize access to the location (HDFS or file system) where DAGs and datasets are stored before being added to Airflow.
To begin configuring an open-source Airflow connection:
- Follow steps to begin creating a Fabric.
- On the Providers page, select Airflow for Provider Type.
- Select Open Source for Provider, then fill in the fields described below.
Airflow Access
Under Airflow Access, you enter the URL and access credentials for your Airflow instance.
| Parameter | Description |
|---|---|
| URL | URL for Airflow instance. |
| Username | Username for Airflow instance. |
| Password | Password for Airflow instance. |
Uploader Service
Prophecy provides an uploader service for on-premises deployments of Airflow.
The uploader service URL and credentials secure the upload channel, the DAG location defines the orchestration logic, the artifact location points to the actual dataset being ingested, and the version ensures compatibility with the target Airflow instance.
| Parameter | Description |
|---|---|
| URL | URL endpoint for the uploader service on the master node of the Hadoop deployment where Airflow instance is located. |
| Username | Username for uploader service. |
| Password | Password for uploader service. |
| DAG Location | Path to the DAG file location on file system (HDFS or local) |
| Artifact Location | Path to the dataset or artifact to be uploaded (also HDFS or local) |
| Version | Airflow version you are using (such as 2.5.x, 2.6.x). |
Runtime Sequence
- Prophecy authenticates with Airflow Fabric.
- Files or datasets are uploaded via the uploader service.
- The associated DAG is registered or triggered.
- Execution logs and status are reported back to Prophecy.