Composer

You can use Prophecy to connect to your Cloud Composer-based Airflow, to create, run, and monitor your Airflow DAGs. For this, you would need to create a Composer Airflow fabric.

How to create a Composer Airflow fabric

1. From the left navigation bar, open the Create Entity page.
2. Select to create a new Fabric.
3. On the Basic Info screen, enter a fabric name and description.
4. Choose the Team to own the fabric.
5. Click Continue.

Since we’re setting up a fabric connected to Cloud Composer Airflow:

1. Choose Airflow as the Provider Type.
2. Choose Cloud Composer as the Provider. Once you select Cloud Composer, you will start seeing fields for credentials as shown below.
3. Type in your GCP project in the Project ID field.
4. Select the Location from dropdown.
5. Upload your Private key file to authenticate.
6. Click on Fetch Environments to fetch the Airflow Instances running in the mentioned GCP project.

Environment

1. Select the desired instance of Airflow in the Airflow Environment dropdown.
2. This autofills the Airflow URL.
3. This autofills the Dag location.
4. Click Continue.

This completes the fabric creation for Composer, and you can now start adding Connections.

Setting up Connections

You need Airflow to talk to various other systems in your Data Platform to be able to do certain tasks like sending Email, triggering Spark pipelines, and SQL models. For these, we create connections in Airflow.

You can map connections already created in your Composer, in the Connections Tab of the fabric. Prophecy will use these connections to fetch the connection-id to generate the correct Airflow Code when you use these in your Airflow gems.

For adding a connection, Click on (1) Add Connection button. This Opens up the Connection form as shown.

Setting up Databricks Spark connection

To be able to schedule your Databricks Spark pipelines via Airflow, you need to have a Databricks Spark Connections from Airflow to your Databricks Workspace. You need to create the connection in Airflow and provide the mapping to Prophecy in this form.

Select (2) Connection Type as Databricks(Spark), and select the (3) Fabric you have in Prophecy for the desired Databricks Workspace. Select the (4) Connection id you have created for this Databricks workspace in your Airflow. Make sure you select the fabric for the same Databricks workspace you have already created the connection for in your Airflow. Once done, hit (4) Save.

Setting up Snowflake SQL Connection

Similarly, setup a connection to a Snowflake fabric following these steps.

Create an Airflow Job

Once the Airflow Composer fabric is setup with the relevant connections, Airflow job scheduling is done with an easy-to-use interface. Follow this guide to Create an Airflow job.