MWAA – Prophecy

You can use Prophecy to connect to your Amazon Managed Workflows for Apache Airflow (Amazon MWAA) , to create, run, and monitor your Airflow DAGs. For this, you would need to create a MWAA Airflow fabric.

How to create an MWAA Airflow fabric

From the left navigation bar, open the Create Entity page.
Select to create a new Fabric.
On the Basic Info screen, enter a fabric name and description.
Choose the Team to own the fabric.
Click Continue.

Since we’re setting up a fabric connected to MWAA Airflow:

Choose Airflow as the Provider Type.
Select MWAA as the Provider.

Once you select MWAA, you will start seeing fields for credentials. MWAA fabric offers two types of authentication.

MWAA Access key auth

By default, you would see option to provide your Access key and Secret key for authentication. Check here on how to set up and get your access and secret keys for authentication. Provide your (1) Access Key and (2) Secret Key and then select the (3) Region where your MWAA Airflow instance is running. Click on (4) Fetch environments to fetch all Airflow instances running in this region.

You can now select the Airflow environment for which you want to create the fabric in Prophecy. As you select the (1) Airflow Environment, the (2) Environment name, (3) Airflow URL and (4) DAG Location would be fetched for you. These details are only shown for verification purposes and cannot be edited. Once verified, click (5) Continue.

This completes the fabric creation and you can start setting up Connections.

MWAA IAM Role auth

If you do not wish to provide Access keys and secret keys authentication to Prophecy, you can use Assume Roles as an alternate authentication mechanism.

Please reach out to Prophecy support team to enable assume role Auth and disable Access keys authentication in your teams.

To create an AssumeRole for Prophecy, we are going to use IAM role access across Account.

Go to the Identity and Access Management (IAM) home page in your AWS account:

Go to Policies.
Click on Create policy.
We need ListEnvironments permission for Airflow, PutObject, GetObject and DeleteObject to the S3 bucket used by your Airflow for uploading DAGs and pipeline Artifacts and CreateCLiToken, GetEnvironment on the Airflow Instance. For simplicity, you can switch to JSON and use below JSON policy and replace values for Airflow Environment and S3 bucket.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": "airflow:ListEnvironments",
      "Resource": "*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:GetObject",
        "airflow:CreateCliToken",
        "s3:DeleteObject",
        "airflow:GetEnvironment"
      ],
      "Resource": [
        "arn:aws:airflow:<location>:<account id>:environment/<mwaa-environment-name>",
        "arn:aws:s3:::<s3-bucketname>/*"
      ]
    }
  ]
}

Once the permissions are selected (or provided via json), click on Next.

Review and create

Give a Policy name.
Optionally, you can add a description.
Once done, click on Create Policy to save this.

Create role

Navigate to Roles from the left sidebar.
Click Create Role.
In Trusted entity type, select AWS Account.
Select Another AWS account in An AWS account section.
Provide Prophecy's account ID as 133450206866.
Click Next.

Select the Policy which you created just above and click on (1) Next.

Provide a role name.
Optionally, provide a description.
Edit the trusted Select trusted entities JSON, to only provide access to prophecy-mwaa users created for this auth and not to all Prophecy Users as shown below.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::133450206866:user/prophecy-mwaa"
      },
      "Action": "sts:AssumeRole",
      "Condition": {}
    }
  ]
}

Once done click on create role.

Now you can go to the created Role and copy the ARN of this role to be provided in the fabric.

Once the assume role authentication is enabled, you would see below fields for authentication in MWAA fabric.

Here you can provide the AssumeRole created for Prophecy and provide the region where your Airflow instance is running. Once done, click on (3) Fetch environment to list the Airflow Instances which are accessible to provided AssumeRole.

This completes the fabric creation and you can start setting up Connections.

Setting up Connections

You need Airflow to talk to various other systems in your Data Platform to be able to do certain tasks like sending Email, triggering Spark pipelines, and SQL models. For these, we create connections in Airflow.

You can map connections already created in your MWAA, in the Connections Tab of the fabric. Prophecy will use these connections to fetch the connection-id to generate the correct Airflow Code when you use these in your Airflow gems.

For adding a connection, Click on (1) Add Connection button. This Opens up the Connection form as shown.

Setting up Snowflake connection

To be able to schedule your Snowflake SQL models via Airflow, you need to have a Snowflake Connection over DBT from Airflow to your Snowflake Airflow. You need to create the connection in Airflow and provide the mapping to Prophecy in this form.

Select (2) Connection Type as Snowflake(DBT), and select the (3) Fabric you have in Prophecy for your desired Snowflake environment. Provide the (4) Profile Directory and (5) Profile Name used while setting up the connection in Airflow.

Make sure you select the fabric for the same Snowflake environment you have already created the connection for in your Airflow. Once done, hit (4) Save.

Create an Airflow Job

Once the MWAA Airflow fabric is setup with the relevant connections, Airflow job scheduling is done with an easy-to-use interface. Follow this guide to Create an Airflow job.