Enable Orchestration

To enable orchestration in a self-hosted Prophecy deployment, follow these steps. 

Prerequisites

To enable Orchestrator in your Prophecy deployment, you need:

• Prophecy version 4.1.3.x or later.

• Access to your Kubernetes cluster with appropriate permissions.

• Prophecy cluster admin access in the Prophecy web environment.

• Access to the postgres pod and database.

• Images pulled for the services you are enabling:

  • Orchestrator: orchestrator, prophecy-orchestrate-sandbox

Step 1: Enable services in configuration

Add the following services to all of the following locations:

• prophecycluster

• athena-pod-metadata-cm

enabledApps:
 - orchestrator

To edit each resource, use the following commands:

kubectl edit prophecycluster -n <namespace>
kubectl edit cm athena-pod-metadata-cm -n <namespace>

Note: Every time you manually edit prophecycluster, you also need to edit athena-pod-metadata-cm to preserve these settings across Prophecy updates.

Step 2: Update cp-env-cm 

Edit cp-env-cm:

kubectl edit cm cp-env-cm -n <namespace>

Add the following environment variables:

REPO_CACHE_MAX_SIZE: "3000"
RESTSERVICE_PIPELINEFILESTORELOCATION: /orchestrator/pipelines/
SANDBOX_NAMESPACE: <namespace>

Step 3: Update orchestrator-cm 

Edit orchestrator-cm:

kubectl edit cm orchestrator-cm -n <namespace>

In the data section, add the following for the Orchestrator:

SANDBOX_MINIDLEPODS: "3"
SANDBOX_GOSANDBOXCONFIG_MEMORY: 1Gi
SANDBOX_GOSANDBOXCONFIG_CPU: "1"
SANDBOX_CONFIGMAP: orchestrator-cm

Step 4: Create the Orchestration Database

Access the Postgres pod:

kubectl -n <namespace> exec -it <postgres-pod-name> -- /bin/bash

Inside the pod, run:

PGPASSWORD=p0stgresdec20 psql -U postgres -h localhost -c "CREATE DATABASE orchestration;"

Step 5: Configure Orchestration in the UI

1. Log in to Prophecy using an admin account.

2. Navigate to Settings → Admin → Orchestration Settings

3. Enter the following to configure your endpoint:

• Endpoint: https://<your-prophecy-url>.prophecy.io/orchestrate 

• Key Type: optional key to identify the data plane

• Key Value: optional value to identify the data plane

• Click Update.

Step 6: Restart Orchestrator and EdWeb Pod

To apply configuration changes, delete the orchestrator and edweb pod:

kubectl delete pod <orchestrator-pod-name> -n <namespace>
kubectl delete pod <edweb-pod-name> -n <namespace>

Alternatively, restart all pods (note: incurs ~20 minutes downtime):

kubectl rollout restart deploy -n <namespace>

Next steps

After completing these steps, your Prophecy environment will support orchestration and knowledge graph. To choose the data plane for a specific fabric:

1. Open Prophecy.

2. Create a new fabric. 

3. Fill out the Basic Info tab.

4. In the Providers tab, choose Prophecy as the Provider Type.

5. Select the appropriate SQL warehouse provider as the Provider.

6. Expand Advanced Settings.

7. Under Prophecy Dataplane URL, you'll see the endpoint you set up.

8. Configure the rest of the fabric details and save.

For more information about Prophecy fabrics, see Prophecy fabrics.