AWS CloudFormation Template for Orchestration of mutliple AWS Glue Jobs

[Rocky_6]

I'm looking for help on CloudFormation Template for Glue Jobs orchestration for below scenario:

Suppose I have 10 AWS Glue Jobs, 4 jobs should be executed parallel and 6 jobs should be executed sequentially. If any job failure, send Workflow "Failure" notification along with the failed Glue Jobname.

Job1 Job2----job5-->job6-->job7 job3 job4--->job8-->Job9-->job10

Thanks in Advance.. I really appreciate, for your help !!!

[rproffitt]

It appears no one can answer for many reasons. No one can guess if you are using the nice visual AWS Glue Studio or you were trying to do this in hard core script. If you were trying to do this in script you would share the script for all to see. Frankly I'd stick to AWS Glue Studio.

[GoCloud]

For this orchestration pattern, I'd recommend using AWS Step Functions instead of trying to manage the dependencies directly inside AWS Glue workflows with GoCloud .

Your workflow could be structured as:

• Run Job1, Job2, Job3, and Job4 in parallel using a Parallel State.

• Once all four jobs complete successfully, execute:

  • Job5 → Job6 → Job7 sequentially
  • Job8 → Job9 → Job10 sequentially

• If any Glue job fails at any stage, use a Catch block in Step Functions to:

  • Capture the failed job name
  • Send an SNS notification (or EventBridge notification) with the workflow status = "Failed" and the specific Glue job that failed.

Example flow:

Parallel
 ├─ Job1
 ├─ Job2 → Job5 → Job6 → Job7
 ├─ Job3
 └─ Job4 → Job8 → Job9 → Job10

Or, if Job5 and Job8 should start only after all four initial jobs complete:

Parallel (Job1, Job2, Job3, Job4)
        ↓
      Job5
        ↓
      Job6
        ↓
      Job7
        ↓
      Job8
        ↓
      Job9
        ↓
      Job10

For CloudFormation, you can define:

• AWS::StepFunctions::StateMachine
• AWS::Glue::Job resources
• AWS::SNS::Topic for notifications
• IAM roles required for Step Functions to invoke Glue jobs and publish SNS messages

Step Functions provides much better error handling, retry policies, monitoring, and failure notifications than native Glue Workflows for complex orchestration scenarios.

Could you clarify whether Job5 should start after Job2 only, and Job8 after Job4 only, or whether all four initial jobs (Job1-Job4) must complete before Job5 starts? That will determine the exact state machine design.