Call glue jobs directly from step functions instead of lambdas

[tommycheek]

Can you insert a glue job into one of the steps of the stage A or stage B state machines instead of calling a glue job from a lambda function? If so, would you need to change anything in the datalakelibrary repo or could you just update the yaml in the stage A and stage B repos?

[cnfait]

You can call a Glue job from Step Functions directly, yes. In Stage B, there is a JSON file containing the Step Functions definition:
https://github.com/awslabs/aws-serverless-data-lake-framework/blob/main/sdlf-stageB/state-machine/stage-b.asl.json

You would edit this file to use the Glue integration documented here: https://docs.aws.amazon.com/step-functions/latest/dg/connect-glue.html. Be mindful of the fact an IAM role is attached to a Step Functions, and it needs permissions to run the Glue job. The current role is defined in sdlf-team.

Overall the framework is very flexible, you're not really forced to use the datalakeLibrary, it is just convenient when using Lambda functions and having to interact with SDLF components (buckets, queues, DynamoDB tables)!