A data engineer must orchestrate a data pipeline that consists of one AWS Lambda function and one AWS Glue job. The solution must integrate with AWS services. Which solution will meet these requirements with the LEAST management overhead?
A.
Use an AWS Step Functions workflow that includes a state machine. Configure the state machine to run the Lambda function and then the AWS Glue job.
B.
Use an Apache Airflow workflow that is deployed on an Amazon EC2 instance. Define a directed acyclic graph (DAG) in which the first task is to call the Lambda function and the second task is to call the AWS Glue job.
C.
Use an AWS Glue workflow to run the Lambda function and then the AWS Glue job.
D.
Use an Apache Airflow workflow that is deployed on Amazon Elastic Kubernetes Service (Amazon EKS). Define a directed acyclic graph (DAG) in which the first task is to call the Lambda function and the second task is to call the AWS Glue job.
Step Functions is a managed service for building serverless workflows. You define a state machine that orchestrates the execution sequence.
This eliminates the need to manage and maintain your own workflow orchestration server like Airflow.
AWS Glue is a fully managed ETL (extract, transform, load) service that makes it easy to orchestrate data pipelines. Using the AWS Glue workflow to run Lambda functions and glue jobs is the easiest and least expensive option because it's a fully managed service that requires no additional workflow tools or infrastructure to configure and manage. Other options require additional tools or resources to configure and manage, and are therefore more expensive to manage.
Option A, using AWS Step Functions, is the best solution to meet the requirement with the least management overhead. Step Functions is designed for easy integration with AWS services like Lambda and Glue, providing a managed, low-code approach to orchestrate workflows. This allows for a more straightforward setup and less ongoing management compared to the other options.
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
pypelyncar
Highly Voted 10 months, 4 weeks agohcong
Most Recent 8 months, 2 weeks agotgv
11 months, 1 week agohnk
11 months, 3 weeks agoFuriouZ
1 year, 1 month agorralucard_
1 year, 3 months ago