Welcome to ExamTopics
ExamTopics Logo
- Expert Verified, Online, Free.
Mail Us [email protected]
Menu

Unlimited Access

Get Unlimited Contributor Access to the all ExamTopics Exams!
Take advantage of PDF Files for 1000+ Exams along with community discussions and pass IT Certification Exams Easily.
Google Discussions

Exam Professional Data Engineer topic 1 question 256 discussion

Actual exam question from Google's Professional Data Engineer
Question #: 256
Topic #: 1
[All Professional Data Engineer Questions]

You are deploying an Apache Airflow directed acyclic graph (DAG) in a Cloud Composer 2 instance. You have incoming files in a Cloud Storage bucket that the DAG processes, one file at a time. The Cloud Composer instance is deployed in a subnetwork with no Internet access. Instead of running the DAG based on a schedule, you want to run the DAG in a reactive way every time a new file is received. What should you do?

  • A. 1. Enable Private Google Access in the subnetwork, and set up Cloud Storage notifications to a Pub/Sub topic.
    2. Create a push subscription that points to the web server URL.
  • B. 1. Enable the Cloud Composer API, and set up Cloud Storage notifications to trigger a Cloud Function.
    2. Write a Cloud Function instance to call the DAG by using the Cloud Composer API and the web server URL.
    3. Use VPC Serverless Access to reach the web server URL.
  • C. 1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance.
    2. Create a Private Service Connect (PSC) endpoint.
    3. Write a Cloud Function that connects to the Cloud Composer cluster through the PSC endpoint.
  • D. 1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance.
    2. Write a Cloud Function instance to call the DAG by using the Airflow REST API and the web server URL.
    3. Use VPC Serverless Access to reach the web server URL.
Show Suggested Answer Hide Answer
Suggested Answer: A 🗳️
by scaenruy at Jan. 3, 2024, 4:56 p.m.

Comments

Chosen Answer:
This is a voting comment (?) , you can switch to a simple comment.
Switch to a voting comment New
Submit Cancel
raaad
Highly Voted 3 months, 3 weeks ago
Selected Answer: C
- Enable Airflow REST API: In Cloud Composer, enable the "Airflow web server" option. - Set Up Cloud Storage Notifications: Create a notification for new files, routing to a Cloud Function. - Create PSC Endpoint: Establish a PSC endpoint for Cloud Composer. - Write Cloud Function: Code the function to use the Airflow REST API (via PSC endpoint) to trigger the DAG. ======== Why not Option D - Using the web server URL directly wouldn't work without internet access or a direct path to the web server.
upvoted 10 times
AllenChen123
3 months, 1 week ago
Why not B, use Cloud Composer API
upvoted 4 times
...
...
chrissamharris
Most Recent 1 month ago
Selected Answer: D
Why not Option C? C involves creating a Private Service Connect (PSC) endpoint, which, while viable for creating private connections to Google services, adds complexity and might not be required when simpler solutions like VPC Serverless Access (as in Option D) can suffice.
upvoted 1 times
chrissamharris
1 month ago
https://cloud.google.com/vpc/docs/serverless-vpc-access: Serverless VPC Access makes it possible for you to connect directly to your Virtual Private Cloud (VPC) network from serverless environments such as Cloud Run, App Engine, or Cloud Functions
upvoted 1 times
...
...
d11379b
1 month ago
Selected Answer: D
The answer should be D Serverless VPC Access makes it possible for you to connect directly to your Virtual Private Cloud (VPC) network from serverless environments such as Cloud Run, App Engine, or Cloud Functions. Configuring Serverless VPC Access allows your serverless environment to send requests to your VPC network by using internal DNS and internal IP addresses (as defined by RFC 1918 and RFC 6598). The responses to these requests also use your internal network. You can use Serverless VPC Access to access Compute Engine VM instances, Memorystore instances, and any other resources with internal DNS or internal IP address. (Reference: https://cloud.google.com/vpc/docs/serverless-vpc-access) When you use Airflow Rest API to tigger the job, the url is based on the private IP address of Cloud Composer Instance, so you need to use Serverless VPC Access for it.
upvoted 1 times
d11379b
1 month ago
Why not C: The reference here (https://cloud.google.com/vpc/docs/private-service-connect#published-services) limits the available use cases: Private Service Connect supports access to the following types of managed services: Published VPC-hosted services, which include the following: Google published services, such as Apigee or the GKE control plane Third-party published services provided by Private Service Connect partners Intra-organization published services, where the consumer and producer might be two different VPC networks within the same company Google APIs, such as Cloud Storage or BigQuery Unfortunately your airflow Rest API is not published as a service in the list, so you can not use it This is also one of the reasons why you should reject A
upvoted 1 times
d11379b
1 month ago
B is not appropriate while Cloud Composer API can really execute Airflow command，but It’s not via web server Url to run a DAG in this case, and I doubt if it is really possible
upvoted 1 times
...
...
...
Matt_108
3 months, 2 weeks ago
Selected Answer: C
Option C, raaad explained well why
upvoted 1 times
...
scaenruy
3 months, 3 weeks ago
Selected Answer: C
C. 1. Enable the Airflow REST API, and set up Cloud Storage notifications to trigger a Cloud Function instance. 2. Create a Private Service Connect (PSC) endpoint. 3. Write a Cloud Function that connects to the Cloud Composer cluster through the PSC endpoint.
upvoted 1 times
...

Get IT Certification

Unlock free, top-quality video courses on ExamTopics with a simple registration. Elevate your learning journey with our expertly curated content. Register now to access a diverse range of educational resources designed for your success. Start learning today with ExamTopics!

Start Learning for free
Community vote distribution
A (35%)
C (25%)
B (20%)
Other
Most Voted
A voting comment increases the vote count for the chosen answer by one.

Upvoting a comment with a selected answer will also increase the vote count towards that answer by one. So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.

SaveCancel