Exam Professional Data Engineer All Questions

View all questions & answers for the Professional Data Engineer exam

Exam Professional Data Engineer topic 1 question 278 discussion

Actual exam question from Google's Professional Data Engineer

Question #: 278
Topic #: 1

[All Professional Data Engineer Questions]

Your car factory is pushing machine measurements as messages into a Pub/Sub topic in your Google Cloud project. A Dataflow streaming job, that you wrote with the Apache Beam SDK, reads these messages, sends acknowledgment to Pub/Sub, applies some custom business logic in a DoFn instance, and writes the result to BigQuery. You want to ensure that if your business logic fails on a message, the message will be sent to a Pub/Sub topic that you want to monitor for alerting purposes. What should you do?

A. Enable retaining of acknowledged messages in your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the subscription/num_retained_acked_messages metric on this subscription.
B. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.
C. Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.
D. Create a snapshot of your Pub/Sub pull subscription. Use Cloud Monitoring to monitor the snapshot/num_messages metric on this snapshot.

Show Suggested Answer

Suggested Answer: B 🗳️

by scaenruy at Jan. 4, 2024, 5:24 a.m.

Comments

Submit Cancel

raaad

Highly Voted 1 year, 6 months ago

Selected Answer: B

- Exception Handling in DoFn: Implementing an exception handling block within DoFn in Dataflow to catch failures during processing is a direct way to manage errors. - Side Output to New Topic: Using a side output to redirect failed messages to a new Pub/Sub topic is an effective way to isolate and manage these messages. - Monitoring: Monitoring the num_unacked_messages_by_region on the new topic can alert you to the presence of failed messages.

upvoted 15 times

...

chrissamharris

Most Recent 10 months, 1 week ago

Option C - dead letter topic is built in and requires no changes https://cloud.google.com/pubsub/docs/handling-failures Enable dead lettering in your Pub/Sub pull subscription, and specify a new Pub/Sub topic as the dead letter topic. Use Cloud Monitoring to monitor the subscription/dead_letter_message_count metric on your pull subscription.

upvoted 2 times

Positron75

8 months, 1 week ago

Dead lettering is used to handle messages that have not been acknowledged, but that's unrelated to the processing that Dataflow does, which takes place later in the chain. A message could still be acknowledged and fail processing for whatever reason, so it would not be sent to the dead letter topic. Also, Google advises against using dead lettering with Dataflow anyway: https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#dead-letter-topics Correct answer is B. The error handling has to be written into the Dataflow pipeline itself.

upvoted 2 times

...

7787de3

10 months, 3 weeks ago

Selected Answer: B

See here: https://cloud.google.com/dataflow/docs/concepts/streaming-with-cloud-pubsub#unsupported-features It's not recommended to use Pub/Sub dead-letter topics with Dataflow (...) Instead, implement the dead-letter pattern explicitly in the pipeline

upvoted 1 times

...

Jeyaraj

1 year ago

Option B. Here's why: Side Output for Failed Messages: Dataflow allows you to use side outputs to handle messages that fail processing. In your DoFn , you can catch exceptions and write the failed messages to a separate PCollection . This PCollection can then be written to a new Pub/Sub topic. New Pub/Sub Topic for Monitoring: Creating a dedicated Pub/Sub topic for failed messages allows you to monitor it specifically for alerting purposes. This provides a clear view of any issues with your business logic. topic/num_unacked_messages_by_region Metric: This Cloud Monitoring metric tracks the number of unacknowledged messages in a Pub/Sub topic. By monitoring this metric on your new topic, you can identify when messages are failing to be processed correctly.

upvoted 1 times

...

joao_01

1 year, 3 months ago

I would like to know why isn't anyone considering the option C.

upvoted 1 times

joao_01

1 year, 3 months ago

I think that C is not right anyways: In order to use dead_letter feature, the message CANNOT be acknowledge (somehow) by the subscriber. In this question it says that the messages are first acknowledge and then applied the business logic. So, if there are error in the business logic we cannot use the feature dead_letter, beacuse the message was already acknowledge. Thus, option B is the right one.

upvoted 2 times

...

hanoverquay

1 year, 4 months ago

Selected Answer: B

option B

upvoted 1 times

...

JyoGCP

1 year, 5 months ago

Selected Answer: B

Option B

upvoted 1 times

...

Matt_108

1 year, 6 months ago

Selected Answer: B

Option B - Raaad explanation is complete

upvoted 1 times

...

scaenruy

1 year, 7 months ago

Selected Answer: B

B. Use an exception handling block in your Dataflow’s DoFn code to push the messages that failed to be transformed through a side output and to a new Pub/Sub topic. Use Cloud Monitoring to monitor the topic/num_unacked_messages_by_region metric on this new topic.

upvoted 1 times

...