Exam Professional Data Engineer All Questions

View all questions & answers for the Professional Data Engineer exam

Exam Professional Data Engineer topic 1 question 120 discussion

Actual exam question from Google's Professional Data Engineer

Question #: 120
Topic #: 1

[All Professional Data Engineer Questions]

You are operating a Cloud Dataflow streaming pipeline. The pipeline aggregates events from a Cloud Pub/Sub subscription source, within a window, and sinks the resulting aggregation to a Cloud Storage bucket. The source has consistent throughput. You want to monitor an alert on behavior of the pipeline with Cloud
Stackdriver to ensure that it is processing data. Which Stackdriver alerts should you create?

A. An alert based on a decrease of subscription/num_undelivered_messages for the source and a rate of change increase of instance/storage/ used_bytes for the destination
B. An alert based on an increase of subscription/num_undelivered_messages for the source and a rate of change decrease of instance/storage/ used_bytes for the destination
C. An alert based on a decrease of instance/storage/used_bytes for the source and a rate of change increase of subscription/ num_undelivered_messages for the destination
D. An alert based on an increase of instance/storage/used_bytes for the source and a rate of change decrease of subscription/ num_undelivered_messages for the destination

Show Suggested Answer

Suggested Answer: B 🗳️

by [deleted] at March 22, 2020, 12:46 p.m.

Comments

Submit Cancel

dambilwa

Highly Voted 3 years, 5 months ago

You would want to get alerted only if Pipeline fails & not if it is running fine. I think Option [B] is correct, because in event of Pipeline failure : 1) subscription/ num_undelivered_messages would pile up at a constant rate as the source has consistent throughput 2) instance/storage/ used_bytes will get closer to zero. Hence need to monitor it's rate of change

upvoted 28 times

Barniyah

3 years, 5 months ago

Yes, you are right, it should be B: Thank you

upvoted 5 times

...

marioferrulli

2 years ago

Why would the instance/storage/used_bytes get closer to zero? If there's an error at a certain point, wouldn't we just see that the used_bytes remain constant while the num_undelivered_messages increases? I don't get why the destination's used bytes should decrease.

upvoted 1 times

baubaumiaomiao

1 year, 12 months ago

"If there's an error at a certain point, wouldn't we just see that the used_bytes remain constant while the num_undelivered_messages increases?" It's the rate of change, not the absolute value

upvoted 2 times

...

szefco

2 years ago

"rate of change decrease of instance/storage/ used_bytes" - if rate of instance/storage/ used_bytes decreases that means less data is written - so something is wrong with the pipeline. It's not used bytes that decreases - it's rate of change decreases. Example: if everything works fine your pipeline writes 5MB/s to the sink. If it decreases to 0.1MB/s it means something is wrong

upvoted 6 times

...

[Removed]

Highly Voted 3 years, 8 months ago

Correct - B

upvoted 21 times

...

desertlotus1211

Most Recent 3 months ago

Selected Answer: A

It should be Answer A. You want to see in being processed versus looking for a bottleneck.

upvoted 1 times

...

midgoo

9 months, 1 week ago

Selected Answer: B

For those who may get confuse at the start by the term 'subscription/num_undelivered_messages', it is not a division. It is the full path of the metric. So we should just read it as 'num_undelivered_messages'. The same for 'used_bytes'. So if we see the source have more backlog (more num_undelivered_messages), or the destination ultilization going down, that is the indicator of something going wrong

upvoted 16 times

desertlotus1211

3 months ago

the Answer is A. You want to see it working.

upvoted 1 times

desertlotus1211

3 months ago

you're looking for evidence that it's working: ' that it is processing data....'

upvoted 1 times

...

kryzo

6 months, 1 week ago

great explanation thanks !

upvoted 2 times

...

musumusu

9 months, 4 weeks ago

Answer B: Trick: In stackdriver always put Alert for Subscriber + CPU Subscriber - num of undelivered message INCREASE alert CPU - Instance or storage DECREASE alert. Make sense right !

upvoted 3 times

...

atlan

11 months, 2 weeks ago

Nobody seems to pay attention to instance/storage/used_bytes. I only find this metric for Spanner. https://cloud.google.com/monitoring/api/metrics_gcp#gcp-spanner While Dataflow processes and stores everything in Cloud Storage, Spanner could only be the source. https://cloud.google.com/spanner/docs/change-streams Also, if it is either A or B, the instance/storage/used_bytes metric does not make sense for the destination, which is Cloud Storage. Can anyone help me understand?

upvoted 1 times

desertlotus1211

11 months ago

look here: https://cloud.google.com/monitoring/api/metrics_gcp instance/storage/used_bytes GA Storage used.

upvoted 2 times

...

AzureDP900

11 months, 3 weeks ago

B. An alert based on an increase of subscription/num_undelivered_messages for the source and a rate of change decrease of instance/storage/ used_bytes for the destination

upvoted 1 times

...

AzureDP900

11 months, 3 weeks ago

B is right

upvoted 1 times

...

Catweazle1983

11 months, 3 weeks ago

Selected Answer: A

An alert based on a decrease of subscription/num_undelivered_messages for the source and a rate of change increase of instance/storage/ used_bytes for the destination 10 subscriptions / 1 undelivered messages = 10 10 subscriptions / 5 undelivered messages = 2 You clearly want to be alerted when the number of undelivered messages increases. The ratio then decreases. In my example from 10 to 2.

upvoted 1 times

squishy_fishy

9 months ago

subscription/num_undelivered_messages is a path, not a division.

upvoted 1 times

...

zellck

1 year ago

Selected Answer: B

B is the answer. https://cloud.google.com/pubsub/docs/monitoring#monitoring_the_backlog Monitor message backlog To ensure that your subscribers are keeping up with the flow of messages, create a dashboard. The dashboard can show the following backlog metrics, aggregated by resource, for all your subscriptions: - Unacknowledged messages (subscription/num_undelivered_messages) to see the number of unacknowledged messages.

upvoted 3 times

...

A1000

1 year, 4 months ago

Selected Answer: B

Increase subscription/num delivered message decrease instance/storage/used bytes

upvoted 1 times

...

Pime13

1 year, 5 months ago

Selected Answer: B

Correct - B

upvoted 1 times

...

JG123

2 years ago

Correct: B

upvoted 2 times

...

Abhi16820

2 years, 1 month ago

isn't B and C are same.

upvoted 2 times

...

JayZeeLee

2 years, 1 month ago

B. It's useful to monitor the source that keeps sending data while the destination that doesn't take anything in.

upvoted 3 times

...

squishy_fishy

2 years, 2 months ago

The answer is B. subscription/num_undelivered_messages: the number of messages that subscribers haven't processed https://cloud.google.com/pubsub/docs/monitoring#monitoring_forwarded_undeliverable_messages

upvoted 2 times

...

squishy_fishy

2 years, 3 months ago

Silly question: what is subscription/ num_undelivered_messages, it is divided by? or per subscription per num_undelivered_messages?

upvoted 2 times

910

1 year, 8 months ago

yes is misleading: the metric "subscription/num_undelivered_messages" is just the path of the API URL actions.googleapis.com/...subscription/num_undelivered_messages ref: https://cloud.google.com/monitoring/api/metrics_gcp#pubsub/subscription/num_undelivered_messages

upvoted 3 times

...

Load full discussion...