Exam Professional Data Engineer topic 1 question 120 discussion

You would want to get alerted only if Pipeline fails & not if it is running fine. I think Option [B] is correct, because in event of Pipeline failure : 1) subscription/ num_undelivered_messages would pile up at a constant rate as the source has consistent throughput 2) instance/storage/ used_bytes will get closer to zero. Hence need to monitor it's rate of change

Correct - B

It should be Answer A. You want to see in being processed versus looking for a bottleneck.

For those who may get confuse at the start by the term 'subscription/num_undelivered_messages', it is not a division. It is the full path of the metric. So we should just read it as 'num_undelivered_messages'. The same for 'used_bytes'. So if we see the source have more backlog (more num_undelivered_messages), or the destination ultilization going down, that is the indicator of something going wrong

Answer B: Trick: In stackdriver always put Alert for Subscriber + CPU Subscriber - num of undelivered message INCREASE alert CPU - Instance or storage DECREASE alert. Make sense right !

Nobody seems to pay attention to instance/storage/used_bytes. I only find this metric for Spanner. https://cloud.google.com/monitoring/api/metrics_gcp#gcp-spanner While Dataflow processes and stores everything in Cloud Storage, Spanner could only be the source. https://cloud.google.com/spanner/docs/change-streams Also, if it is either A or B, the instance/storage/used_bytes metric does not make sense for the destination, which is Cloud Storage. Can anyone help me understand?

B. An alert based on an increase of subscription/num_undelivered_messages for the source and a rate of change decrease of instance/storage/ used_bytes for the destination

B is right

An alert based on a decrease of subscription/num_undelivered_messages for the source and a rate of change increase of instance/storage/ used_bytes for the destination 10 subscriptions / 1 undelivered messages = 10 10 subscriptions / 5 undelivered messages = 2 You clearly want to be alerted when the number of undelivered messages increases. The ratio then decreases. In my example from 10 to 2.

B is the answer. https://cloud.google.com/pubsub/docs/monitoring#monitoring_the_backlog Monitor message backlog To ensure that your subscribers are keeping up with the flow of messages, create a dashboard. The dashboard can show the following backlog metrics, aggregated by resource, for all your subscriptions: - Unacknowledged messages (subscription/num_undelivered_messages) to see the number of unacknowledged messages.

Increase subscription/num delivered message decrease instance/storage/used bytes

Correct - B

Correct: B

isn't B and C are same.

B. It's useful to monitor the source that keeps sending data while the destination that doesn't take anything in.

The answer is B. subscription/num_undelivered_messages: the number of messages that subscribers haven't processed https://cloud.google.com/pubsub/docs/monitoring#monitoring_forwarded_undeliverable_messages

Silly question: what is subscription/ num_undelivered_messages, it is divided by? or per subscription per num_undelivered_messages?