Exam Professional Data Engineer topic 1 question 153 discussion

Option A is the correct answer. Reasons:- a) Kafka IO and Dataflow is a valid option for interconnect (needless where Kafka is located - On Prem/Google Cloud/Other cloud) b) Sliding Window will help to calculate average. Option C and D are overkill and complex, considering the scenario in the question, https://cloud.google.com/solutions/processing-messages-from-kafka-hosted-outside-gcp

Option A is the correct answer. Reasons:- a) Kafka IO and Dataflow is a valid option for interconnect (needless where Kafka is located - On Prem/Google Cloud/Other cloud) b) Sliding Window will help to calculate average. Option C and D are overkill and complex, considering the scenario in the question,

Option A is correct answer. Option B is not correct. There could be a chance middle of 1st window to middle of 2nd window less messages(i.e > 4000). Option C & D out of scope.

Dataflow with Sliding Time Windows: Dataflow allows you to work with event-time windows, making it suitable for time-series data like incoming IoT messages. Using sliding windows every 5 minutes allows you to compute moving averages efficiently. Sliding Time Window: The sliding time window of 1 hour every 5 minutes enables you to calculate the moving average over the specified time frame. Computing Averages: You can efficiently compute the average when each sliding window closes. This approach ensures that you have real-time visibility into the message rate and can detect deviations from the expected rate. Alerting: When the calculated average drops below 4000 messages per second, you can trigger an alert from within the Dataflow pipeline, sending it to your desired alerting mechanism, such as Cloud Monitoring, Pub/Sub, or another notification service. Scalability: Dataflow can scale automatically based on the incoming data volume, ensuring that you can handle the expected rate of 5000 messages per second.

Option A Pros: This option is relatively simple to implement. It can be used to compute the moving average over any time window. Cons: This option can be computationally expensive, especially if the data stream is large. It can be difficult to troubleshoot if the alert does not fire when it is supposed to.

the correct answer is between A and B since it doesn't make sense to use Pub/Sub combined with Kafka. To have a Moving Average then we should go for A, updating the average estimation every 5 minutes using the new data that came in and eliminating the "most far" 5 minutes.

A is the answer. https://cloud.google.com/dataflow/docs/concepts/streaming-pipelines#windows Windowing functions divide unbounded collections into logical components, or windows. Windowing functions group unbounded collections by the timestamps of the individual elements. Each window contains a finite number of elements. You set the following windows with the Apache Beam SDK or Dataflow SQL streaming extensions: - Hopping windows (called sliding windows in Apache Beam) A hopping window represents a consistent time interval in the data stream. Hopping windows can overlap, whereas tumbling windows are disjoint. For example, a hopping window can start every thirty seconds and capture one minute of data. The frequency with which hopping windows begin is called the period. This example has a one-minute window and thirty-second period.

as explained by Alasmindas

Correct Answer: A

Correct: A

A is enough

A: the correct answer is between A and B, But because used "Moving Average" then we should go for A.

yes , using KafkaIO , we can connect to Kafka cluster.

yes A is correct , because sliding window can only help here.

A To take running averages of data, use hopping windows. You can use one-minute hopping windows with a thirty-second period to compute a one-minute running average every thirty seconds.

I don't think its A or B. Dataflow can't connect directly to kafka.

"You operate an IoT pipeline built around Apache Kafka" The statement in question states. Therefore, building with kafka is the requirement definition for this problem. Just in case you are wondering, a case along with this problem is listed on google by the architects. "Using Cloud Dataflow to Process Outside-Hosted Messages from Kafka" https://cloud.google.com/solutions/processing-messages-from-kafka-hosted-outside-gcp Therefore, A is the correct answer.