Exam Professional Machine Learning Engineer topic 1 question 113 discussion

Previously Google pattern was Pub/Sub -> Dataflow -> BQ but now it looks as there is new Pub/Sub -> BQ https://cloud.google.com/blog/products/data-analytics/pub-sub-launches-direct-path-to-bigquery-for-streaming-analytics

This approach provides a scalable, flexible, and efficient solution for real-time data ingestion and transformation, ensuring that user activity data is seamlessly integrated into BigQuery for analysis and explanations. BigQuery can perform powerful data transformations using SQL. However, there are key differences in how BigQuery and Dataflow handle data, especially in the context of real-time data ingestion.

Pub/Sub and Dataflow are needed for real-time ingestion. Pub/Sub cannot do it alone

Pub/Sub is used for message ingestion and it cannot directly load data into BigQuery. Pub/Sub only delivers messages, but we will need Dataflow or another processing tool to transform and load the data into BigQuery. Hence it is D in my opinion.

Since Data transformation is needed.

The question specifies that transformation occurs in Bigquery. This means the new direct pub/sub to bigquery streaming path is correct.

Need PubSub and Dataflow both for this

Werner123 i agree

User data would most likely include PII, for that case it is still recommended to use Dataflow since you need to remove/anonymise sensitive data.

I would have added "with / without data transformation" to the question to choose the right answer between A or D

I had my doubts between A and D. But since the transformation will occur in bigquery I think Pubsub suffices.

Agree with TNT87. From the same link: “For Pub/Sub messages where advanced preload transformations or data processing before landing data in BigQuery (such as masking PII) is necessary, we still recommend going through Dataflow.” It’s “analyze user activity data”, not merely streaming IoT into BigQuery so that concerns like privacy are per se n/a. One can deal with PII after landing in BigQuery as well, but apparently that’s not what they recommend.

Pub/Sub -> DataFlow -> BigQuery

D. Configure Pub/Sub and a Dataflow streaming job to ingest the data into BigQuery. This solution involves using Google Cloud Pub/Sub as the messaging service to receive the data from the mobile application, and then using Google Cloud Dataflow to transform and load the data into BigQuery in real time. Pub/Sub is a scalable and reliable messaging service that can handle high-volume real-time data streaming, while Dataflow provides a unified programming model to develop and run data processing pipelines. This solution is suitable for handling large volumes of user activity data from mobile applications and ingesting it into BigQuery in real-time for analysis and ML experimentation.

A agree with pshemol

need dataflow