Exam Professional Data Engineer topic 1 question 193 discussion

This has to be D. How could it even be B? The source is a proprietary format. Dataflow wouldn't have a built-in template to ead the file. You will have to create something custom.

For me it's clearly D It's between B and D, but read B, store raw data in Big Query? Use a Dataflow pipeline just to store raw data into Big Query, and transform later? You'd need to do another pipeline for that, and is not efficient.

Option D is the best approach given the constraints - use an Apache Beam custom connector to write a Dataflow pipeline that streams the data into BigQuery in Avro format. The key reasons: • Dataflow provides managed resource scaling for efficient stream processing • Avro format has schema evolution capabilities and efficient serialization for flight telemetry data • Apache Beam connectors avoid having to write much code to integrate proprietary data sources • Streaming inserts data efficiently compared to periodic batch jobs In contrast, option A uses Cloud Functions which lack native streaming capabilities. Option B stores data in less efficient JSON format. Option C uses Dataproc which requires manual cluster management. So leveraging Dataflow + Avro + Beam provides the most efficient way to stream proprietary flight data into BigQuery while using minimal resources.

Its talking about streaming? none of the options talk about triggering a load to begin. We need a trigger or schedule to run first.

Option D allows you to use a custom connector to read the proprietary data format and write the data to BigQuery in Avro format.

the keyword is streaming

Between B and D. Firstly transformation is not mentioned in the question, So B is less probable. Then Efficient import is mentioned in the question, Converting to Avro will consume less space. I am going with D

Answer is D , Why not B, changing data format before uploading to bigquery is good approach.

I believe keyword here is "An aerospace company uses a proprietary data format" So if we list the connectors available in Apache Beam, we are listed with these options; https://beam.apache.org/documentation/io/connectors/ So I believe, we have to create our own custom connector to read from the proprietary data format hence the answer should be B

D is right

D is the answer.

B is the most efficient

https://cloud.google.com/spanner/docs/change-streams/use-dataflow#core-concepts

Ans B https://cloud.google.com/architecture/streaming-avro-records-into-bigquery-using-dataflow Is there a reason to use apache beam connector yet there is dataflow which is a standard solution for that scenario?

D. Use an Apache Beam custom connector to write a Dataflow pipeline that streams the data into BigQuery in Avro format. Reveal Solution

B is the most efficient for me.