Exam Professional Data Engineer topic 1 question 176 discussion

C. Create a Dataflow workflow that reads the data from Cloud Storage, checks for values outside the expected range, sets the value to an appropriate default, and writes the updated records to a new dataset in Cloud Storage. You can't filter out data using BQ load commands. You must imbed the logic to filter out data (i.e. time ranges) in another decoupled way (i.e. Dataflow, Cloud Functions, etc.). Therefore, A and B add additional complexity and deviates from the Data Lake design paradigm. D is wrong as the question strictly implies that the existing data set needs to be retained for compliance.

Strange answers... Since when does cloud storage have datasets? Lol Keeping this in mind, the answer must be C, but none is really correcg

why not D if the versioning is activated while creating your bucket ?

Option C is the best approach in this situation. Here is why: Option A would remove data which may be needed for compliance reasons. Keeping the original data is preferred. Option B makes a copy of the data but still removes potentially useful records. Additional storage costs would be incurred as well. Option C uses Dataflow to clean the data by setting out of range values while keeping the original data intact. The fixed records are written to a new location for further analysis. This meets the requirements. Option D writes the fixed data back to the original location, overwriting the original data. This would violate the compliance needs to keep the original data untouched. So option C leverages Dataflow to properly clean the data while preserving the original data for compliance, at reasonable operational costs. This best achieves the stated requirements.

C. Create a Dataflow workflow that reads the data from Cloud Storage, checks for values outside the expected range, sets the value to an appropriate default, and writes the updated records to a new dataset in Cloud Storage.

C is the answer.

C is correct