Exam Professional Data Engineer topic 1 question 110 discussion

Should be B. The Partition transform would require the element identifying the valid/invalid records for partitioning the pcollection that means there is some logic to be executed before the Partition transformation is invoked. That logic can be implemented in a ParDO transform and which can both identify valid/invalid records and also generate two PCollections one with valid records and other with invalid records.

B: ParDo is a Beam transform for generic parallel processing. ParDo is useful for common data processing operations, including: a. Filtering a data set. You can use ParDo to consider each element in a PCollection and either output that element to a new collection, or discard it. b. Formatting or type-converting each element in a data set. c. Extracting parts of each element in a data set. d. Performing computations on each element in a data set. A does not help C Partition is a Beam transform for PCollection objects that store the same data type. Partition splits a single PCollection into a fixed number of smaller collections. Again, does not help D GroupByKey is a Beam transform for processing collections of key/value pairs. GroupByKey is a good way to aggregate data that has something in common

A - SideInput is often used to validate data, however, we need to create the SideInput first. When using SideInput to filter data, it is actually another ParDo call. C, D - This is common way to filter too, but we will need the key in order to partition or GroupByKey B - ParDo is the most basic method, it can do anything to the PCollection

B. Add a ParDo transform in Cloud Dataflow to discard corrupt elements.

B is the answer. https://cloud.google.com/dataflow/docs/concepts/beam-programming-model#concepts ParDo is the core parallel processing operation in the Apache Beam SDKs, invoking a user-specified function on each of the elements of the input PCollection. ParDo collects the zero or more output elements into an output PCollection. The ParDo transform processes elements independently and possibly in parallel.

vote B :https://beam.apache.org/documentation/programming-guide/#pardo Filtering a data set. You can use ParDo to consider each element in a PCollection and either output that element to a new collection or discard it. Formatting or type-converting each element in a data set. If your input PCollection contains elements that are of a different type or format than you want, you can use ParDo to perform a conversion on each element and output the result to a new PCollection. Extracting parts of each element in a data set. If you have a PCollection of records with multiple fields, for example, you can use a ParDo to parse out just the fields you want to consider into a new PCollection. Performing computations on each element in a data set. You can use ParDo to perform simple or complex computations on every element, or certain elements, of a PCollection and output the results as a new PCollection.

Filtering with ParDo. ParDo is a Beam transform for generic parallel processing. ParDo is useful for common data processing operations/

vote for 'B', ParDo can discard the elements. https://beam.apache.org/documentation/programming-guide/

B - seems to be better option since we need to filter out, question does not specify that we do need to store it into different Pcollection. https://beam.apache.org/documentation/transforms/python/overview/ ParDo is general purpose whereas partition splits the elements into do different pcollections. https://beam.apache.org/documentation/transforms/python/elementwise/partition/

B is correct

B is correct

B, ParDo is useful for a variety of common data processing operations, including: Filtering a data set. You can use ParDo to consider each element in a PCollection and either output that element to a new collection or discard it.

Looks like C it is https://beam.apache.org/documentation/programming-guide/

Correct answer should be "C". A Pardo transform will allow the processing to happening in parallel using multiple workers. Partition transform will allow data to be partitions in two different Pcollections according to some logic. Using partition transform once can split the corrupted data and finally discard it.

Correct B.

Correct - B