Exam Certified Data Engineer Professional topic 1 question 51 discussion

Predicate push-down is an optimization where conditions (such as filters) are pushed as close to the data source as possible (often to the database or file system level), reducing the amount of data read and processed. If predicate push-down isn't being leveraged, it can result in reading unnecessary data, leading to performance degradation. Execute a query --> Click View and go to Spark UI --> Navigate to SQL/DataFrame tab in SparkUI --> Click on any stage --> Navigate to details to find Physical Plan

when predicated pushdown is working properly, the amount of data read should be much lower because the data source is able to filter out the rows at read time based on the query predicates. if predicate pushdown is not levaraged, stages might read a much larger volume of data than necessary, which can be observed in the input column in the stage detail screen therefore B is the correct option not A : executor logs might contain some information, but they are niot the most direct way to assess predicate push-down performance not C : used to check RDD caching and persistence, not predicate push-down not D : it holds meta data and statistics but is not viewed via the spark UI for diagnosing query performance not E : while physical plan in the query detail screen might filter push-down, interpreting it requires more expertise, and the metric on the input data size(option B) is more straight forward indicator.

E

E is correct : https://docs.datastax.com/en/dse/6.9/spark/predicate-push-down.html

Query plan. Correct is E