Exam Certified Data Engineer Professional topic 1 question 63 discussion

Delta Lake maintains a transaction log that records details about every change made to a table. When you execute a count operation on a Delta table, Delta Lake can use the information in the transaction log to calculate the total number of records without having to scan all the data files. This is because the transaction log includes information about the number of records in each file, allowing for an efficient aggregation of these counts to get the total number of records in the table.

Answer C https://delta.io/blog/2023-04-19-faster-aggregations-metadata/#:~:text=You%20can%20get%20the%20number,a%20given%20Delta%20table%20version.

Metadata could be outdated

Suretly C

Answer D. Parquet Metadata Usage: Delta Lake does utilize Parquet file metadata for COUNT(*) operations. Parquet files store metadata, including row counts. Delta efficiently reads this metadata to get the total count without scanning the actual data within the files. This is a key optimization for performance. Why not always scan: Scanning all data files for every COUNT(*) would be extremely inefficient, especially for large tables. This defeats the purpose of using a columnar storage format like Parquet and the optimizations built into Delta Lake and Spark. The transaction log tracks changes to the table (adds, deletes, updates) but doesn't store pre-computed row counts. It's used for time travel, ACID properties, and other Delta features.

"stats": "{\"numRecords\": 3, \"minValues\": {\"x\": 1}, \"maxValues\": {\"x\": 3}, \"nullCount\": {\"x\": 0}}", numRecords - In Delta tx logs will give you the value

Delta transaction log

Transaction log provides statistics about the delta table.

C - transaction logs contains info about files rows count

The answer is C

The answer should be C

total rows will be calculated from delta logs