Exam Certified Data Engineer Professional topic 1 question 101 discussion

C. Size on Disk is > 0 When using Spark's MEMORY_ONLY storage level, the ideal scenario is that the data is fully cached in memory, and the Size on Disk should be 0 (indicating that the data is not spilled to disk). If the Size on Disk is greater than 0, it suggests that some data has been spilled to disk, which can lead to degraded performance as reading from disk is slower than reading from memory.

In the Spark UI’s Storage tab, when you're using the MEMORY_ONLY storage level, Spark tries to cache the RDD/table completely in memory. If a partition of the RDD does not fit into memory, Spark does not cache that partition and recomputes it when needed. Indicator: * in the Block Name A * (asterisk) next to an RDD block name in the Storage tab indicates that Spark failed to cache that block.

In the Spark UI's Storage tab, an indicator that a cached table is not performing optimally would be the presence of the _disk annotation in the RDD Block Name. This annotation indicates that some partitions of the cached data have been spilled to disk because there wasn't enough memory to hold them. This is suboptimal because accessing data from disk is much slower than from memory. The goal of caching is to keep data in memory for fast access, and a spill to disk means that this goal is not fully achieved.

Under MEMORY_ONLY, Spark does not write to disk, so Size on Disk should be 0. Under MEMORY_ONLY, off-heap memory is not used In the Storage tab, an asterisk (*) next to the RDD block name (e.g., rdd_42_3*) indicates the partition could not be cached due to memory constraints

suretly C

I think is C

In this case any data on disk means that cache is not performing optimally