Exam Professional Machine Learning Engineer topic 1 question 82 discussion

Converting a large 5 terabyte CSV file to a TFRecord can be a time-consuming process, and you would still be dealing with a single large file.

This is a bad question. But imho, Answer: A. - TFRecords will improve read speeds with its binary format. Presumably the large file was there for a reason, possibly the output of a upstream process whose data may change in the future. TFRecords is a straightforward FIRST step as a part of a pipeline. - The other option is parallel interleave. Also improves read speeds, but not as straightforward as a first step and requires lots of file in the database that require version control.

A. Preprocessing your data into TFRecord format can significantly improve I/O performance and reduce the time spent on parsing and loading data, which is critical for optimizing the input pipeline for large-scale datasets.

according to the official doc A, C seems to pre TFX solution

Based on the official documentation, Option A (converting to TFRecord format) is actually the correct first action to try, and the claim is incorrect. Why TFRecord is the Best First Option TFRecord format is specifically recommended for large datasets because: - It provides extremely high throughput when reading from Cloud Storage, especially for large-scale training[2] - It's the recommended format for structured data and large files[2] - It's designed for efficient serialization of structured data and optimal performance with TensorFlow

c is the right answer

Preprocessing the input CSV file into a TFRecord file optimizes the input data pipeline by enabling more efficient reading and processing. TFRecord is a binary format that is faster to read and more efficient for TensorFlow to process compared to CSV, which is a text-based format. This change can significantly reduce the time spent on data input operations during model training.

A) Convert CSV file into TFRecord is more effecient and processing CSV in parallel (C)

While preprocessing the input CSV file into a TFRecord file (Option A) can improve the performance of your input pipeline, it is not the first action to try in this situation. Converting a large 5 terabyte CSV file to a TFRecord can be a time-consuming process, and you would still be dealing with a single large file.

i think C based on the consideration: "Which action should you try first ", meaning it should be less impactful to continue using CSV.

https://www.tensorflow.org/guide/data_performance#best_practice_summary

Went with C

Option A, preprocess the input CSV file into a TFRecord file, is not as good because it requires additional processing time. Hence, I think C is the best choice.

I think it could be A. https://cloud.google.com/architecture/best-practices-for-ml-performance-cost#preprocess_the_data_once_and_save_it_as_a_tfrecord_file

Clearly both A and C works here, but I can't find any documentation which suggests C is any better than A.

"Which action should you try first" seems to be key -- C seems more intuitive as first step! A is valid as well (interleave works w TFRecords) & definitely more efficient IMO, but maybe 2nd step!

Option B (randomly selecting a 10 gigabyte subset of the data) could lead to a loss of useful data and may not be representative of the entire dataset. Option C (splitting into multiple CSV files and using a parallel interleave transformation) may also improve the performance, but may be more complex to implement and maintain, and may not be as efficient as converting to TFRecord. Option D (setting the reshuffle_each_iteration parameter to true in the tf.data.Dataset.shuffle method) is not directly related to the input data format and may not provide as significant a performance improvement as converting to TFRecord.