Exam AWS Certified Data Engineer - Associate DEA-C01 topic 1 question 114 discussion

https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html

Answer D https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. Parquet and ORC are columnar data formats that save space and enable faster queries compared to row-oriented formats like JSON. If you want to convert an input format other than JSON, such as comma-separated values (CSV) or structured text, you can use AWS Lambda to transform it to JSON first. For more information, see Transform data in Amazon Data Firehose.

Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. Parquet and ORC are columnar data formats that save space and enable faster queries compared to row-oriented formats like JSON. If you want to convert an input format other than JSON, such as comma-separated values (CSV) or structured text, you can use AWS Lambda to transform it to JSON first https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html

Kinesis Data Firehose has native capability to convert files into json and then parquet format. There's no need to create lambda function.

simplest and most efficient - Firehose to convert to JSON and store in Parquet - no need for Lambda function

Amazon Kinesis Data Firehose does not natively support CSV to JSON conversion. However, it does support JSON to Parquet conversion. Given that, the best approach with the least development effort is: D. Use Kinesis Data Firehose to invoke an AWS Lambda function that transforms the .csv files to JSON. Use Kinesis Data Firehose to store the files in Parquet format.

Kinesis Data Firehose natively supports data format conversion to Parquet, reducing development effort. AWS Lambda is needed only for the CSV to JSON conversion, as Firehose does not support direct CSV to JSON transformation. Firehose then automatically converts JSON to Parquet and stores it in S3, minimizing custom code.

https://aws.amazon.com/ar/about-aws/whats-new/2016/12/amazon-kinesis-firehose-can-now-prepare-and-transform-streaming-data-before-loading-it-to-data-stores/

Lambda handles both the CSV-to-JSON and JSON-to-Parquet transformations before Firehose stores the data in Amazon S3

If you want to convert an input format other than JSON, such as comma-separated values (CSV) or structured text, you can use AWS Lambda to transform it to JSON first and then you can use Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC.

I would go with C. D is close but Kinesis Data Firehose does not really store files in Parquet format.

https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html You need firstly a JSON (using Lambda) to be able using Kinesis to store it in Parquet

Firehose can't convert csv to json. So, that's D

If you want to convert an input format other than JSON, such as comma-separated values (CSV) or structured text, you can use AWS Lambda to transform it to JSON first. For more information

Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. Parquet and ORC are columnar data formats that save space and enable faster queries compared to row-oriented formats like JSON. If you want to convert an input format other than JSON, such as comma-separated values (CSV) or structured text, you can use AWS Lambda to transform it to JSON first. For more information, see Transform source data in Amazon Data Firehose. Answer D

Kinesis Data Firehose: It has built-in support for data transformation and format conversion. It can directly convert incoming data from .csv to JSON format and then further convert the data to Apache Parquet format before storing it in Amazon S3. Minimal Development Effort: This option requires the least development effort because Kinesis Data Firehose handles both the transformation (from .csv to JSON) and the format conversion (to Parquet) natively. No additional AWS Lambda functions or custom code are needed.

B. Why? Amazon Data Firehose can convert the format of your input data from JSON to Apache Parquet or Apache ORC before storing the data in Amazon S3. https://docs.aws.amazon.com/firehose/latest/dev/record-format-conversion.html With that LEAST development effort, why do we need to use Lambda additionally? :D