Exam AWS Certified Big Data - Specialty topic 1 question 39 discussion

The Answer is D. S3Distcp has native capability to combine multiple small files into large files and does not require any coding. So, minimal change to existing processes. https://aws.amazon.com/blogs/big-data/seven-tips-for-using-s3distcp-on-amazon-emr-to-move-data-efficiently-between-hdfs-and-amazon-s3/#5 Also through a process of elimination, we can exclude A) DynamoDB is not a good service for large scale data analytics. C) Moving from EMR to ElasticSearch requires significant changes to existing processes and applications. B) Moving from MR to Spark requires significant changes to existing processes and applications. E) Moving from S3 to firehose requires significant changes to existing processes and applications.

D. Hadoop is optimized for reading a fewer number of large files rather than many small files, whether from S3 or HDFS. You can use S3DistCp to aggregate small files into fewer large files of a size that you choose, which can optimize your analysis for both performance and cost. https://aws.amazon.com/blogs/big-data/seven-tips-for-using-s3distcp-on-amazon-emr-to-move-data-efficiently-between-hdfs-and-amazon-s3/#5

D is correct. You can aggregate files using DistCp command. Hadoop is optimized for reading a fewer number of large files rather than many small files

The choice is really between B and D. With B- Spark has the capability to improve performance over the existing AMR setup if lets says its using Hive or Pig instead. However it still needs to deal with large number of small data sets to process which is really the issue. So at the end of the day it won't solve the root cause of the issue which is about I/O not about compute. With D- This is the simplest solution that doesn't need the businesses to change the way they ingest those files into the organization today. It's a change that solves the root cause of the problem which is I/O yet keeps the overall process and flow unchanged. So the correct answer is D.

It is B Wrong for : A- is wrong because new function C- Lambda with elastic search - why something new D- data is already there, its just gorwing the issue is not how to store or move data - but how to process E- a big change for all

my selection D

Answer us D https://aws.amazon.com/blogs/big-data/seven-tips-for-using-s3distcp-on-amazon-emr-to-move-data-efficiently-between-hdfs-and-amazon-s3/

I support B since you can add a custom Jar so I believe you could use your own map-reduce app there

I would choose D as it is simple solution to improve performance with minimal changes to the existing process.

D looks reasonable. Using S3DistCp, you can efficiently copy large amounts of data from Amazon S3 into HDFS where it can be processed by subsequent steps in your Amazon EMR cluster.

What is the right answer for this? A or D?

Option A, require EMR to access DynamoDB to get the index files which needs change in current setup. Using Option D, you can add a S3DistCp step in EMR job which is minimal change. Option D is correct

A,E & C are not minimal changes. the question says minimal changes to existing processes and application. Answer should be D or B.

Any thought on B?

A is correct

a require some additional services. the question asks with minimal changes.

Thoughts on A?