You are responsible for writing your company's ETL pipelines to run on an Apache Hadoop cluster. The pipeline will require some checkpointing and splitting pipelines. Which method should you use to write the pipelines?
Réponse : C
Explication :
La bonne réponse ici est C. Java using MapReduce.
Explication rapide :
Tu parles de besoin de "checkpointing" et "splitting pipelines".
PigLatin (Pig) et HiveQL (Hive) sont des langages déclaratifs (de plus haut niveau), pas faits pour un contrôle précis sur la manière dont les jobs se séquencent, checkpointent ou s'articulent.
MapReduce en Java te donne un contrôle total sur :
L'ordonnancement précis.
Le checkpointing manuel.
Le découpage et le chaînage des étapes.
Python avec MapReduce est possible mais beaucoup moins natif ; Hadoop est conçu principalement pour Java MapReduce, donc en Python ce serait plus compliqué, plus fragile et moins performant.
Résumé rapide pour que tu t'en souviennes : 👉 Besoin de contrôle précis sur la logique ETL complexe = MapReduce en Java.
Writing your pipeline in Java using MapReduce allows you to implement these custom controls and fine-tune the execution, ensuring robust and manageable ETL processes on your Hadoop cluster
Pig Latin supports both splitting pipelines and checkpointing, allowing users to create complex data processing workflows with the ability to restart from specific points in the pipeline if necessary.
I would go to A.
C, D are similar. So both are excluded. B, Hive is actually a data warehouse system. I don't use Apache Pig. But, BCD are wrong. Then A should be correct.
This answer depends which language you are comfortable with.
Hadoop is your framework, where mapReduce is your Native programming model in JAVA, which is designed to scale, parallel processing, restart pipeline from any checkpoint etc. , So if you are comfortable with JAVA, you can customize your checkpoint at lowlevel in better way. otherwise, choose PIG which is another programming concept run over JAVA but then you need to learn this also, if not choose python as it can be deployed with hadoop because hadoop has been making updates for python clients regularly.
Option C: is the best one.
C. Java using MapReduce or D. Python using MapReduce
Apache Hadoop is a distributed computing framework that allows you to process large datasets using the MapReduce programming model. There are several options for writing ETL pipelines to run on a Hadoop cluster, but the most common are using Java or Python with the MapReduce programming model.
A. PigLatin using Pig is a high-level data flow language that is used to create ETL pipelines. Pig is built on top of Hadoop, and it allows you to write scripts in PigLatin, a SQL-like language that is used to process data in Hadoop. Pig is a simpler option than MapReduce but it lacks some capabilities like the control over low-level data manipulation operations.
B. HiveQL using Hive is a SQL-like language for querying and managing large datasets stored in Hadoop's distributed file system. Hive is built on top of Hadoop and it provides an SQL-like interface for querying data stored in Hadoop. Hive is more suitable for querying and managing large datasets stored in Hadoop than for ETL pipelines.
Both Java and Python using MapReduce provide low-level control over data manipulation operations, and they allow you to write custom mapper and reducer functions that can be used to process data in a Hadoop cluster. The choice between Java and Python will depend on the development team's expertise and preference.
It has to be C
because while Pig can be used to simplify the writing of complex data transformation tasks and can store intermediate results, it doesn't provide the detailed control over checkpointing and pipeline splitting in the way that is typically implied by those terms.
also, while one can write MapReduce jobs in languages other than Java (like Python) using Hadoop Streaming or other similar APIs, it may not be as efficient or as seamless as using Java due to the JVM-native nature of Hadoop.
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
[Removed]
Highly Voted 5 years, 1 month ago[Removed]
Highly Voted 5 years, 1 month agooliiivier
Most Recent 3 days, 16 hours agoParandhaman_Margan
1 month, 2 weeks agodesertlotus1211
1 month, 2 weeks agogrshankar9
3 months, 1 week agoSamuelTsch
6 months, 1 week agoAnonymousPanda
1 year, 8 months agoOleksandr0501
2 years agojuliobs
2 years, 1 month agomusumusu
2 years, 2 months agosamdhimal
2 years, 3 months agosamdhimal
2 years, 3 months agocetanx
1 year, 11 months agoKoushik25sep
2 years, 7 months agoBigDataBB
3 years, 2 months agorbeeraka
3 years, 3 months agodavidqianwen
3 years, 3 months agomaddy5835
3 years, 6 months ago