Exam AI-100 All Questions

View all questions & answers for the AI-100 exam

Exam AI-100 topic 2 question 59 discussion

Actual exam question from Microsoft's AI-100

Question #: 59
Topic #: 2

Your company has a data team of Scala and R experts.
You plan to ingest data from multiple Apache Kafka streams.
You need to recommend a processing technology to broker messages at scale from Kafka streams to Azure Storage.
What should you recommend?

A. Azure Databricks
B. Azure Functions
C. Azure HDInsight with Apache Storm
D. Azure HDInsight with Microsoft Machine Learning Server

Show Suggested Answer

Suggested Answer: C 🗳️
Reference:
https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-streaming-at-scale-overview?toc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%
2Fhdinsight%2Fhadoop%2FTOC.json&bc=https%3A%2F%2Fdocs.microsoft.com%2Fen-us%2Fazure%2Fbread%2Ftoc.json

by Karl at Sept. 17, 2019, 8:15 a.m.

Comments

Submit Cancel

Karl

Highly Voted 5 years, 10 months ago

I think it is Databricks because of Scala and R support

upvoted 18 times

giusecozza

5 years, 2 months ago

Although it could be a valid option, I found that Storm supports multi-language: https://storm.apache.org/about/multi-language.html Hard to say what's the best answer...

upvoted 3 times

...

UpsetUser

4 years, 6 months ago

Scala and R support is also there with HDinsight . Your logic does not holds.

upvoted 1 times

Messatsu

4 years ago

Not sure that HDinsight supports R (for Scala it does) : https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-overview

upvoted 1 times

...

abbam

Highly Voted 5 years, 3 months ago

Storm supports C# & Java Databricks supports C#/F#, Java, Python, R, Scala https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing

upvoted 6 times

zemplenib

5 years, 1 month ago

Databricks, as Storm do not support either scala or R https://docs.microsoft.com/en-us/azure/hdinsight/storm/apache-storm-overview

upvoted 3 times

LeoBritto

5 years ago

Storm supports scala and other languages https://www.educba.com/apache-storm-vs-apache-spark/ so HDInsight with Apache Storm is the correct answer

upvoted 2 times

...

rveney

Most Recent 2 years, 1 month ago

C. Azure HDInsight with Apache Storm Explanation: To broker messages at scale from Kafka streams to Azure Storage, you can recommend using Azure HDInsight with Apache Storm. Apache Storm is a distributed real-time stream processing system that can handle high-volume data streams. It provides fault-tolerant processing and can integrate well with Apache Kafka for data ingestion.

upvoted 1 times

...

jdev

4 years, 2 months ago

For real-time Kafka data streams - HD Insight with Apache Storm is best. The key context here is 'streams'.

upvoted 1 times

lollo1234

4 years, 1 month ago

Kafka as an input is not supported in Storm, but it is in Databricks. So I would say Databricks is the answer https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing

upvoted 1 times

lollo1234

4 years, 1 month ago

You can also see that in the Sinks or Outputs, Azure Storage is supported in Databricks while it's not in Storm

upvoted 1 times

...

jdev

4 years, 2 months ago

HD Insight with apache Strom is preferred - for processing real-time data streams.

upvoted 1 times

...

SGZoom

4 years, 6 months ago

C is the correct Ans, because question does NOT mention utilization of your data team R and Scala expertise (believe they are meant to distract you). Question is to suggest technology to “broker” messages and Apache Storm is the right choice since it is meant for ingestion / streaming. DataBricks is an “Data Analytics” platform requiring ingestion thru Data factory / Kafka / Event Hub, etc.. so DataBricks is incorrect

upvoted 2 times

Cornholioz

4 years, 5 months ago

Disagree. There are just 3 lines in the question to make the distinction and you suggest we ignore the 1st line? About your 2nd comment on brokering messages, it means "integration capabilities". Having the capability to Input and/or Sink. Both are Real-time stream processing techs to consume messages from queue etc. Both Databricks (on Spark) and HDInsight with Storm can "Sink" Kafka while only Databricks has a direct input integration capability. HDInsight with Storm does not support Scala and R... I think this does need consideration. Databricks is the answer IMO Read again in detail: https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing

upvoted 1 times

Cornholioz

4 years, 5 months ago

Going back on my inference. Databricks won't work here as it is a Spark-based analytics solution that allows you to create big data pipelines. Moreover, it runs on Spark clusters and not Hadoop clusters. Another reliable & verified practice test states HDInsight as the solution. It discards ML Service but the option doesn't combine it with HDInsight. Given this, I'll go with HDInsight + Storm

upvoted 2 times

...

UpsetUser

4 years, 6 months ago

I think C and A both could be possible answers,, but since here big data is not mentioned, so we can eliminate A. So answer is correct : C

upvoted 1 times

...

vilas94

4 years, 9 months ago

Answer is Azure Databricks which supports R and Scala , which is not supported by Apache Storm

upvoted 1 times

...

CeliaZhou

4 years, 9 months ago

I believe the answer is A: Azure Databricks, according to this article from Microsoft: https://docs.microsoft.com/en-us/azure/architecture/data-guide/technology-choices/stream-processing Apache Spark in Azure Databricks: C#/F#, Java, Python, R, Scala Azure Functions:C#, F#, Java, Node.js, Python HDInsight with Storm: C#, Java

upvoted 2 times

...

sayak17

4 years, 10 months ago

Apache Storm with Apache Kafka on HDInsight seems correct as per https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-storm-with-kafka And yes Apache Spark supports all languages mentioned in question https://spark.apache.org/ However Azure Databricks can also be a answer because of: https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/kafka https://docs.databricks.com/spark/latest/sparkr/index.html https://docs.databricks.com/languages/scala.html So not sure what to pick here

upvoted 1 times

...

Nova077

4 years, 10 months ago

1. Databricks can interface with Kafka. It supports R and Scala https://docs.databricks.com/spark/latest/structured-streaming/kafka.html https://databricks.com/r-programming 2. Storm and Kafka works together. HD insights also supports R and Scala https://docs.microsoft.com/en-us/azure/hdinsight/hdinsight-apache-storm-with-kafka Can't find any article which specifically says that storm supports both R and Scala. 3. Ruling out Machine learning R server. This is primarily used if someone has need to do things on R So my vote is for Databricks

upvoted 1 times

...

princesskay

5 years, 3 months ago

https://docs.microsoft.com/en-us/azure/databricks/spark/latest/structured-streaming/kafka

upvoted 1 times

...

SepidehJah

5 years, 5 months ago

Apache Spark in Azure Databricks https://docs.microsoft.com/en-us/azure/azure-databricks/what-is-azure-databricks

upvoted 1 times

...

MK1977

5 years, 7 months ago

https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-introduction

upvoted 1 times

...

Swatishri

5 years, 7 months ago

Option C,HDInsight with Apache Storm is the right answer as has been shown in the solution - correct

upvoted 4 times

...