Exam DP-700 topic 2 question 39 discussion

Eventstream has 1MB max size limitation, streaming dataflow has been retired in Fabric. We just can choose between Apache Spark Structured Streaming and data pileline(normally it will be used for batch loging solution not Fabric streaming solution). I cannot understand if all of 3 sources have to match Apache Spark Structured Streaming. I have to make these answers: Source 1, semi-structured, contains a bigint column, Apache Spark Structured Streaming can match it if we choose longType. Source 2 is hard to make decision because lost details, I tend to choose data pipleline. Source 3 has image, we can use copy activity in data pipeline and set file format to Binary.In order to match "The solution must minimize development effort."

| Source | Data Type | Message Size | Recommended Option | Justification | | ----------- | -------------------------- | ------------ | --------------------------------------- | ---------------------------------------------------------------------------- | | **Source1** | Semi-structured (bigint) | **10 MB** | ✅ **Apache Spark Structured Streaming** | Streaming Dataflow can't reliably handle >4 MB; Spark handles large messages | | **Source2** | Structured (19 columns) | **25 MB** | ✅ **Apache Spark Structured Streaming** | Too large for Streaming Dataflow; Spark is built for this scale | | **Source3** | Unstructured (with images) | **5 MB** | ✅ **Apache Spark Structured Streaming** | Binary data + medium size → Spark is the only suitable and scalable option |

Como o objetivo é minimizar o esforço de desenvolvimento, a melhor escolha costuma ser: ✅ A streaming dataflow – quando streaming é necessário mas com pouco código. ❌ Apache Spark Structured Streaming – mais código. ❌ An eventstream – requer integração adicional. ❌ A data pipeline – bom para batch, não para streaming.

My selection: - An eventstream - A streaming dataflow - Apache Spark Structured Streaming

Source1: An eventstream Source2: A streaming dataflow Source3: Apache Spark Structured Streaming Confirmed with Chatgpt

Source1 Apache Spark Structured Streaming Source2 An eventstream Source3 Apache Spark Structured Streaming

This is such a tricky question. Usually, data from streaming source is not so big. If data size is smaller than 1MB => event stream however data size > 1 MB => Apache Spark structured streaming. Data pipeline doesn't support streaming.

eventstream has a message limit of 1 MB. Data pipeline for batch processing, streaming dataflow is retired. should be Spark Structured Streaming for all.

Eventstream: Max message size 1 MB https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/overview?tabs=enhancedcapabilities#limitations

1. **Source1 (Semi-structured data, 10 MB)**: - **Solution**: **Eventstream** - Eventstream is optimized for handling semi-structured data and can efficiently process messages with a `bigint` column, while providing real-time streaming capabilities. 2. **Source2 (Structured data, 25 MB)**: - **Solution**: **Stream Dataflow** - Stream Dataflow is an appropriate choice for structured data. It allows easy ingestion and transformation of structured datasets, minimizing development effort. 3. **Source3 (Unstructured data with images, 5 MB)**: - **Solution**: **Spark Structured Streaming** - Spark Structured Streaming is ideal for processing unstructured data, such as images, in real-time. It provides flexibility for working with diverse data formats.

I think both Apache Spark Structured Streaming and data pipeline(if we think it can play role of Fabric streaming solution) can match it. Howerver, question said "The solution must minimize development effort", data pipeline is better? Source2 didn't give enough details, if this structured data is database and configured with CDC, then eventstream is option. Source 3 should be data pipeline.