Exam DP-203 topic 2 question 43 discussion

1) Incremental Load 2) Tumbling Window Seems like you could go with either Schedule trigger or Tumbling Window here. I would use the latter option, and pass the windowStart system variable to the pipeline as a parameter, allowing me to more easily navigate to the proper directory in the storage account.

Since, we are loading NEW data and not going back in time, it should be Schedule as we are scheduling it for every 1 hour in the future. It would've been Tumbling if we scheduled it for every 1 hour in the past.

Got this question today on the exam

Correct, chatgpt : For the scenario described, to load data from an Azure Storage account to an Azure Data Lake hourly and to minimize load times and costs, you would configure the Azure Data Factory solution as follows: - **Load methodology**: Incremental Load - Because you are loading new data every hour, and the goal is to minimize the load times and costs, you would incrementally load only the new data that has arrived since the last load. - **Trigger**: Tumbling window - This trigger is suitable for fixed-duration, repeating intervals in Azure Data Factory, which fits the requirement of loading data hourly. Using a tumbling window trigger ensures that each window of time is processed once and only once, and by doing an incremental load, you are only processing the new data that has appeared since the last hour, rather than reprocessing all existing data.

I think one thing very important here is that Tumbling window manages state between runs, that means that it will not be count twice.

From Azure Data Factory Studio when you create a new trigger, you can choise TYPE in ('Schedule', 'Tumbling window', 'Storage events', 'Custom events'). We should exclude "Fixed Schedule" becuase of 'fixed'! :) So my final answer will be Incremental Load and Tumbling Window.

Hi there

To minimize load times and costs for loading new data from the storage account to an Azure Data Lake once hourly, you should configure the solution to use incremental load and a trigger based on new files arriving. Load methodology: With 200,000 new files generated daily, a full load every hour could be time-consuming and expensive. Incremental load is a better option in this scenario because it only loads new or changed data since the last successful execution of the pipeline, which can significantly reduce load times and costs. Trigger: A trigger based on new files arriving is the most efficient option because it only runs the pipeline when new files are detected in the storage account. This avoids unnecessary pipeline executions and reduces costs. A fixed schedule trigger runs the pipeline at fixed intervals, regardless of whether there is new data to process or not. A tumbling window trigger runs the pipeline at specified intervals, but still processes all data within the window, regardless of whether there is new data or not. Therefore, a new file trigger is the best option in this scenario.

A schedule for an activity creates a series of tumbling windows with in the pipeline start and end times I think is "Fixed schedule" because "Tumbling windows" are more related to streams analytics questions according to MS doc. https://learn.microsoft.com/en-us/azure/data-factory/v1/data-factory-scheduling-and-execution

1) Incremental Load 2) Tumbling Window

With Scheduled trigger executions can overlaps if the process does not finish within 1 hour, Tumbling window is better, with concurrency setting it can allow only one ongoing execution.

both Tumbling Window and Schedule trigger will reach the goal. Which one is more cost effective?

why not schedule trigger?

incremental, fixed schedule every hour.

As a input we are receiving csv files so why not trigger mechanism to the pipeline when file arrived.