the answer given is correct:
Maximum concurrent runs: Set to 1. There must be only one instance of each query concurrently active.
Retries: Set to Unlimited.
https://docs.databricks.com/en/structured-streaming/query-recovery.html
New Job Cluster: Using a new job cluster ensures that the compute resources are appropriately sized and dedicated to the job, which can help in managing costs and performance more effectively than using an existing all-purpose cluster.
Retries: Unlimited: Setting retries to unlimited ensures that the job will automatically recover from failures by retrying until it succeeds.
Maximum Concurrent Runs: 1: Limiting the maximum concurrent runs to 1 prevents multiple instances of the job from running simultaneously, which can help in controlling costs and avoiding resource contention.
Databricks recommends using jobs compute instead of all-purpose compute when scheduling workflows, as it helps in managing resources more efficiently and reduces costs.
https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/production
https://learn.microsoft.com/en-us/azure/databricks/jobs/continuous
When running Structured Streaming jobs in production, you want:
Automatic failure recovery → Requires setting Retries: Unlimited
Efficient cost control → Use a New Job Cluster, which auto-terminates after job completion
Concurrency control → Maximum Concurrent Runs: 1 prevents overlapping runs, which can corrupt streaming state or double-process data
Use databricks jobs as it as native integration with Streaming use case. See the example Job here https://docs.databricks.com/en/structured-streaming/query-recovery.html#configure-structured-streaming-jobs-to-restart-streaming-queries-on-failure
A voting comment increases the vote count for the chosen answer by one.
Upvoting a comment with a selected answer will also increase the vote count towards that answer by one.
So if you see a comment that you already agree with, you can upvote it instead of posting a new comment.
8605246
Highly Voted 2 years ago363c4c5
Most Recent 1 month ago79f0e18
1 month, 1 week agoKadELbied
3 months, 1 week agocodebender
4 months, 1 week agoEelkeV
6 months, 1 week agoakashdesarda
10 months, 2 weeks agoimatheushenrique
1 year, 2 months agoimatheushenrique
1 year, 2 months agojuliom6
1 year, 4 months agoAziLa
1 year, 6 months agoJay_98_11
1 year, 7 months agokz_data
1 year, 7 months agosturcu
1 year, 9 months ago