A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?
ofed
1 month agosturcu
2 months agoEertyy
2 months, 3 weeks agoazurearch
3 months agoazurearch
3 months agoazurearch
3 months agocotardo2077
3 months, 1 week agoEertyy
3 months, 1 week agoasmayassineg
4 months, 1 week ago