A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?
RafaelCFC
Highly Voted 1 year, 6 months agoarekm
6 months, 1 week agoKadELbied
Most Recent 2 months agoarekm
6 months, 1 week agoASRCA
6 months, 1 week agoarekm
6 months, 1 week agoThameur01
7 months agowdeleersnyder
11 months, 1 week agowdeleersnyder
11 months, 1 week agoimatheushenrique
1 year, 1 month agoojudz08
1 year, 4 months agoDAN_H
1 year, 5 months agokz_data
1 year, 6 months agoervinshang
1 year, 6 months agoofed
1 year, 8 months agosturcu
1 year, 9 months agoEertyy
1 year, 9 months agoazurearch
1 year, 10 months agoazurearch
1 year, 10 months agoazurearch
1 year, 10 months ago