A Structured Streaming job deployed to production has been experiencing delays during peak hours of the day. At present, during normal execution, each microbatch of data is processed in less than 3 seconds. During peak hours of the day, execution time for each microbatch becomes very inconsistent, sometimes exceeding 30 seconds. The streaming write is currently configured with a trigger interval of 10 seconds.
Holding all other variables constant and assuming records need to be processed in less than 10 seconds, which adjustment will meet the requirement?
imatheushenrique
1 month, 3 weeks agoojudz08
5 months, 2 weeks agoDAN_H
5 months, 3 weeks agokz_data
6 months, 2 weeks agoRafaelCFC
6 months, 3 weeks agoervinshang
7 months agoofed
8 months, 3 weeks agosturcu
9 months, 2 weeks agoEertyy
10 months, 1 week agoazurearch
10 months, 3 weeks agoazurearch
10 months, 3 weeks agoazurearch
10 months, 3 weeks agocotardo2077
10 months, 3 weeks agoEertyy
11 months agoasmayassineg
11 months, 4 weeks ago