Exam Professional Data Engineer topic 1 question 231 discussion

If an Airflow worker pod is evicted, all task instances running on that pod are interrupted, and later marked as failed by Airflow. The majority of issues with worker pod evictions happen because of out-of-memory situations in workers. You might want to: - (D) Increase the memory available to workers. - (C) Reduce worker concurrency. In this way, a single worker handles fewer tasks at once. This provides more memory or storage to each individual task. If you change worker concurrency, you might also want to increase the maximum number of workers. In this way, the number of tasks that your environment can handle at once stays the same. For example, if you reduce worker Concurrency from 12 to 6, you might want to double the maximum number of workers. Source: https://cloud.google.com/composer/docs/composer-2/optimize-environments

CD On the Monitoring dashboard, in the Workers section, observe the Worker Pods evictions graphs for your environment. The Total workers memory usage graph shows a total perspective of the environment. A single worker can still exceed the memory limit, even if the memory utilization is healthy at the environment level. According to your observations, you might want to: - Increase the memory available to workers. - Reduce worker concurrency. In this way, a single worker handles fewer tasks at once. This provides more memory or storage to each individual task. If you change worker concurrency, you might also want to increase the maximum number of workers. In this way, the number of tasks that your environment can handle at once stays the same. For example, if you reduce worker Concurrency from 12 to 6, you might want to double the maximum number of workers.

Answer is B,D

Answer C,D According to your observations, you might want to: Increase the memory available to workers. Reduce worker concurrency. In this way, a single worker handles fewer tasks at once. This provides more memory or storage to each individual task. If you change worker concurrency, you might also want to increase the maximum number of workers. In this way, the number of tasks that your environment can handle at once stays the same. For example, if you reduce worker Concurrency from 12 to 6, you might want to double the maximum number of workers.

C. Increase the maximum number of workers and reduce worker concurrency. Most Voted D. Increase the memory available to the Airflow workers.

If an Airflow worker pod is evicted, all task instances running on that pod are interrupted, and later marked as failed by Airflow. The majority of issues with worker pod evictions happen because of out-of-memory situations in workers. You might want to: - Increase the memory available to workers. - Reduce worker concurrency. In this way, a single worker handles fewer tasks at once. This provides more memory or storage to each individual task. If you change worker concurrency, you might also want to increase the maximum number of workers. In this way, the number of tasks that your environment can handle at once stays the same. For example, if you reduce worker Concurrency from 12 to 6, you might want to double the maximum number of workers. Source: https://cloud.google.com/composer/docs/composer-2/optimize-environments

CD It is clear

C & D to me

C and D Check ref for memory optimization - https://cloud.google.com/composer/docs/composer-2/optimize-environments

B&D See this- https://cloud.google.com/composer/docs/composer-2/troubleshooting-dags#task-fails-without-logs go through the suggested fixes for If there are airflow-worker pods that show Evicted

C and D

B&D: B : - Scaling up the environment size can provide more resources, including memory, to the Airflow workers. If worker pod evictions are occurring due to insufficient memory, increasing the environment size to allocate more resources could alleviate the problem and improve the stability of your data processing jobs. D: - Increase the memory available to the Airflow workers. - Directly increasing the memory allocation for Airflow workers can address the issue of high memory usage and worker pod evictions. More memory per worker means that each worker can handle more demanding tasks or a higher volume of tasks without running out of memory.