Exam AWS Certified Data Analytics - Specialty topic 1 question 93 discussion

A is wrong, read that locks means: Locking is a protection mechanism that controls how many sessions can access a table at the same time. To solve a locking problem, identify the session (PID) that is holding the lock and then terminate the session. If the session doesn't terminate, reboot your cluster. So... If you increase concurrency, you increase a number of sessions that makes things worse. Hence C

Not sure but should be A for number of re-tries and increased concurrency.

D makes sense, increasing timeout leaves time for locks to resolve, also, job concurrency should be 1 for COPY commands, it doesnt make sense to have multiple ones running.

A is the least wrong, since it theoretically will increase fault tolerance. But it's an incorrect by design. Increasing concurrency will lead to an increase in locks, leading to more timeouts, leading to more retries, so once we reach the max retries, some data still might be missed. The right answer would be to set concurrency to 1 and increase retries while decreasing timeouts. B, C and D: Keep the number of retries at 0. - whatever the other configuration you set, when the job fails due to a lock that job will never be retried and data will be missed 100%.

Option C is the best solution to optimize fault tolerance and improve data availability. Keeping the number of retries at 0 will ensure that the job does not attempt to retry if it fails, which may cause further locks and missed data. Decreasing the timeout value will ensure that the job fails quickly if it is unable to complete the COPY command. Keeping the job concurrency at 1 will ensure that the job runs on a single node, which will reduce the chances of locks occurring.

ets go in reverse to answer this. D is out - Glue job that is scheduled to run every 5 minutes so timeout can't increase C is out - locks will still exist when spike is high B is out - jobs will fail due to decreased timeout when there is no concurrency.

lets go in reverse to answer this. D is out - Glue job that is scheduled to run every 5 minutes C is out - locks will still exist B is out - jobs will due to decreased timeout when there is no concurrency.

Locks generally happen in case of DDL statements. Refer this link - https://docs.aws.amazon.com/redshift/latest/dg/r_LOCK.html Question mentions about Glue job reading data every 5 mins and inserting it in Redshift. In case of lock present on table, the job would timeout. If we don't retry, there is obviously data loss. Job generally completes in few seconds so decreasing timeout definitely makes sense and everyone agree. We definitely need to increase retry count so if lock exist, job will retry with same data insertion in hope that lock is released. Even if lock is not released in next attempt and by the time next glue job sequence will pick the data for insertion and if we don't increase concurrency, initial failed job would lead to data loss. So important to increase concurrency as well so that previous instance and current instance of glue job can run together.

Suggested Concurrency is 1

D. Increasing the timeout is not conflict with "typically finish with a few seconds", and it can be helpful in the peak time

The best way to solve this problem is to increase re tries, wait less for a re try to complete and concurrently copy data to redshift. So ans is "A".

Selected Answer: A

A looks right. Timeout should be reduced and retries should be increased. Also, concurrency should be increased so that the copy operation completes quickly. Concurrency will not cause locking as the copy command running in concurrency mode should be part of the same transaction. Hence there is no question of locking.

Increasing number of retries will add stress on locks when Copy command completes in few seconds, there is no need to wait till 5 min rather, TimeOut should be decreased. Concurrency at 1 is perfect

Agree with NTP's first answer, and added some comments. The question requires "Data avaibility" and "fault tolerant", so data loss should be avoid. I aggree with your previoud answer: 1. COPY finish within a few seconds, so decrease timeout and increase retries. It can reduce the table lock as you said. 2. increase concurrent jobs, it can execute COPY and QUERY at the same time, if only they are not on the same table. And if only most of the COPY can finish in seconds, there will be no LOCK because of this.

If locks exists, reads/writes are made to wait till the session holding the lock completes Ref: https://docs.aws.amazon.com/redshift/latest/dg/r_LOCK.html Therefore, increasing timeout will help. Keeping concurrency at 1 is recommended to avoid multiple COPY command and prevent multiple instances of glue task. Maybe answer = D

A COPY commands typically finish within a few seconds -> decrease timeout to 30 seconds. If there is a lock, this task will fail and release a lock, but we still loose the data. So we need to add more retries, e.g. 2 retries. If we add so many retires, there could be concurrent jobs, concurrent COPY is not recommended but this case of lock happens many times continuously is rare