Exam Certified Data Engineer Associate topic 1 question 18 discussion

Actual exam question from Databricks's Certified Data Engineer Associate

Question #: 18
Topic #: 1

[All Certified Data Engineer Associate Questions]

A data analyst has a series of queries in a SQL program. The data analyst wants this program to run every day. They only want the final query in the program to run on Sundays. They ask for help from the data engineering team to complete this task.
Which of the following approaches could be used by the data engineering team to complete this task?

A. They could submit a feature request with Databricks to add this functionality.
B. They could wrap the queries using PySpark and use Python’s control flow system to determine when to run the final query.
C. They could only run the entire program on Sundays.
D. They could automatically restrict access to the source table in the final query so that it is only accessible on Sundays.
E. They could redesign the data model to separate the data used in the final query into a new table.

Show Suggested Answer

Suggested Answer: B 🗳️

by knivesz at April 4, 2023, 1:49 a.m.

Comments

Submit Cancel

Atnafu

Highly Voted 2 years ago

B The answer is B. Option A: Submitting a feature request with Databricks to add this functionality is not a feasible solution because it would require Databricks to implement new functionality. Option C: Only running the entire program on Sundays would be inconvenient for the data analyst because they would have to remember to run the program on Sundays. Option D: Automatically restricting access to the source table in the final query so that it is only accessible on Sundays would be difficult to implement and would not be a reliable solution. Option E: Redesigning the data model to separate the data used in the final query into a new table would be a major undertaking and would not be a feasible solution for this specific problem. Therefore, the only feasible solution to the problem is to wrap the queries using PySpark and use Python's control flow system to determine when to run the final query. python code

upvoted 11 times

vibha2119

1 year, 2 months ago

Also to add for option C is: the requirement says: data analysts wants to run the sql program every day, but only the final query to run on sundays. So the option c violates the requirements

upvoted 1 times

...

SoumyaHK

Most Recent 3 months, 1 week ago

Selected Answer: B

The response is B

upvoted 1 times

...

806e7d2

8 months, 3 weeks ago

Selected Answer: B

Wrapping the SQL program in PySpark allows the data engineering team to leverage Python’s control flow (e.g., if statements) to determine when the final query should execute. For example: The PySpark program can check the current day using Python's datetime module. If the current day is Sunday, it will execute all queries, including the final one. If it’s not Sunday, the program will skip the final query. This approach ensures flexibility and allows precise control over which queries run on which days, meeting the data analyst's requirements.

upvoted 4 times

...

80370eb

12 months ago

Selected Answer: B

B. They could wrap the queries using PySpark and use Python’s control flow system to determine when to run the final query. This approach involves using PySpark to control the execution flow based on the day of the week, allowing the final query to execute conditionally while the other queries run daily.

upvoted 1 times

...