Exam Certified Associate Developer for Apache Spark topic 1 question 56 discussion

Actual exam question from Databricks's Certified Associate Developer for Apache Spark

Question #: 56
Topic #: 1

[All Certified Associate Developer for Apache Spark Questions]

The code block shown below contains an error. The code block is intended to return a new DataFrame that is the result of a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF. Identify the error.
Code block:
storesDF.unionByName(acquiredStoresDF)

A. There is no DataFrame.unionByName() operation – the concat() operation should be used instead with both DataFrames as arguments.
B. There are no key columns specified – similar column names should be the second argument.
C. The DataFrame.unionByName() operation does not union DataFrames based on column position – it uses column name instead.
D. The unionByName() operation is a standalone operation rather than a method of DataFrame – it should have both DataFrames as arguments.
E. There are no column positions specified – the desired column positions should be the second argument.

Show Suggested Answer

Suggested Answer: C 🗳️

by 4be8126 at May 3, 2023, 11:54 a.m.

Comments

Submit Cancel

juliom6

8 months, 3 weeks ago

Selected Answer: C

C is correct according to documentation: https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.unionByName.html "The difference between this function and union() is that this function resolves columns by name (not by position)"

upvoted 1 times

...

newusername

9 months ago

Selected Answer: C

C is correct - https://spark.apache.org/docs/3.1.1/api/python/reference/api/pyspark.sql.DataFrame.unionByName.html

upvoted 1 times

...

4be8126

1 year, 3 months ago

Selected Answer: C

The error in the code block is: C. The DataFrame.unionByName() operation does not union DataFrames based on column position – it uses column name instead. The unionByName() operation performs a position-wise union based on column names, not based on column positions. Therefore, the error in the code block is that the intended operation should be union(), which performs a position-wise union regardless of column names. The correct code block to perform a position-wise union between DataFrame storesDF and DataFrame acquiredStoresDF would be: storesDF.union(acquiredStoresDF)

upvoted 3 times

...