Exam Certified Associate Developer for Apache Spark topic 1 question 70 discussion

Actual exam question from Databricks's Certified Associate Developer for Apache Spark

Question #: 70
Topic #: 1

[All Certified Associate Developer for Apache Spark Questions]

The code block shown below contains an error. The code block is intended to return a new DataFrame where column managerName from DataFrame storesDF is split at the space character into column managerFirstName and column managerLastName. Identify the error.

A sample of DataFrame storesDF is displayed below:

Code block:

storesDF.withColumn("managerFirstName", col("managerName").split(" ").getItem(0))
.withColumn("managerLastName", col("managerName").split(" ").getItem(1))

A. The index values of 0 and 1 are not correct – they should be 1 and 2, respectively.
B. The index values of 0 and 1 should be provided as second arguments to the split() operation rather than indexing the result.
C. The split() operation comes from the imported functions object. It accepts a string column name and split character as arguments. It is not a method of a Column object.
D. The split() operation comes from the imported functions object. It accepts a Column object and split character as arguments. It is not a method of a Column object.
E. The withColumn operation cannot be called twice in a row.

Show Suggested Answer

Suggested Answer: D 🗳️

by zozoshanky at July 30, 2023, 7:35 p.m.

Comments

Submit Cancel

tmz1

6 months, 1 week ago

In my opinion both C and D are correct because split function from package pyspark.sql.functions accepts both column object or string as a first parameter

upvoted 2 times

...

Sowwy1

1 year, 3 months ago

D. The split() operation comes from the imported functions object. It accepts a Column object and split character as arguments. It is not a method of a Column object.

upvoted 1 times

...

Ahlo

1 year, 5 months ago

Answer C pyspark.sql.functions provides a function split() to split DataFrame string Column into multiple columns. https://sparkbyexamples.com/pyspark/pyspark-split-dataframe-column-into-multiple-columns/

upvoted 1 times

...

newusername

1 year, 9 months ago

Selected Answer: C

I think it is C data = [ ("John Smith",), ("Jane Doe",), ("Mike Johnson",) ] df = spark.createDataFrame(data, ["managerName"]) df.show() df = df.withColumn("managerFirstName", split(col("managerName"), " ").getItem(0)) \ .withColumn("managerLastName", split(col("managerName"), " ").getItem(1)) df.show()

upvoted 2 times