Exam Certified Associate Developer for Apache Spark topic 1 question 25 discussion

Both C or D are correct. Function split accepts both col and str. https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.split.html?highlight=split#pyspark.sql.functions.split

Only C is correct pyspark.sql.functions.split(str, pattern, limit=-1) The exam is about Spark 3.0, the option to also use col is now existing (3.5, but not in 3.0)

Both C or D work in Spark 3.5.1, but C is probably better for backward compatibility. See code example below: from pyspark.sql import SparkSession from pyspark.sql.functions import split, col spark = SparkSession.builder.appName("MyApp").getOrCreate() data = [ (0, True, 10020, "VALUE_MEDIUM"), (1, True, 10050, "MAINSTREAM_SMALL"), (2, False, 10070, "PREMIUM_LARGE"), ] storesDF = spark.createDataFrame(data, ["storeID", "open", "openDate", "storeCategory"]) (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]).withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])).show() (storesDF.withColumn("storeValueCategory", split("storeCategory", "_")[0]).withColumn("storeSizeCategory", split("storeCategory", "_")[1])).show()

C you can check, by running the code below: from pyspark.sql import SparkSession # Initialize Spark session spark = SparkSession.builder.appName("split_test").getOrCreate() # Create synthetic data data = [ {"storeCategory": "value1_size1"}, {"storeCategory": "value2_size2"}, {"storeCategory": "value3_size3"}, ] storesDF = spark.createDataFrame(data) storesDF.show() from pyspark.sql.functions import split, col # Option C newDF = (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) newDF.show()

c is correct

Option C returns a DataFrame where column storeCategory from DataFrame storesDF is split at the underscore character into column storeValueCategory and column storeSizeCategory. The correct code is: (storesDF.withColumn("storeValueCategory", split(col("storeCategory"), "_")[0]) .withColumn("storeSizeCategory", split(col("storeCategory"), "_")[1])) Explanation: split(col("storeCategory"), "_") splits the values in column storeCategory by the "_" character and returns an array of strings. [0] gets the first element of the resulting array and assigns it to the new column storeValueCategory. [1] gets the second element of the resulting array and assigns it to the new column storeSizeCategory. withColumn is used to create the new columns and returns a new DataFrame.