Exam Certified Associate Developer for Apache Spark topic 1 question 44 discussion

The answer should be E because Year is already a python list.

it uses spark.createDataFrame correctly with the Python list years and the appropriate data type IntegerType(). All other options have errors either in syntax or the use of PySpark methods and types.

E is the right answer See code below: # Create a DataFrame from a list of integers from pyspark.sql import SparkSession from pyspark.sql.types import IntegerType spark = SparkSession.builder.appName("MyApp").getOrCreate() years = [2017, 2018, 2019] df = spark.createDataFrame(years, IntegerType()) df.show() df.printSchema()

E is the most suitable, but it also contains an error. In PySpark, the correct class name for the integer data type is IntegerType (not "IntegertType").

e is the right

E is correct: from pyspark.sql.types import IntegerType years = [2023, 2024] print(type(years)) storesDF = spark.createDataFrame(years, IntegerType()) storesDF.show() <class 'list'> +-----+ |value| +-----+ | 2023| | 2024| +-----+

D from pyspark.sql.types import IntegerType spark.createDataFrame([1991,2023],IntegerType()).show() +-----+ |value| +-----+ | 1991| | 2023| +-----+

1. spark 2. createDataFrame 3. years 4. IntegertType()

if years is variable, it works, just tested it: years = [1, 3, 4, 5 , 9] df7 = spark.createDataFrame(years, IntegerType()) df7.show() this works as well: df7 = spark.createDataFrame([1, 3, 4, 5 , 9], IntegerType()) df7.show() this won't work: df7 = spark.createDataFrame([years], IntegerType()) df7.show() so, the answer is E

E. D is giving an error .

D throws a big error. /usr/local/spark/python/pyspark/sql/types.py in verify_acceptable_types(obj) 1291 # subclass of them can not be fromInternal in JVM 1292 if type(obj) not in _acceptable_types[_type]: -> 1293 raise TypeError(new_msg("%s can not accept object %r in type %s" 1294 % (dataType, obj, type(obj)))) 1295 TypeError: field value: IntegerType can not accept object [1, 2, 3, 4, 5] in type <class 'list'> E is correct answer from pyspark.sql.types import IntegerType a = [1,2,3,4,5] spark.createDataFrame(a, IntegerType()).show()

Two responses 1. D is an error. E will split the array into rows 2. spark.createDataFrame([arraryVar_name],ArrayType(IntegerType())) will store the whole array as a row

Agreed