Columns

Question 1

What are the different ways you can select columns from customerDf

Answer 1

.select("column name")
.select('column_name)
.select($"column_name")
.select(col("column_name")
.select(column("column_name"))
.select(customerDf.col("column_name"))

Question 2

How do you combine two columns

Answer 2

.select(expr(“concat(firstname, lastname) name”))

Question 3

Is there an overload that takes a string and a column object

Answer 3

No, you cannot do variations of column objects, but you can’t mix strings and column objects

Question 4

How can you select columns using sql

Answer 4

.selectExpr(“column”)

Question 5

With SQL, how do you get and rename a column

Answer 5

.selectExpr(“birthdate birthday”)

Question 6

How do you see all the columns for a DataFrame

Answer 6

customerDf.columns

Question 7

How do you rename a column in the data frame

Answer 7

.withColumnRenamed(“old_name”, “new name”)

Question 8

If you rename a column that does not exist, spark will fail

Answer 8

False, it will succeed but do nothing.

Question 9

can columnRenamed take in column objects

Answer 9

No, strings only

Question 10

How do you print the schema of the data frame

Answer 10

.printSchema

Question 11

How can you change a datatype of a column not using apache spark types

Answer 11

.select($”column_object”.cast(“long”))

Question 12

How can you change the data type of the column using apache spark types

Answer 12

import org.apache.spark.sql.types._ (the _ means all tyoes)

.select($”column_object”.cast(StringType))

Question 13

How can you change the data type of a column using a select expression

Answer 13

.selectExpr(“cast(complex_object.property_in_there[0]) as double) rename_if_want”)

example is a changing the property of a complex type and renaming it. The same works with any column

Question 14

How can you add a column, make it of two existing columns with a space

Answer 14

.withColumn(“new_column_name”, concat_ws(“ “, $”first_column”, $”second_column”))

Question 15

How can you remove a column using a string

Answer 15

.drop(‘‘column_name”)

Question 16

How do you remove multiple columns using string

Answer 16

.drop(“column1”, “column2”)

Question 17

Can the remove column function take column objects

Answer 17

Yes

Question 18

What are the different ways you can use column objects

Answer 18

'column_name
$"column_name"
col("column_name"
column("column_name")
customerDf.col("column_name")

Question 19

Using column objects, how to you remove multiple columns

Answer 19

You can’t. With column objects you can only remove one a. time

Question 20

How do you create a new column by multiplying two other columns together

Answer 20

.withColumn(“column_name”, $”column_a” * $”column_b”)

Question 21

How do you create a new column by dividing two other columns, using expression

Answer 21

.withColumn(“new_column”, expr(“column1 / column2”))

not required to use expression, just how it would work

Question 22

How do you create a new column by rounding another column

Answer 22

withColumn(“new_column”, round($”column_name”, 2))

Question 23

with .toDf how can you specify columns

Answer 23

.toDf(“column_name”)

Question 24

.withColumn will put a column where

Answer 24

at the end