Rows

Question 1

How do you filter a dataframe based on values being greater than using where

Answer 1

.where(year($”birthdate”) > 1980)

Question 2

How do you filter, using filter, to a specific month

Answer 2

.filter(month(`birthdate) === 1)

Question 3

Using a SQL expression, how do you filter a dataframe

Answer 3

.where(“date(birthdate)>15”)

Question 4

How do you check for inequality when filtering

Answer 4

=!=

Question 5

How do you make sure your data frame only has unique values taking all columns into account

Answer 5

.distinct

Question 6

How can your remove duplicates and only take one column into account

Answer 6

.dropDuplicates(“column”)

Question 7

How can your remove duplicates and only take multiple columns into account

Answer 7

.dropDuplicates(List(“column1”, “column2”))

Question 8

if dropDistinct does not have any columns passed in, what columns are taken into account to determine distinct values or does it fail?

Answer 8

All columns, equivalent to .distinct

Question 9

How can you filter out null values from a dataframe

Answer 9

.where($”column_object”.isNotNull)

Question 10

How can you drop rows with all null values

Answer 10

.na.drop(how=”all”)

Question 11

How to remove a row where the any value in the row is null

Answer 11

.na.drop(“any”)

Question 12

How do you remove the row if two specific columns have nulls

Answer 12

.na.drop(“all”, Seq(“column_a”, “column_b”))

Question 13

How do you remove a row with a null value in either of two columns

Answer 13

.na.drop(“any”, Seq(“column_a”, “column_b”))

Question 14

How can you replace all null values with “nope”

Answer 14

.na.fill(“nope”)

Question 15

True/False

.na.fill(“Nope”) will only replace null where the column type is string

Answer 15

True

Question 16

What is the default sorting order

Answer 16

ascending

Question 17

how can you sort

Answer 17

.sort(“column_name”)
.orderBy()
.sort(expr(“column_name”)

Question 18

How do you sort using an expression

Answer 18

.sort(expr(“column_name”))

Question 19

Sort by month in a column where the type is date, and sort in descending order

Answer 19

.orderBy(expr(“month(birthdate)”).desc)

Question 20

If you have two columns, one for customer id and one for items ids, how can you show how many items are associated with each customer

Answer 20

.groupBy(“customer_id”).agg(count(“item_id”).alias(“total”))

Question 21

What is it called when an operation that causes spark to move data across the cluster

Answer 21

A Shuffle

Question 22

When one input partition contributes to multiple output partitions, what is it called

Answer 22

Wide Transformation or Wide Dependencies

Question 23

What are some examples of wide transformations

Answer 23

GroupBy
Join
Distinct
Repartion
Coalesce
OrderBy

Question 24

for the SQL year function, what can be passed in

Answer 24

Column object only

Question 25

For the max sql function, what can be passed in

Answer 25

String or Column object

Question 26

How can you get the max of a column and the min of another column

Answer 26

.agg(
max(“column_name”),
min(“other_column”))

Question 27

How do you count rows where there is a value in the column “price”

Answer 27

.select(count(“price”))

Question 28

What is a way to pull the common aggregate values for each column in a dataframe

Answer 28

.describe()

Question 29

.describe takes what as input

Answer 29

column names

Question 30

.desc function takes in what input

Answer 30

coumn names