minTemperatures

Question 1

basic Spark import

Answer 1

from pyspark import SparkConf, SparkContext

Question 2

create Spark conf (for my machine)

Answer 2

conf = SparkConf().setMaster(“local”).setAppName(“MinTemperatures”)

Question 3

create Spark context

Answer 3

sc = SparkContext(conf = conf)

Question 4

split delimited string into fields

Answer 4

fields = line.split(‘,’)

Question 5

function syntax

Answer 5

def myFunctionName(myInput):
    ... 
    return (...myOutputTuple...)

Question 6

convert string to float

Answer 6

temperatureC = float(fields[3])

Question 7

read text file into RDD

Answer 7

lines = sc.textFile(“file:///path/filename.csv”)

Question 8

map an RDD

Answer 8

parsedLines = lines.map(parseLine) where parseLine is my map function

Question 9

filter an RDD

Answer 9

minTemps = parsedLines.filter(lambda x: “TMIN” in x[1])

Question 10

map subset of fields

Answer 10

stationTemps = minTemps.map(lambda x: (x[0], x[2]))

Question 11

reduce by key with min lambda

Answer 11

minTemps = stationTemps.reduceByKey(lambda x, y: min(x,y))

Question 12

move RDD to collection

Answer 12

results = minTemps.collect()

Question 13

loop through a collection “results”

Answer 13

for result in results:

…some action…

Question 14

print value result[0]

Answer 14

print result[0]

Question 15

print tab escape sequence

Answer 15

\t

Question 16

format a float “result” as a string w/ 2 decimals

Answer 16

”“.format(result)

Question 17

add an int and string value to a string in format

Answer 17

”{} {}”.format(myInt,myStr)

Question 18

split a space delimited line and take only third field (as a string)

Answer 18

line.split()[2]

Question 19

sort a dictionary

Answer 19

sortedResult = sorted(result.items())

Question 20

loop through a dictionary

Answer 20

for key, value in sortedResult:

Question 21

count values in RDD

Answer 21

result = ratings.countByValue()

Question 22

clean word to show as ascii

Answer 22

cleanWord = word.encode(“ascii”,”ignore”)

Question 23

how do you import regular expressions

Answer 23

import re

Question 24

what library use for natural language?

Answer 24

nltk - natural language tool kit

Question 25

what are regular expressions?

Answer 25

language for text processing

Question 26

split “text” by word, allow for unicode

Answer 26

re.compile(r’\W+’, re.UNICODE).split(text.lower())

Question 27

broadcast a dictionary in Spark

Answer 27

nameDict = sc.broadcast(loadMovieNames())

Question 28

retrieve value from dictionary

Answer 28

nameDict.value[keyVal]