Pyspark

Question 1

A distributed process has

Answer 1

access to the computational resources across a number of machines connected through a network

Distributed machines also have the advantage of easily scaling, you can just add more machines

They also include fault tolerance…

Question 2

Hadoop is a way …

MapReduce

Answer 2

to distribute very large files across multiple machines.

MapReduce distribute a computational task to a distributed data set

Question 3

Spark is

Answer 3

one of the latest technologies being used to quickly and easily handle Big Data.

You can think of Spark as a flexible alternative to MapReduce.

Question 4

Spark vs MapReduce

Answer 4

MapReduce requires files to be stored in HDFS, Spark doesn’t.

Spark also can perform operations up to 100x faster than MapReduce

Question 5

Scala

Answer 5

Spark itself is not a programming language. It’s just a framework for dealing with large data and distributing it and doing those calculations across a distributor network Spark itself is written in a programming language known as Scala.

So the Scala API for Spark is the one that gets the latest features which makes sense because Spark has literally written in Scala.

Scala is written in Java.

Question 6

DataBreaks : AWS

Answer 6

DataBreaks basically running on top of Amazon Web Services.

Question 7

show DataFrame

Answer 7

df.show()

display(df)

Question 8

schema

Answer 8

df.printSchema()

Question 9

df column names

Answer 9

df.columns

Question 10

df stat

Answer 10

df.describe().show()

Question 11

a column values

Answer 11

df.select(‘net_bkd_pax’).show()

Question 12

two column values

Answer 12

df_gdd.select([‘net_bkd_pax’,’net_ia_pax’]).show()

Question 13

Add a new column

Answer 13

df.withColumn(‘new’, df[‘net_ia_pax’]*2).show()

These changes are not permanent on our original dataframe. You would have to save this to a new variable.

Question 14

Rename a column

Answer 14

df.withColumnRenamed(‘net_ia_pax’, ‘new_name’).show()

Question 15

create temp view

Answer 15

df_gdd.createOrReplaceTempView(‘gdd’)
result = spark.sql(“SELECT * FROM gdd”)
result.show()

Question 16

read csv

Answer 16

df = spark.read.csv(“address”, header=True, inferSchema=True)

Question 17

head

Answer 17

df.head(3)[0]

Question 18

filter

Answer 18

df.filter(“runid<10”).show()

Question 19

filter and show one column

Answer 19

df.filter(“runid<10”).select(“forecastname”).show()

df.filter(df[“runid”]<10).select(“forecastname”).show()

Question 20

filter based on two conditions

Answer 20

df.filter((df[“runid”]<10) & (df[“start_date”]<20)).select(“forecastname”).show()

used () for each condition

Question 21

.collect

Answer 21

if we use .collect() than .show(), we will get a list and the data can be used in future operations

Question 22

list to dic

Answer 22

result.asDict()