Quiz 1

Question 1

what are the steps of data analysis pipeline

Answer 1

1. Figure out the question.
2. Find/​get relevant data.
3. Clean & prepare the data.
4. Analyze the data.
5. Interpret & present results.

Question 2

why is full data analysis broken up into many steps

Answer 2

if is impractical to rerun the first few steps over an over. ex API calls.

Question 3

what does np.vectorize do

Answer 3

turns a function into a function that can operate on an entire array in an element by element fashion

Question 4

what is numPy good for

Answer 4

storing and operating on arrays

Question 5

what is pandas good for

Answer 5

pandas is good for manipulating data

Question 6

what is a series in panda

Answer 6

a 1d array, stored as a numpy array, column

Question 7

what is a dataframe in panda

Answer 7

a collection of data

Question 8

define the steps of extract-transform-load

Answer 8

extract: get the data you need

transform: fix the data, clean data, get it in a form you want to work with

load: load into next step of your pipeline

Question 9

what is signal processing and filtering algorithms

Answer 9

signal process uses filtering algorithms to remove noise from a signal

Question 10

what does LOESS smoothing do

Answer 10

it’s a technique to smooth a curve, to remove the noise

Question 11

LOESS smoothing takes a local area of the data and fit’s a line to it. We have to make a decision on how big this area is.

What happens if we pick a small area or a large area?

Answer 11

if small, we are more sensitive to noise

if large, we are less sensitive to signal changes

Question 12

is LOESS better with lots of samples or sparce samples

Answer 12

better with more samples

Question 13

true or false: LOESS’s parameters are y then x

Answer 13

true

Question 14

what is kalman filtering

Answer 14

it allows you to express what you know to predict the most likely value for the truth

Question 15

what does the kalman operation need

Answer 15

we need to give the variance of
1. our observations
2. our predications

and the covariance between each pair

we also need matrices for both out observations and predictions

the covariance matrix express our uncertainty in the measurement and predictions

Question 16

in your observation-covariance matrix, which expresses errors in the observations, what does lower and higher values mean

Answer 16

lower values: less sensor error, allow the observations/measurements to have more of an effect on the result

higher values: more noise exists

Question 17

the transition_covariance says what you think about the error in your prediction. What does lower and higher values mean

Answer 17

lower: less prediction error, let prediction affect the results more, less noise

higher: less accurate

Question 18

what does it mean to impute data

Answer 18

replacing missing or deleted outliers with plausible, calculated values

Question 19

what is entity resolution or record linkage

Answer 19

the process of finding multiple values that actually refer to the same entity

Question 20

what is the difference between
city_data = city_data[city_data[‘area’] <= 10000]
city_data = city_data[‘area’] <= 10000

Answer 20

city_data = city_data[city_data[‘area’] <= 10000]
will filter based on if area is <= 10k

city_data = city_data[‘area’] <= 10000
will change the dataset to a single column of true/false based off if area <= 10k

Question 21

how do you write sums in numpy and sums in pandas

Answer 21

numpy:
np.sum(totals, axis=x)

pandas:
totals.sum(axis=x)

Question 22

you have a df called counts, and you have a column in the dataframe called ‘date’ make it a datetime column

Answer 22

counts[‘date’] = pd.to_datetime(counts[‘date’])