Exploratory Data Analysis

Question 1

Select columns (.loc) from DataFrame with ALL non-zeros

Answer 1

df.loc[ : , df.all( ) ]

Question 2

Select columns (.loc) from DataFrame with ANY non-zeros

Answer 2

df.loc[ : , df.any( ) ]

Question 3

Select columns (.loc) from DataFrame with ANY NaNs

Answer 3

df.loc [ : , df.isnull( ).any( ) ]

Question 4

Select columns (.loc) from DataFrame with NO NaNs

Answer 4

df.loc [ : , df.notnull( ).all( ) ]

Question 5

Drop rows with ANY NaNs

Answer 5

df.dropna(how = ‘any’)

Question 6

Import local file.xlsx using pandas (as data)

Answer 6

data = pd.ExcelFile(file.xlsx)
# print(data.sheet_names) 
# df = data.parse('sheetname') or (0)

Question 7

Initial GET requests using urllib.

6 lines : import statements, url, request, response, read and close.

Answer 7

1. from urllib.request import urlopen, Request
2. url = “https://www.wikipedia.org”
3. request = Request(url)
4. response = urlopen(request)
5. html = response.read( )
6. response.close( )

Question 8

Initial GET requests using requests.

4 lines : import, url, request, read.

Answer 8

1. import requests
2. url = “https://www.wikipedia.org”
3. r = requests.get(url)
4. text = r.text

Question 9

Tidy Data: Principles. ( 3 )

Answer 9

1. Columns represent separate variables containing values
2. Rows represent individual observations
3. Observational units form tables

Question 10

Tidy Data: Melting and Pivoting

Answer 10

Turn analysis-friendly into report-friendly
Melting: turn columns into rows.
Pivoting: turn unique values into separate columns

Question 11

Tidy Data: Melting syntax

Answer 11

pd.melt(frame=df, id_vars=’col-2b-fixed’, value_vars=[’ ‘,’ ‘ ], var_name=’name’, value_name=’name’)

Question 12

Tidy Data: Pivoting syntax

Answer 12

df.pivot_table(values=’ ‘, index=’ ‘, columns=’ ‘, aggfunc=np.mean)

Question 13

Change column-type from ‘object’ to ‘numeric’

Answer 13

df [ ‘object_col’ ] = pd.to_numeric ( df [ ‘object_col’ ], errors=’coerce’)

Question 14

Change column-type to ‘category’

Answer 14

df [ ‘column’ ].astype( ‘category’ )

Question 15

Plot idioms for DataFrames (3)

Answer 15

df. plot( kind=’hist’)
df. plt.hist( )
df. hist( )

Question 16

Syntax for .loc accessor

Answer 16

df.loc [ ‘Row_Label’ ] [ ‘Col_Label’ ]

Question 17

Syntax: sns barplot - use.

Answer 17

sns.barplot(x= ‘categorical’, y= ‘numerical’, data= df, estimator=np.mean)

Question 18

Syntax: sns countplot - use.

Answer 18

sns.countplot(x=’column’, data=df, hue=’category’)

Question 19

Syntax: sns histogram plot - use.

Answer 19

sns.distplot ( df [ ‘continuous’ ], kde=False, bins=30 )

Question 20

Syntax: sns scatterplot - use.

Answer 20

sns.lmplot ( x= ‘numerical’, y= ‘numerical’, data=df, hue=’category’, fit_reg=False )

Question 21

Syntax: sns pairplot - use.

Answer 21

sns.pairplot ( df, hue=’categorical’, palette=’coolwarm’ )

Question 22

sns categorical plots

Answer 22

sns. barplot(x= ‘categorical’, y= ‘numerical’, data= df, estimator=np.mean)
sns. boxplot( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’)
sns. countplot( x= ‘categorical’, data= df )
sns. factorplot( x= ‘categorical’, y= ‘numerical’, data=df, kind= ‘bar’ (or = ‘point’, or = ‘violin’)
sns. stripplot( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’, jitter=True, dodge=True
sns. violinplot( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’, split= True)

Question 23

Syntax: sns heatmap - use.

Answer 23

sns.heatmap ( PivotTable, cmap=’ ‘, lw=’ ‘. lc=’ ‘ )

Question 24

sns categorical plots

Answer 24

sns. barplot ( x= ‘categorical’, y= ‘numerical’, data= df, estimator=np.mean)
sns. boxplot ( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’)
sns. countplot ( x= ‘categorical’, data= df )
sns. factorplot ( x= ‘categorical’, y= ‘numerical’, data=df, kind= ‘bar’ (or = ‘point’, or = ‘violin’)
sns. stripplot ( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’, jitter=True, dodge=True
sns. violinplot ( x= ‘categorical’, y= ‘numerical’, data=df, hue=’categorical’, split= True)

Question 25

sns categorical plots (6)

Answer 25

sns. barplot ( x= 'categorical', y= 'numerical', data= df, estimator=np.mean) sns. boxplot ( x= 'categorical', y= 'numerical', data=df, hue='categorical') sns. countplot ( x= 'categorical', data= df ) sns. factorplot ( x= 'categorical', y= 'numerical', data=df, kind= 'bar' (or = 'point', or = 'violin') sns. stripplot ( x= 'categorical', y= 'numerical', data=df, hue='categorical', jitter=True, dodge=True sns. violinplot ( x= 'categorical', y= 'numerical', data=df, hue='categorical', split= True)

Question 26

sns distribution plots (4)

Answer 26

sns. distplot ( df [ 'continuous' ], kde=False, bins=30 ) sns. jointplot ( x= 'continuous', y= 'numerical', data=df ) sns. pairplot ( df, hue='categorical', palette='coolwarm' ) sns. rugplot ( df [ 'continuous' ] )

Question 27

sns matrix plots (2)

Answer 27

sns. heatmap ( PivotTable, cmap=' ', lw=' '. lc=' ' ) sns. heatmap ( df.corr( ), annot = True ) sns.clustermap (PivotTable, cmap= ' ' )

Question 28

sns PairGrid

Answer 28

g = sns.PairGrid ( df ) g. map ( plt.scatter ) g. map_diag ( sns.distplot ) g. map_upper ( plt.scatter ) g. map_lower ( sns.kdeplot )

Question 29

sns FacetGrid

Answer 29

g = sns.FacetGrid ( data=df, col='category', row='category' ) g.map ( sns.distplot, 'numerical' )

Question 30

sns lmplot (regression)

Answer 30

sns.lmplot ( x= 'numerical', y= 'numerical', data=df, hue='category', markers['o' , 'v'], scatter_kws={'s':100} )

Question 31

Scatter plots

Answer 31

df. plot.scatter(x= 'col_A', y='col_B', color='col_C', size=df ['col_C']*100 ) plt. scatter( x, y) sns. lmplot ( x= 'numerical', y= 'numerical', data=df, hue='category', fit_reg=False ) df. iplot ( kind=scatter, x= 'col_A', y='col_B', mode='markers' )

Question 32

Data Quality Control - find missing data - drop columns - drop rows

Answer 32

df. isnull( ) # returns bool - NaN = True | sns. heatmap ( df.isnull( ), yticklabels=False, cbar=False, cmap='viridis' )

Question 33

Histogram plots

Answer 33

df [ 'continuous' ].hist(bins=20) df [ df [ 'column1' ] == 1 [ 'column2' ].hist ( bins=30, color='blue', label='label', alpha=0.5 )

Question 34

Reproducible Data Analysis 1/10 Get data from web Jake Vanderplas

Answer 34

``` from urllib.request import urlretrieve URL = 'https://data.seattle.gov/api....' urlretrieve(URL, 'file.csv') data = pd.read_csv("file.csv", index_col='Date', parse_dates=True) data.resample('W').sum( ).plot( ) ```

Question 35

Reproducible Data Analysis 2/10 Exploratory Data Analysis Jake Vanderplas

Answer 35

data.columns = [ 'West', 'East' ] data['Total'] = data['West'] + data['East'] ax = data.resample('D').sum( ).rolling(365).sum( ).plot( ) ax.set_ylim(0, None) data.groupby(data.index.time).mean( ).plot( ) pivoted = data.pivot_table('Total', index=data.index.time, columns=data.index.date) pivoted.plot(legend=False, alpha=0.01) # line for every day

Question 36

Reproducible Data Analysis 3/10 Version control with Git & GitHub Jake Vanderplas

Answer 36

``` https://github.com Create new repository (Name, description, README, .gitignore(Python), MIT License) Copy Clone or download link Terminal window: git clone mv JupyterNotebook.ipynb into git folder git status git add JupyterNotebook.ipynb git commit -m "Add initial analysis" git push origin master ``` open JupyterNotebook.ipynb from correct location git status > file.csv vim .gitignore > # data > file.csv

Question 37

Reproducible Data Analysis 4/10 Working with Data and GitHub Jake Vanderplas

Answer 37

import os from urllib.request import urlretrieve URL = 'https://data.seattle.gov/api....' def get_file(filename='file.csv', url=URL, force_download=False): if force_download or not os.path.exists(filename): urlretrieve(url, filename) data = pd.read_csv("file.csv", index_col='Date', parse_dates=True) data.columns = [ 'West', 'East' ] data['Total'] = data['West'] + data['East'] return data data = get_file( )

Question 38

Reproducible Data Analysis 5/10 Creating a Python package Jake Vanderplas

Answer 38

Terminal window mkdir jupyterworkflow touch jupyterworkflow/__init__.py vim jupyterworkflow/data.py ``` """ Download and cache the data Parameters: filename : string (optional) location to save the data url : string (optional) web location force_download : bool (optional) Returns """ < replace 4/10 with following > from jupyterworkflow.data import get_file ```

Question 39

Confusion matrix

Answer 39

True pos (tp). False pos (fp). False neg (fn ). True neg (tn).

Question 40

Confusion matrix Accuracy =

Answer 40

Fraction of correct predictions Accuracy = correct / total = tp + tn/ tp fp tn fn

Question 41

Confusion matrix Precision =

Answer 41

How accurate positive predictions were. tp / tp + fp

Question 42

Confusion matrix Recall

Answer 42

What fraction of positives the model identified. tp / tp + fn

Question 43

Confusion matrix F1 score

Answer 43

The harmonic mean of precision and recall- lies between them 2 * prec * recall / prec + recall

Question 44

Model trade-off between precision and recall.

Answer 44

Too many “yes” gives high fp- high recall, low precision Too few “yes” gives high fn- low recall, high precision.

Question 45

Input feature categories Naive Bayes classifier

Answer 45

Suited to yes or no features

Question 46

Input feature categories Regression models

Answer 46

Numerical features

Question 47

Input feature categories Decision tree

Answer 47

Numeric or categorical features

Question 48

Input feature categories SVM

Answer 48

Numerical features

Question 49

Common way to analyze the relationship between a categorical feature and a continuous feature

Answer 49

Boxplot

Question 50

Check for null values in the dataset.

Answer 50

print( df.isnull ( ).values.sum( ) )

Question 51

Check column-wise distribution of null values

Answer 51

print( df.isnull ( ).sum( ) )

Question 52

Frequency distribution of categories within a feature

Answer 52

print(df['category_col'].value_counts( ) )

Question 53

Dictionary comprehension to map category strings to numeric values. eg. {'carrier': {'AA': 1, 'OO': 7, 'DL': 4, 'F9': 5, 'B6': 3, 'US': 9, 'AS': 2, 'WN': 11, 'VX': 10, 'HA': 6, 'UA': 8}}

Answer 53

``` labels = df['category_col'].astype('category').cat.categories.tolist( ) replace = {'category_col' : {k: v for k,v in zip(labels,list(range(1,len(labels)+1)))}} ``` print(replace)

Question 54

Categorical Data - 3 types

Answer 54

Nominal: No intrinsic order Ordinal: Ordered or ranked. Dichotomous: Nominal with only 2 categories