MSCDSA04 - Data Exploration

Question 1

In Python, how do you define a function

Answer 1

use ‘ def ‘ then function name, then in () the inputs it expects, and finish with a ‘ : ‘ eg

def myFunction (x,y):
    z = x+y
    return z

Question 2

What are descriptive statistics?

Answer 2

Statistics that summarise the data concisely, and use different ways to visualise the data.

Scatter graphs
Mean, media, mode, skewdness etc

Question 3

What is Exploratory data analysis?

Answer 3

looking for patterns, differences, and other features that address the questions we are interested in.

At the same time checking for inconsistencies and identifying limitations.

Question 4

What is a cross-sectional study vs a longitudinal study?

Answer 4

a cross-sectional study captures a snapshot of a group at a point in time.
Everyone in population Should have an equal chance of being selected
a longitudinal study observes a group repeatedly over a period of time.

Question 5

What is the name for people who make who participate in a survey

Answer 5

Respondents

Question 6

In Co-Lab, how do you load a module?

Answer 6

from collections load ..

Question 7

In Python, how do you get an index number for each item in a list?

Answer 7

use and for loop and the ‘ enumerate ‘ function

for myIndexNumber, i in enumerate(myListName):
print (myIndexNumber, “\t”, i)

The above will iterate through a list and print out the index number and a tab and the list value for that index

0 0
1 23
2 34
3 17

Question 8

In Python, how do you sort a list of items?

Answer 8

The sorted() function returns a sorted list of the specified iterable object.

You can specify ascending or descending order. Strings are sorted alphabetically, and numbers are sorted numerically.

a = (“h”, “b”, “a”, “c”, “f”, “d”, “e”, “g”)
x = sorted(a, reverse=True)
print(x)

produces: [‘h’, ‘g’, ‘f’, ‘e’, ‘d’, ‘c’, ‘b’, ‘a’]

Question 9

What domes first? Machine learning or Data Exploration?

Answer 9

Data exploration comes before machine learning.

You cannot do the machine learning until you understand something about what the data has to tell you

Question 10

What are the 5 basic terms in statistics?

Answer 10

1) Population = is everything
2) Sample = is the group sampled from the population
3) Statistic = is the quantity we calculate from the sample data
4) Parameter = is a number that is a property of the population. A statistic is an estimate of a parameter e.g. the average mean measurement of the variables (say ‘height’)
5) The variable = a characteristic of interest for each person or thing in the population. It is the data: what is measured , can be numeric (weight, time etc), or categorical (eye colour, gender, ethnicity)

Data are the actual values of the variable: may be numbers or words.

Question 11

What are inferential statistics?

Answer 11

1) verify a hypothesis
2) trying to find a line of best fit etc (relies on probability) -
3) test if there is a relationship.

Question 12

What happens to the mean of the data is skewed?

Answer 12

it is bad for the mean, it is an average and so the mean can be too big or too small due to more high or low value outliers

Question 13

What is more sensitive to skewness? Mean, Median, or Mode

Answer 13

Mean is more sensitive.

Mode and Median are less sensitive

Question 14

What is a ‘ repository ‘ in GitHub and what is it for?

Answer 14

A repository is usually used to organize a single project.

Repositories can contain folders and files, images, videos, spreadsheets, and data sets – anything your project needs.

GitHub recommend including a README, or a file with information about your project.

Question 15

What are ‘Pull Requests ‘ in GitHub? What are they for and how do they work?

Answer 15

Pull Requests are the heart of collaboration on GitHub.

When you open a pull request, you’re proposing your changes and requesting that someone review and pull in your contribution and merge them into their branch.

Pull requests show ‘ diffs ‘, or differences, of the content from both branches. The changes, additions, and subtractions are shown in green and red.

As soon as you make a commit, you can open a pull request and start a discussion, even before the code is finished.

By using GitHub’s @mention system in your pull request message, you can ask for feedback from specific people or teams, whether they’re down the hall or 10 time zones away.

You can even open pull requests in your own repository and merge them yourself.

Question 16

Who was the Swedish statistician who presented ‘ the joy of statistics ‘

Answer 16

Professor Hans Rosling

Question 17

Who was the Dr, who presented ‘ The Joy of Data ‘

Answer 17

Dr Hannah Fry

Question 18

Describe 5 key steps of the ‘data exploration and analysis process?

Answer 18

1) sample the population.
2) Pick the population parameters desired.
3) Choose variables (numeric or categorical). Understand the type(s) of data (NOIR).
4) Use descriptive statistics such as mean, median and mode, variance, standard deviation.
5) Use Inferential Statistics to draw conclusions such as relationships, or to test hypotheses.

Question 19

What are the differences between the different types of statistical data?

Answer 19

N) Nominal = Qualitative; variables are ‘named categories’ with no mathematical properties

O) Ordinal = Quantititive; The variable is a number but indicates only RANK order and can NOT be used for calculations (e.g. means)

I) Interval = Quantititive; The variable is numbers with equal sized steps, but no ZERO point. Scores can be added, means calculated.

R) Ratio = Quantititive; same as interval but the variable is a numeric scale with a ZERO point. (e.g. age, miles per hour, etc). Ratios can be calculated

Question 20

Demonstrate how the different types of data influence the types of analysis?

Answer 20

Nominal or ordinal could be displayed in a frequency table or pie chart / bar graph

Question 21

What does NOIR stand for and what does it mean

Answer 21

Different types of data in statistics:

N) Nominal = Qualitative; variables are ‘named categories’ with no mathematical properties

O) Ordinal = Quantitative; The variable is a number but indicates only RANK order and can NOT be used for calculations (e.g. means)

I) Interval = Quantitative; The variable is numbers with equal sized steps, but no ZERO point. Scores can be added, means calculated. (e.g a temperature scale) - differences make senses but ratios do not (20˚ / 10˚)

R) Ratio = Quantitative; same as interval but the variable is a numeric scale with a ZERO point. (e.g. age, miles per hour, etc). Ratios can be calculated (e.g. exam scores)

Question 22

What sort of data comes from counting or measuring?

Answer 22

Quantitative, always the result of counting

Question 23

What are the 2 types of Quantitative data, and what are their properties?

Answer 23

Discrete and Continuous

Discrete are the result of counting items (whole)

Continuous are the result of measuring (assuming accurate measurement) i..e height or speed

Question 24

What 2 categories can statistical analysis be separated into?

Answer 24

1) Descriptive Statistics
2) Inferential Statistics

1) Descriptive is about organising and summarising data, through:
a) Plotting / visualising data
b) Using numbers(e.g. finding averages etc)

2) Inferential is methods from probability theory to draw conclusions from data, to:
a) Test if a relationship between variables
b) test a hypothesis about a parameter to do with the population

Question 25

What are 3 Central Tendencies?

Answer 25

1) Mean - sum of x / n of x or
mean of x = 1/n * ∑x

2) Median - the middle value or mean of the middle 2 values where n is even
3) Mode = most common data value

Question 26

What is the ‘range’ of a data set and how is calculated

Answer 26

The ‘Range’ is a measure of dispersion or ‘spread’.

It is the arithmetic distance between the largest and smallest number in the set.

Question 27

What is the ‘Variance’ of a population, how is it calculated?

Answer 27

The variance is a measure of dispersion. It is Sigma squared (σ or ∑)²

1) Find the difference between each data point and the mean of the data set
2) Square the differences
3) Calculate the arithmetic mean of the the squared differences

But it produces quite strange results unless you continue on to calculate the Standard Deviation which is simply the square root of the variance

Question 28

What is the standard Deviation of a population, how is it calculated?

Answer 28

The Standard Deviation is a measure of dispersion. It is the square root of the variance.

Or simply √Sigma squared
or √σ ² or √∑²
or simply ∑ or σ

1) Calculate the variance
2) Take the Square root of the variance

Question 29

What is stratification and how can it improve on random sampling, and under what cases?

Answer 29

is possible, in some instances, to improve on simple random sampling by stratification of the population.

This is particularly true where the population is heterogeneous (i.e. made up of dissimilar groups) and the population can be stratified into homogeneous (i.e. similar) classes. These classes should define mutually exclusive categories.

Question 30

What does heterogeneous mean?

Answer 30

made up of dissimilar groups

Question 31

What does homogeneous mean?

Answer 31

Homogeneous means similar.

E.g the population can be stratified into homogeneous (i.e. similar) classes

Question 32

In Pandas, what is a DataFrame? What is it used for?

Answer 32

a DataFrame is a collection of values of potentially different types

It is used for data manipulation
———-
Imagine it as a relational data table, with rows and named columns.

The data frame is a commonly used abstraction for data manipulation. Similar implementations exist in Spark and R.

Question 33

In Pandas, what is a Series?

Answer 33

In Pandas a series is a single column.

A DataFrame contains one or more Series and a name for each Series.

Question 34

In Pandas, what DataFrame.method can be used to produce a set of descriptive statistics automatically?

Answer 34

dfName.describe()

Question 35

What two Pandas DataFrame.methods show the:

1) The first few records in the DataFrame
2) The last few records in the DataFrame

Answer 35

1) dfName.head()
2) dfName.tail()

in both cases the default is 5 records but you can provide a parameter of the number of records

Question 36

What is NumPy?

Answer 36

NumPy is a popular toolkit for scientific computing.

Pandas Series can be used as arguments to most NumPy functions:

Question 37

In Python, what is lambda for?

Answer 37

The lambda keyword in Python provides a shortcut for declaring small anonymous functions, that can operate ‘in line’.

Lambdas have only an implicit single return statement (the result of the calculation) so some people refer to lambdas as ‘single expression functions’

Lambdas can also remember the values from their scope even when they are no longer in scope

Question 38

What are the 4 elementary terms in probability

Answer 38

four terms −

1) experiment, (tossing a coin)
2) outcome, (observable result of an experiment
3) event (outcome or set of outcomes to an experiment of interest to the experimenter.)
4) sample space (a sample space is the set of all possible outcomes of an experiment.)

For example, if we throw a die then the sample space is {1, 2, 3, 4, 5, 6} and two possible events are

(a) a score of 3 or more, represented by the set: {3, 4, 5, 6}
(b) a score which is even, represented by the set: {2, 4, 6}.

Question 39

Which takes ‘account of order’; a combination or a permutation?

Answer 39

Permutation takes account of order

Combination does not take account of order

Question 40

Which distribution goes with which type:

1) Discrete distribution
2) Continuous distribution

With

A) Measurement
B) Counting

Answer 40

1) and B) discrete is from counting

2) and A) continuous is from measurement

Question 41

In Co-Lab how can you render mathematical symbols?

Answer 41

You can add math to text cells using LaTeX

It will be rendered by MathJax.

Just place the statement within a pair of $ signs.

For example $\sqrt{3x-1}+(1+x)^2$ becomes √3𝑥−1 +(1+𝑥)2.

Question 42

The alpha α symbol in t tests means what?

Answer 42

α is the significance level that we’re testing a hypothesis against

Question 43

in many statistical calculations ‘degrees of freedom’ actually means.. what?

Answer 43

its the count of datapoints in the sample in question, usually minus 1