Lecture Notes Flashcards

Question

How can we create a new string with slicing?

Answer 1

We can store our sub-string as a new variable (then this can be manipulated) string2 = string1[:8]

Answer 2

String splitting is a very useful method for manipulating strings - it involves breaking a string into multiple parts. string.split(' ')

Answer 3

A tuple is a type which holds an arbitrary sequence of items, which can be of different types. They are used to store multiple items in a single variable. Think multiple

Answer 4

my_tuple = ('A', 'tuple', 'of', 5, 'entries')

Answer 5

Similar notation as characters in a string my_tuple[0]

Answer 6

Tuples only use a small amount of memory but once created, the items cannot be changed. Tuples are immutable, like strings A list is a similar but more flexible data type compared to a tuple. Lists are also comma-separated, but use square brackets

Answer 7

Tuples Strings

Answer 8

Both are comma-separated lists. Tuples - () Lists - [] Tuples are immutable Lists support assignment - you can access an item and change its value

Answer 9

You can access an item and change its value.

Answer 10

list[index] for a list of lists list[i][j]

Answer 11

Store it as another variable copied = list.copy()

Answer 12

list.append(value)

Answer 13

Four spaces - can usually tab in most editors

Answer 14

A handy way to store and access data. A dictionary is a set of keyword and value pairs. You use the keyword to access the value. The value can be of any type, including another dictionary. dict = { x:y, a:b } The name of a key is always a string and needs quotation marks.

Answer 15

dict = { x:y, a:b } The name of a key is always a string and needs quotation marks.

Answer 16

Controlling which parts of your code get executed when, in what order, how many times, under what conditions, where to start and stop etc. It is essential to making sure your program actually does what you want it to do. Flow is controlled mainly by using conditional logic and loops.

Answer 17

We don't need to care about where the value we want is, we just have to remember what we called it. The name of a key is always a string and needs quotation marks.

Answer 18

A block of code which first checks if a specified condition is true, and only in that case will it carry out the task if condition : # body It will only be applied to the indented code which follows the :

Answer 19

If statements only execute if the condition is true. The else statement executes if the condition is false. if condition : # code else : # code

Answer 20

If-elif-else if condition 1 : # code elif condition 2 : # code else : # code

Answer 21

A block of code that will iterate (execute consecutively) multiple time.

Answer 22

A for loop requires something to iterate over, ie an "iterable" like a list (do something for every time in the list) or a string (do something for every character in the string) for var in iterable : # code for i in range(10) # code

Answer 23

range(x) list(range(x))

Answer 24

Pass - do nothing Continue - stop this iteration of the loop early, and go on to the next one Break - end the loop entirely

Answer 25

open() function r - reading only w - for writing, if the file exists it overwrites it, otherwise it creates a new file a - opens for file appending only, if it doesn't exist, it creates the file x - creates a new file, if the file exists it fails + - opens a file for updating syntax: f = open('zen_of_python.txt', 'r')

Answer 26

'r' - opens a file for reading only.

Answer 27

'w' - opens a file for writing. If the file exists, it overwrites it. Otherwise, it creates a new file.

Answer 28

'a' - opens a file for appending only. If the file doesn't exist, it creates the file.

Answer 29

'+' - opens a file for updating.

Answer 30

When the file is closed Use the .close() method if not using with/as

Answer 31

'x' - creates a new file. If the file exists, it fails.

Answer 32

Close it, to release memory used in opening the file. When writing to a file, the changes are not saved until the file is closed. Use the .close() method

Answer 33

f = open("file_name.txt.", "r") then use print(f.read()) pr print(f.readline())

Answer 34

The name of the file you want to look at and the mode with which you want to interact with the file

Answer 35

.read() reads the entire contents of the file .readline() reads only the next line, it can be called repeatedly until the entire file has been read .readlines() is the most useful, it reads each line, one line at a time and then stores it all into a single list

Answer 36

The first output will print the entire contents of the file. The second output will be blank. Once the file object has been read to the end, any subsequent calls return an empty string.

Answer 37

Results in an error

Answer 38

.readlines() f = open("file_name.txt", "r") lines = f.readlines() f.close() print(lines) The file is closed but we have the contents written to a variable, we can then get the lines we want by indexing

Answer 39

We can make sure that files are only open for as long as we need them by using a with statement with open("file_nmae.txt", "r") as d: # put file operations in here print(f.read())

Answer 40

An error will be produced - the with/as syntax closes the file automatically at the end. This is important for file writing, less important for file reading.

Answer 41

with open("file_name.txt", "w") as f: f.write("String") Basic input and output only reads and writes strings. The code below will cause an error and result in an empty file.

Answer 42

It erases any previous contents

Answer 43

%s - string %d - integer %f - float %e - float, but using scientific notation eg('%f', %length) or ("This is a %d word %s" %(length, datatype)) - can include as many variables as you want by putting several % signs in the string, and providing a tuple after the string. The first % (inside the string) indicates that we are writing a variable. The letter that follows indicates what type of variable. The second % sign (after the string) tells your code which variable to write at the first % sign.

Answer 44

A JSON file is structured like a Python dictionary. JavaScript Object Notation (JSON) is a standard text-based format for representing structured data based on JavaScript object syntax. It is commonly used for transmitting data in web applications .

Answer 45

Using specialised modules

Answer 46

Using the JSON module Use json.dump to write to file import json define dictionary eg masses with open("planets.json", "w") as f: json.dump(masses,f) - this is the thing you want to dump and the file you want to dump it into

Answer 47

Using the JSON module Use json.load to write to file import json with open("planets.json", "r") as f: new_dictionary = json.load(f) print(new_dictionary) -- to investigate that we have successfully read the JSON dictionary

Answer 48

import json write - json.dump() read - json.load()

Answer 49

Double quotes You can define it with single quotes - python doesn't care but JSON does, so it will convert it eg so that all keys are ""

Answer 50

Dictionaries may be stored as a string if the dictionary is one entry within a larger database

Answer 51

Simply add quotation marks Can check the type with print(type(item)) If there are "" used in the string, then we create the overall string with ' ' - if we try to use the same type of quote both around and within the string, it would end the string early

Answer 52

json.loads() pronounce load-S the extra s is for string eg dict = json.loads(string) print(type(dict)) to check it was successful

Answer 53

Errors - a fundamental issue where python cannot understand your code (syntax error) Exceptions - code is written in valid Python syntax, but an operation cannot be completed successfully

Answer 54

The try/except code try: # code except: The except prevents the code from crashing and implementing an emergency fallback option.

Answer 55

It will catch all exceptions - even if the error is not what you think it is. You should try to catch specific errors.

Answer 56

Raised when an operation or function receives an argument that has the right type but an inappropriate value, and the situation is not described by a more precise exception such as IndexError

Answer 57

try: # code except: # code else: # code - do if no exception finally: # code - always do this at the end

Answer 58

Passing arguments of the wrong type (e.g. passing a list when an int is expected) should result in a TypeError , but passing arguments with the wrong value (e.g. a number outside expected boundaries) should result in a ValueError.

Answer 59

Functions are re-usable. We often want to do the same operation at different times or with different data,

Answer 60

A separate, named block of code for a specific purpose. The code inside a function is "walled off" from the main code.

Answer 61

every function has a name, a list of (required and optional) inputs in parentheses, and returns something at the end. def my_function(): return

Answer 62

def my_function(): return You should give your function a meaningful name

Answer 63

Keyword arguments

Answer 64

Call the function using its name, including the brackets (and any arguments required to be passed in) eg hello_world()

Answer 65

We get an error message

Answer 66

The variable will be assigned the value returned by the function

Answer 67

A global variable is a variable defined in the main body of the code. Any code executed after the variable has been defined is able to "see" the variable. A local variable is a variable defined inside the function or other object. Its value is only accessible within the function or object (ie cannot be accessed outside of the function)

Answer 68

Give it a default value def my_sum(a, b =1): return a +b - If you provide a value for b it will overwrite - If you don't provide a value for b, it will use b = 1 as a default

Answer 69

list.revers()

Answer 70

def arb_function (*nums): # code Within the code, you then loop over nums

Answer 71

You may not know in advance how much data you will need to work with

Answer 72

All functions in Python return something. If you do not specify a value (or leave out the return statement entirely), the function will return a None value by default. Otherwise it returns the value we specify

Answer 73

You can return more than one value from a function, and return different types. For the output: - Provide the same number of variables as the number of values returned/ Each returned value then goes to a separate variable. - Provide a single variable, this will then contain a list of the values returned by the function

Answer 74

Returns variables, ends the function call and returns to the main code. Therefore any code in the function after the return will not be executed. This can be convenient if you want to put conditions for what to return.

Answer 75

A quick way to make short functions that can be defined in one line. They can take any number of arguments, but can only have one expression. name = lambda vars : code eg doubler = lambda x: x*2

Answer 76

name = lambda vars : code eg doubler = lambda x: x*2

Answer 77

If we need to create a function for temporary use eg within another function.

Answer 78

list.append(i)

Answer 79

sorted(list)

Answer 80

A paradigm is like a philosophy informing how we write code. Usually there are many different ways to solve a problem with code. Different paradigms help to shape which approach we choose to use. Procedural programming. Object-oriented programming.

Answer 81

Procedural programming - the code is organised as a sequence of instructions (do this, then this). Each block performs a prescribed task to solve the problem. OOP - data are stored as "objects" belonging to pre-defined "classes". These objects have a set of "attributes" stored internally, which can be updated using built in "methods.

Answer 82

A class is like a template designed in advance to handle a particular data structure, with a set of properties called attributes. It also provides implementations of behaviour (member functions or methods). The syntax looks like class_name.function_name()

Answer 83

list.reverse()

Answer 84

dir(x) or print(dir(x))

Answer 85

Attributes - properties Methods - functions

Answer 86

Attributes internal to python that cannot be updated.

Answer 87

my_list = [x,y,z]

Answer 88

Any object is an instance of a class, created to represent a particular item of data. An instance ie one specific example

Answer 89

Update the internal state of the object eg reversing the list

Answer 90

object.__class__

Answer 91

eg class Animal(): # Can list attributes # Can define functions using def function():

Answer 92

object = Class() Passing in attributes as appropriate. In this case, the attributes would be set to their defaults.

Answer 93

Instantiation

Answer 94

object.attribute

Answer 95

object.attribute = value

Answer 96

object.attribute = value We can add attributes to class instances, we can't edit the parent class

Answer 97

Objects store data in a way where it is easy to update and display the internal state of that data, using built-in methods. OOP allows you to put your methods next to the data. Once we have defined useful classes and instantiated objects, an OO code will mainly interact with the data object through its built-in methods.

Answer 98

The __init__ function

Answer 99

The python __init__ method is declared within a class and is used to initialise the attributes of an object as soon as the object is formed.

Answer 100

class Animal(): def __init__(self, attribute1, attribute2): self.att1 = val self.att2 = val Give the __init__ function a list of arguments, the first argument is always self. This is a special variable which represents the object itself once we have created it ( a self-referential thing) __init__ initialises the attributes of the class

Answer 101

The self parameter is a reference to the current instance of the class, and is used to access variables that belongs to the class.

Answer 102

The attribute "x" belonging to the object "self"

Answer 103

Taking the "self" function as an input, you can then access any attribute with self.x

Answer 104

The object can be created and attributes defined in one line.

Answer 105

Hierarchical inheritance is a type of inheritance in which multiple classes inherit from a single superclass. "Parent" class - Animal "Child" class - Cat, Dog etc. Object - your pet A child class inherits the attributes of its parents class, but we can also add new method and attributes.

Answer 106

Put the name of the parent class in the brackets when creating the child. eg class Cat(Animal): # put attributes from the parent class that should be fixed for the child class first # then use super().__init__(attribute1, attribute2) within __init__ for the attributes we want to specify new values for

Answer 107

def __init__(self, attribute1, attribute2): super().__init__(attribute1, attribute2)

Answer 108

Make the code into a Python module. This is a python file (extension .py) containing one or more classes or functions. You can then import the class or function from the module easily.

Answer 109

A python file (extension .py) containing one or more classes or functions. You can then import the class or function from the module easily.

Answer 110

Ensure the .py file is in the same directory as the current notebook. eg for the class Dog from the animals.py module from animals import Dog From this import, you can create Dog objects (can also define and import functions, they don't need to be part of a class, making it very easy to reuse them)

Answer 111

A collection of modules. You can install (download) these packages and then have access to incredibly useful functions.

Answer 112

Numerical python - a built-in module A module is a pre-defined collection of functions we can load into our programs. NumPy arrays are multidimensional array objects. We can use NumPy arrays for efficient computations on large data sets.

Answer 113

import numpy as np Then call functions as eg np.sin(0) Alternatives (don't use): import numpy - importing the entire numpy module from numpy import sin - importing only a specific function

Answer 114

If we don't want to use up memory on the whole library. We would need to know the name of the function/class hat we want to import in advance. The specific function is now a global name, we don't need to specify the module. This could cause issues when there are functions with the same name.

Answer 115

sin(), cos() and tan() The value passed in this function should be in radians.

Answer 116

import module first dir() eg dir(np)

Answer 117

A "grid" of values, all of the same dtype ie all floats, or all strings etc.

Answer 118

They look similar at a first. - Array use less memory. Therefore array is much more efficient than a list, particularly for a large collection of elements. - Lists are more flexible (can have mixed types)

Answer 119

my_list = [] for i in range(x): my_list.append(i)

Answer 120

print(my_list[:x])

Answer 121

my_array = np.array(my_list)

Answer 122

print(my_array[:10])

Answer 123

print(my_array[0])

Answer 124

print(my_array[-1])

Answer 125

Lists - operations can only be performed on items, so calculates have to be one at a time. Arrays - operation is performed on all elements with single function call - this is much quicker for large data sets.

Answer 126

You can write one line of code rather than a loop. Eg 3D array requires 2 nested for loops. With NumPy,, we don't have to worry about the array shape (it is automatically preserved)

Answer 127

From a list: my_array = np.array(my_list) Create an array of zeros (n = how many elements you want) np.zeros(n) Create an array of ones np.ones(n) Create an array of numbers from a to b, with spacing c np.arange(1, 10, 1) Create an evenly spaced array from a to b, with c points np.linspace(1, 10, 19) Create an array of random numbers from 0 to 1 of length n np.random.random(n)

Answer 128

Creates an array of numbers from a to b with spacing c. Pass in where to start, where to stop and the spacing that you want Stops before the stop number ie if you want 1 - 10 np.arange(1, 11, 1)

Answer 129

Creates an evenly spaced array from a to be with C points. State how many points you want. This enables better precision and can be used to control the number of samples that you want.

Answer 130

np.arange(a, b, c)

Answer 131

Create an array of random numbers from 0 to 1 of length n

Answer 132

np.random.random(n) This creates an array of random numbers ranging from 0 to 1. Can apply transformations to get it into a range you want.

Answer 133

You need to specify the data along each axis. eg my_2d_array = np.array( [ [1, 2, 3], [4, 5, 6] ] ) This is a nested list. The items are the rows. Items at the same position (the same index) within each sub-list form the columns eg my_3d_array = np.array( [ [ [1,2], [3,4], [5,6]], [ [7,8], [9,10], [11,12] ] ] )

Answer 134

We need to supply N indexes, equal to the number of axes (dimensions) print(my_2d_array[0,0]) print(my_2d_array[0][0]) The first index is the row, the second is the column.

Answer 135

Get the first row, all columns my_2d_array[0,:] Get all rows, the first column my_2d_array[:,0] Get all rows, last two columns my_2d_array[:,1:3] The slice of the array is itself returned as an array.

Answer 136

Determine the dimensions an array has AND the size of each dimension my_array.shape

Answer 137

We can do this, the operation is applied to every element in the array. We are adding a scalar (constant value) to the array. Can do add, sub, mult, div

Answer 138

We can add, sub, mult, div one array with another - but the behaviour is different to scalar mathematic expressions. Each element of the array will operate on the element in the same position in the other array. eg my_array * my_array - the array is squared If arrays are different shapes/sizes you can get errors or unexpected behaviours.

Answer 139

Combine array with shape (m,n) with: - Array with shape (1, n) - Array with shape (m, 1) ie a 1D array with the same number of rows or columns as the data. When you add/multiply, it will repeat the 1D array as many times as needed, in order to match the rows.columns in your data. The new array is "broadcast" to the shape of your data.

Answer 140

Masking is the term used for selecting entries in arrays, e.g. depending on its content. We can apply that mask to our data to retrieve a new array that only contains masked values. We can specify conditions and return sub-sets of the array.

Answer 141

even_numbers = (my_array %2 == 0) my_array[even_numbers] Testing each element for the condition

Answer 142

The conditional statement returns an array of Boolean True/False, with the same shape as the array. This can be used as a mask to pick out only the array elements where the condition is true. We mask arrays using square bracket notation, similar to slicing.

Answer 143

They are good for holding tabular data.

Answer 144

Import numpy as np np.pi

Answer 145

x = np.linspace(0, 2*np.pi, 100) y = np.sin(x) Combine the two arrays into a new array data = np.column_stack([x,y])

Answer 146

Using column_stack() or row_stack() functions Takes one argument - a list of arrays to stack The arrays to stack must be the same length as each other

Answer 147

We transpose the array using .T eg transposed_data = data.T For more complicated manipulation we can use the shape and reshape methods - data.shape to check the current shape - rd = data.reshape(2,100) - instead of 100 rows and 2 columns, reshape to 2 rows and 100 columns

Answer 148

data.T The rows are now the columns

Answer 149

reshaped_data = data.reshape(2,100) The size of the new array (n rows * n columns) must match the original ie the product of the axis lengths is constant. eg instead of 2 columns and 100 rows we can have 2 rows and 100 columns OR three_d_data = data.reshape(2,50,2) rows, columns, number of elements in each

Answer 150

- OOP data.sum() - Procedural np.sum(data)

Answer 151

Method approach data.min() data.max()

Answer 152

Possible because the slice is just another array. eg mean of the first column print(np.mean(data[:,1]))

Answer 153

Using the savetxt() function Required arguments are the name of the file to save to (created if it does not exist, otherwise it will be overwritten by default) and the array to save. We can also specify the format of the data and the character to separate the data. np.savetxt('name.csv', data, fmt='%.4f', delimiter=',')

Answer 154

loadtxt() or genfromtxt() functions Required argument - file name. You can also specify the delimiter and dtype to ensure desired behaviour eg arr = np.genfromtxt("file.csv", delimiter=',', dtype= 'float')

Answer 155

Matplotlib

Answer 156

A comprehensive library for creating static, animated and interactive visualisations in Python. It makes easy things easy and hard things possible.

Answer 157

import matplotlib.pyplot as plt

Answer 158

A set of functions that can be used to create a figure with procedural programming. For better control over plotting, it is recommended to use an OO approach with Matlib objects.

Answer 159

Figures - the entire area of the figure Axes - the area for plotting data

Answer 160

fig, ax = plt.subplots() Do this at the start of the plot

Answer 161

print(fig)

Answer 162

The default resolution is 100 pixels per inch.

Answer 163

Using the figsize argument fig, ax = plt.subplots(figsize=(7,5)) Size in inches

Answer 164

The points along the lines can be given as a list of x and y coordinates which must be the same length.

Answer 165

fig, ax = plt.subplots() x = [...] y = [...] ax.plot(x,y) ax.set_xlabel("X") ax.set_ylabel("Y") plt.show() This is the OOP approach

Answer 166

ax.set_xlabel("X") ax.set_ylabel("Y") Procedural: plt.xlabel("X") plt.ylabel("Y")

Answer 167

plt.show() is used in Python to display the plot, not always needed in Jupyter.

Answer 168

Procedural approach - we call functions from pyplot Using methods (OOP) often start with set_, functions often do not eg plt.xlabel. In procedural, we don't tell Pyplot which axis to plot the data on, it infers which axis to use (the most recent one).

Answer 169

Numpy arrays

Answer 170

ax.plot(data[:,0],data[:,1])

Answer 171

- Changing the units - Changing the upper and lower limits on the axes - Changing the axes tick marks - Adding another curve to a figure using the legends - Changing line styles and colours - Add arbitrary text labels - Add a title

Answer 172

Apply conversion in ax.plot. Can do numpy operations directly in the .plot as long as it produces another array. eg ax.plot(data[:,0]/np.pi*180, data[:,1])

Answer 173

degrees = radians * pi/180

Answer 174

ax.set_xlim() eg ax.set_xlim(0,360)

Answer 175

ax.set_xticks([...]) ax.set_yticks([...]) Pass in a list of the tick marks you want.

Answer 176

Use two ax.plot functions in the one plot.

Answer 177

Add a legend ax.legend()

Answer 178

Using the loc keyword - 'lower', 'center', 'upper' for vertical placement - 'left', 'center', 'right' for horizontal placement eg ax.legend(loc = 'upper center')

Answer 179

frameon=True eg ax.legend(loc = 'upper center', frameon=True)

Answer 180

Add lifestyle='-' and line width = 2 to ax.plot Available line styles include '-' (solid), '--' (dashed), ':' (dotted), '-.' (dash-dot), '--.' eg ax.plot(data[:,0]/np.pi*180, cosine, label='cos(x)', color='deeppink', linestyle='--', linewidth=2)

Answer 181

Add color="" to ax.plot

Answer 182

ax.text(120, 1, "Maximum", fontsize=20) Providing the coordinates where you want to write the text and the string you want to put in.

Answer 183

fontsize =

Answer 184

ax.set_title("Title")

Answer 185

This means showing different information on different panels of a single figure. Using the plt.subplots() function we can specify how many axes in the vertical direction with nrows and the horizontal direction with ncols. fig, axes = plt.subplots(figsize=(8,8), nrows=2, ncols=1) ax1 = axes[0] ax2 = axes[1] now we access using ax1 and ax2 etc.

Answer 186

ax.scatter(x, y, marker="o")

Answer 187

plt.errorbar(x, y, xerr, yerr, fmt="o", color="r")

Answer 188

Shape and colour of the plots for errorbar Outline: - '.' : point - '+', 'x' : crosses Filled: - 'o' : circle - 's' : square - '^', '<', '>', 'v' : triangles in different directions - 'd', 'D'; 'p', 'P'; 'h', 'H' : different types of diamond, pentagon or hexagon - '*' : star Line plots: - '-', '--', ':' etc fmt='s' color='gold' markersize=6 markeredgewidth=2 markeredgecolor='k' ecolor='k

Answer 189

fmt (format)

Answer 190

ax.hist(x) Specifying the bins: ax.hist(x, bins=20)

Answer 191

10 bins (of equal width)

Answer 192

With finer bins, we can see more detail in the distribution. But if we use too many bins we can overdo it and end up with lots of misleading gaps.

Answer 193

- Changing colour - Changing from a filled histogram to an outline - Normalise the histogram to plot the probability density rather than total frequency ax.hist(heights, bins=20, color='teal', histtype='step', density='True')

Answer 194

The hist function returns these already. counts = ax.hist(x) numbers in each bin - counts[0] boundary edges - counts[1]

Answer 195

Bar charts and pie charts

Answer 196

ax.bar(categories, counts, color=bar_colors) These are all lists to be passed in

Answer 197

ax.pie(counts, labels=categories, colors=bar_colors, autopct='%d') Include optional argument, auto percent to print the percentages - d means it prints as an integer

Answer 198

Matplot has an easy way to make plots using images (eg a picture or photograph) Data must be provided as a 2D NumPy array. Matplotlib will display the array as a grid of pixels, with the intensity of each pixel determined by the value of the array at that position. image = np.gemfromtxt("pixels.txt") fig, ax = plt.subplots(figsize=(8,8)) ax.imshow(image, origin='lower', cmap='Greys_r', vmin=0, vmax=300) - Origin determines which way up it will be printed - CMAP - what colour do you want it print - Vmin and max are saturation points (less than 0 = fully black, above 300 = fully white, important for contrast)

Answer 199

ax.set_xticks([]) ax.set_yticks([])

Answer 200

Using array slicing to zoom in ax.imshow(image[80:220,80:220], origin='lower', cmap='Greys_r', vmin=0, vmax=1000) Highlighting key features ax.scatter(70,70,marker='o',s=10000,c='None',edgecolors='r',label='Supernova')

Answer 201

Reduce whitespace around your figure: plt.tight_layout(pad=0.5) Save your plot: plt.savefig('image.png')

Answer 202

Pandas builds on NumPy and introduces a new object called a data frame (or a series if one-dimensional)

Answer 203

Pandas series is one-dimensional (more similar to a list rather than a tabular structure)

Answer 204

- A data frame looks like a table or spreadsheet, with convenient column and row labels - A data frame includes methods for sorting, filtering and performing complex operations on data - Columns can be of different data types (unlike an array) - Provides some of the functionality of an array

Answer 205

import pandas as pd df = pd.read_csv("data.csv") When we load the data into Pandas, the first row is assumed to be the column headings. If we wanted to we could override this behaviour by providing a list of column names to an optional keyword, names=.

Answer 206

import pandas as pd

Answer 207

Length len(df)

Answer 208

df.head(x) - where x is the number of rows to display

Answer 209

print(df) but this isn't very nice can call df directly but this must be the last command in the cell

Answer 210

The rows are given numerical indices by default. Sometimes one of the columns in the data is already a convenient index. We can assign this as the index df = pd.read_csv('titanic.csv', index_col='PassengerId')

Answer 211

index_col="Column Name"

Answer 212

Clean up the data set - removing unwanted data, missing values or duplicates

Answer 213

df = df/drop(columns=["X"]) you can drop multiple columns at once by providing a list of column labels

Answer 214

NaN Not a number value - usually this represents a missing value

Answer 215

df.isna() eg df.isna().head(6)

Answer 216

df.dropna() eg df = df.dropna(subset=['Age']) If we don't specify a subset of columns to use, it will remove all rows that have a NaN in any column.

Answer 217

df.drop_duplicates() df.drop_duplicates(subset='Ticket')

Answer 218

Using loc() and iloc() NB: they use square brackets like in array indexing loc gets rows (and/or columns) with particular labels. iloc gets rows (and/or columns) at integer locations.

Answer 219

df['Age'] Slicing syntax

Answer 220

df.iloc[0] As this is only 1D it displays a series.

Answer 221

df.iloc[100:110]

Answer 222

df.iloc[100:110, :4] This will return 5 columns in total - the index column and then the first 4 columns

Answer 223

We may not know the specific index to search for but we do know the column title.

Answer 224

df.loc[:,'Name']

Answer 225

df.loc[:10, 'Name':'Age'] df.loc[:10, ['Name','Age', 'Fare']] NB: you can retrieve non-consecutive rows/columns by providing a list. Therefore df.loc is very flexible.

Answer 226

Masking - provide an array of T/F to loc. eg df.loc[(df['Pclass']==1)]

Answer 227

Using the .isin() method eg df.loc[df['Pclass'].isin([1,2])]

Answer 228

df['Age'].mean() mean / min / max - these calculations automatically ignore NaN values

Answer 229

df['Age'].sort_values()

Answer 230

df.sort_values(by='Age') df.sort_values(by='Age', ascending=False)

Answer 231

len() python function - number of rows df.size pandas property - number of cells ie rows by columns

Answer 232

A 1D data frame ( a series) - the original indices are still present. To get only the values, we can access df.values - this is a property not a method, so no()

Answer 233

df.values No brackets, it is a property not a method.

Answer 234

df["Name"].values

Answer 235

df.add() method eg relatives = df['SibSp'].add(df['Parch'], fill_value=0) store it in a new column: df['Relatives'] = df['SibSp'].add(df['Parch'], fill_value=0)

Answer 236

It will not try to add a NaN if the column has missing values, you can specify what value to use in place of NaNs by including a fill_value This is safer for handling NaN values rather than simple addition

Answer 237

df.add() df.subtract() df.multiply() df.divide()

Answer 238

Matplotlib can naturally understand data frames just like Numpy arrays. You can pass the columns directly to plotting commands. eg ax.hist(df['Age'], bins=30)

Answer 239

The df.apply() method Define the function needed if required. df["Name"].apply(function_name) Returns a series of Quicker = lambda functions, define as a temporary function inside apply() df['Name'].apply(lambda x: x.split(',')[0])

Answer 240

my_string.split(',')

Answer 241

my_string.split(',')[0] my_string.split(',')[-1]

Answer 242

It is defined as a temporary function and avoids using memory for a function that is used only once

Answer 243

df.groupby() classes = df.groupby('Pclass') This provides a dictionary, where the keys are the groups and each contains a list of row indexes. The object produced has the function/attribute .groups print(classes.groups) to see the index of the rows belonging to each key.

Answer 244

new_groups = df.groupby('Embarked') new_groups.groups.keys()

Answer 245

We can quickly calculate statistics separately on each of the different groups. Allows us to investigate aggregated data rather than on the whole dictionary directly. We can do this with any column of our grouped data using square bracket notation.

Answer 246

classes = df.groupby('Pclass') classes['Fare'].mean() This returns a value for each of the group keys

Answer 247

Look at the size of each group - classes.size() NB: for a dataframe object, size is a property (no parentheses) but for a grouped object is is a method, requires parentheses

Answer 248

classes.size().sort_values(ascending=False)

Answer 249

Use get_group() first = classes.get_group("Group Name") The argument "Group Name" should match one of the keys in classes.groups.keys().

Answer 250

data = np.random.randint(low=40, high=100, size=12) This will make a 12 x 1 array

Answer 251

Use the pd.DataFrame function df = pd.DataFrame(data) where data is a NumPy array The shape and content is preserved, but the rows and columns now have explicit names. By default the NumPy row and column indices.

Answer 252

Using familiar square bracket notation with the name of the column df["Name"] or df[1] or more explicitly (better for more complex selections) df.loc[:, "Name"]

Answer 253

use index= and columns= attributes eg df = pd.DataFrame(data, index=['Matt','Jonathan','Fiona','Deepak'], columns=['DSA8001','DSA8002','DSA8003'])

Answer 254

data_dict = { "Module 1": {"Matt":80, "John":60}, "Module 2": {"Matt":70, "John":63}, "Module 3": {"Matt":82, "John":76}, } df = pd.DataFrame(data_dict) Outer keys define the column headings ie modules will be the column Each nested dictionary defines one row

Answer 255

Pandas will use the dictionary keys

Answer 256

Insert method df.insert(loc=1, column="Name", value=data) The length of the array data needs to match the number of rows in the df.

Answer 257

Concat function Concatenates a new data frame to the end of the current one df = pd.DataFrame(new_student, index=['New Student'], columns=['DSA8001','DSA8002','DSA8003','DSA8021']) May need to reshape the data to be added after creation, using reshape. Concatenation only works well if the column labels match. It will fill in things with NaN an may convert existing data (NaN is not an integer, things may be converted to float).

Answer 258

to_csv() - typically we save as a CSV file using this built-in dataframe method df.to_csv("file_name.csv")

Answer 259

When you open a data frame with read_csv(), it adds a numerical index column by default. We may end up doing this repeatedly, adding another index column each time. Best to specify a particular column to use for the row indices when reading in the CSV df.read_csv("file.csv", index_col=0)

Answer 260

df.read_csv("file.csv", index_col=0)

Answer 261

JSON is a string, formatted like a dictionary. It is very flexible for dataframe columns that need to contain complex information.

Answer 262

JSON data - can be stored as a dictionary

Answer 263

import json

Answer 264

eg df = pd.DataFrame(index=['Matt','Jonathan','Fiona','Deepak'], columns=['module_scores'])

Answer 265

json.loads(x) eg getting the "DSA8002" column info. df['module_scores'].apply(lambda x: json.loads(x)['DSA8002'])

Answer 266

The json.loads() method can be used to parse a valid JSON string and convert it into a Python Dictionary

Answer 267

Index the series returned from json.loads(x) like any other data frame eg df['module_scores'].apply(lambda x: json.loads(x)['DSA8002']).loc['Matt']

Answer 268

pd.to_datetime df['datetime'] = pd.to_datetime(df['datetime'])

Answer 269

df['datetime'].dtypes

Answer 270

df['hour'] = df['datetime'].dt.hour or dt.year etc.

Answer 271

Group by and then sum spend_by_hour = df.groupby('hour') spend_by_hour['total'].sum()

Answer 272

If two data frames contain only some columns in common, it is often more useful to merge rather than concatenate.

Answer 273

We find the columns in common between the two databases and return a set of rows with those columns.

Answer 274

JOIN statements in SQL.

Answer 275

pd.merge(df1, df2, on="Column", how="left")

Answer 276

- Left join - keep everything in the left table and what's in the right table if available - Right join - keep everything in the right table and what's in the left table if available - Inner join - return only entries that are present in both tables - Outer merge - returns all rows across both tables

Answer 277

The outer join - returns everything across both tables

Answer 278

The outer merge / full join in SQL

Answer 279

No, new data can be inserted by adding rows or columns

Answer 280

Structured Query Language Used as a tool to search relational databases. Can search, filter, group or combine databases to return entries matching certain criteria. Most popular language to manage relational databases

Answer 281

- Data are stored as a table or tables with rows (records or tuples) and columns (attributes) - Each record has a unique key - Each table represents a particular type of data eg one table to store information on customers, another to store products

Answer 282

It is written closer to natural language, so queries can be constructed more intuitievely..

Answer 283

Numeric - eg INTEGER, FLOAT(p) with p digits of precision String types - CHARACTER (L) with fixed length L, or VARCHAR(L) with a maximum length L DATE, TIME BOOLEAN (True, False)

Answer 284

Pandas data frames are relational databases.

Answer 285

import pandas as pd import pandasql as ps

Answer 286

A handy python module to query pandas data frames

Answer 287

!pip install pandasql

Answer 288

query = ''' ''' ps.sqldf(query)

Answer 289

query = ''' SELECT * FROM dataframe ''' ps.sqldf(query)

Answer 290

Combination of "clauses" with the names of tables and/or columns

Answer 291

Returns entries

Answer 292

Tells SQL which database to select the columns from

Answer 293

They are not case sensitive. Writing in capitals helps differentiate them from the names of tables etc.

Answer 294

A Pandas DataFrame - which is very convenient for further database operations.

Answer 295

query = ''' SELECT "Column Name1", "Column Name2" FROM dataframe ''' ps.sqldf(query)

Answer 296

Use conditional searches using the WHERE clause. query = ''' SELECT * FROM dataframe WHERE city = "Belfast" ''' ps.sqldf(query) NB: single = sign, not == NB: Need to put string of interest in different quotation marks to overall string query

Answer 297

- =, <, <=, >, >= - BETWEEN X AND Y - number in a specified range - IN ('X', 'Y') - values in a given list - LIKE '%YZ%' - value matches a given pattern YZ where % is used to represent free text before and/or after the pattern

Answer 298

Use the AND clause

Answer 299

query = ''' SELECT * FROM dataframe ORDER BY column DESC ''' ps.sqldf(query) Can specify ASC or DSC

Answer 300

query = ''' SELECT * FROM dataframe ORDER BY column DESC, column2 ASC ''' ps.sqldf(query) The ordering is applied one after the other

Answer 301

The LIMIT clause. This is similar to the head() function in Pandas. query = ''' SELECT * FROM dataframe LIMIT 5 ''' ps.sqldf(query)

Answer 302

- COUNT - returns the number of records - MIN, MAX - returns the smallest/largest entries in a column - SUM - sum of entries in a column - AVG - average of entries in a. column

Answer 303

query = ''' SELECT COUNT(*) FROM dataframe WHERE Gender = "Female" ''' ps.sqldf(query) NB: you would get the same answer whether you count the whole database or a single column - the number of rows will be the same either way.

Answer 304

query = ''' SELECT Number, Income/1000 AS [Income ($k)] FROM dataframe ''' ps.sqldf(query) NB: be aware of automatic rounding here, because the column was an integer, it automatically converts a float to an integer.

Answer 305

Using the AS clause It should come right after the column or expression of interest. Need to include square brackets if you want the column name to have a space.

Answer 306

The CASE clause query = ''' SELECT Number, Gender, Age, City, CASE WHEN City='New York City' THEN 'North' WHEN City='Dallas' THEN 'South' END AS Region FROM citizens ''' ps.sqldf(query)

Answer 307

Allows us to return different values depending on the content of a column. The CASE and END start and end the logical criteria AS specifies to column name to store the result

Answer 308

Using the GROUP BY clause query = ''' SELECT Age, COUNT(*) FROM citizens GROUP BY Age ''' ps.sqldf(query)

Answer 309

To calculate aggregated statistics (counts, averages etc.). You can'y display columns from grouped tables, you just see the first record in each group. Only the column used for grouping and aggregated statistics should be included with the SELECT clause.

Answer 310

Select the two columns you want to group by, and select the aggregate statistic for the third column. The order typed is the order they are grouped by query = ''' SELECT Gender, Age, AVG(Income) FROM citizens GROUP BY Age, Gender ''' ps.sqldf(query)

Answer 311

Order matters Need to apply the WHERE clause before grouping the data, so that undesired rows are not included in the grouping stage.

Answer 312

HAVING clause

Answer 313

query = ''' SELECT Gender, Age, AVG(Income) FROM citizens GROUP BY Gender, Age HAVING Age > 30 ''' ps.sqldf(query) "Group by gender and age, but show me only the ones having ages over 30"

Answer 314

It allows us to perform more complex conditions, by applying criteria to the statistics of each group. Eg you only want groups with an average income over X. We need to do the grouping first before we can apply the condition.

Answer 315

JOIN The first table is always the left table The second table is always the right table Joined based on a common column

Answer 316

Inner join ie only returns records present in both tables.

Answer 317

You must provide the column to use for the join, otherwise the entire right table will be repeated for every record in the left table. Specify columns to join on using ON ON citizens.Number = welfare.IDNum It is best to specify which table the column is coming from, to avoid any ambiguity if there is a column with the same name in each table.

Answer 318

query = ''' SELECT * FROM citizens JOIN welfare ON citizens.Number = welfare.IDNum ''' ps.sqldf(query)

Answer 319

Give each table a shorthand name, query = ''' SELECT * FROM citizens c JOIN welfare w ON c.Number = w.IDNum ''' ps.sqldf(query)

Answer 320

When it takes more than one query to get what we want. Avoids hardcoding, which is not very efficient.

Answer 321

Specified using round brackets. The table or value the sub-query returns directly feed into the overall query. query = ''' SELECT * FROM citizens WHERE Income > (SELECT MAX(Income)/2 FROM citizens) ''' ps.sqldf(query).

Answer 322

When you need to compare a column against a list of values. Eg finding all citizens belonging to age groups with average incomes over 55000

Answer 323

JSON_EXTRACT JSON_EXTRACT(table.column, "$[x].key") AS new_colimn Specifying which element of the JSON list we want to extract (0 is the first, 1 is the second etc.) The key we want is optional

Answer 324

query = ''' SELECT title, JSON_EXTRACT(credits.cast, "$[0].name") AS starring FROM credits ''' ps.sqldf(query)

Answer 325

Look at the logarithm of the data (convert to powers of 10)

Answer 326

using np.log10() function create new columns with the logarithmic conversions and then plot these. OR plot directly with plt.xscale('log')

Answer 327

Looking for the slope of the relationship. Using np.polyfit() to fit a polynomial to data.

Answer 328

It takes an x array, y array and a degree (first = linear, second = quadratic etc.) f = np.polyfit(x=df['log_galaxy_mass'], y=df['log_bh_mass'], deg=1) f[0] is the slope f[1] is the intercept We can then plot this using dummy data x_arr = np.arange(x1,x2,0.1) y_mod = f[0] * x_arr + f[1] plt.plot(x_arr, y_mod, color='r')

Answer 329

Calculating the Mean Squared Error - the average difference between model and data MSE = mean((data - model)**2) Calculate the predicted y values linear_prediction = f[0] * df['log_galaxy_mass'].values + f[1] Calculate MSE mse = np.mean( (linear_prediction - df['log_bh_mass'])**2 )

Answer 330

When the range of the y axis is several times bigger than 1 - so the fit is doing better than random

Answer 331

A Python scientific module which provides algorithms for many mathematical problems. We use it for correlation in this module (does bigger x really mean bigger y)

Answer 332

Using a statistic called Spearman's Rank Correlation coefficient from scipy.stats import spearmanr print(spearmanr(df['log_galaxy_mass'], df['log_bh_mass']))

Answer 333

" ".join(x)

Answer 334

list.remove(item)

Answer 335

list.insert(position, item)

Answer 336

list.append(item)

Answer 337

dict.keys()

Answer 338

XeY is short-hand in Python for “X times 10 to the power of Y”

Answer 339

1 x 1 000 000

Answer 340

Within the with open as - json.dumps(data, f)

Answer 341

Within the with open as - json.load(f)

Answer 342

isinstance(string1, str)

Answer 343

def paint(self, colour): try: if isinstance(colour, str): self.colour = colour else: raise TypeError("Paint should be provided as a string") except TypeError: print(TypeError, "- the colour remains", self.colour)

Answer 344

del list[0]

Answer 345

In a simplified sense, you are storing a list in your computer memory, and store the address of that list, so where the list is in your computer memory in x. This means that x does not actually contain all the list elements, rather it contains a reference to the list.

Answer 346

y = list(x) - this is a more explicit copy of the list rather than y = x

Answer 347

Round function round(value, precision)

Answer 348

help(function_name)

Answer 349

sort(list, reverse=False)

Answer 350

list.index(item)

Answer 351

list.count(element)

Answer 352

string.captialise()

Answer 353

string.replace("x","y")

Answer 354

string.upper()

Answer 355

list.reverse() - this changes the list it is called on

Answer 356

The NumPy list

Answer 357

np_array = np.array(list) Assumes the list contains elements of the same type

Answer 358

As 1 and 0

Answer 359

array.shape

Answer 360

array[0][2] or array[0,2]

Answer 361

np.mean(dataset[:,0])

Answer 362

np.corrcoef(dataset[:,0], dataset[:,1]) correlation coefficient

Answer 363

np.std(dataset[:,0])

Answer 364

data = np.round(np.random.normal(1.75, 0.2, 5000), 2) mean = 1.75, std = 0.2, 5000 samples

Answer 365

Matplotlib import matplotlib.pyplot as plt

Answer 366

When time is on the x-axis

Answer 367

s=numpy array

Answer 368

plt.grid(True)

Answer 369

dict.keys()

Answer 370

immutable objects

Answer 371

Strings, Booleans, integers and floatsH

Answer 372

"key" in dictionary - see if it returns True or False

Answer 373

del(dictionary["key"])

Answer 374

np.broadcast_to()

Answer 375

np.max(array)

Answer 376

np.argmax(array)

Answer 377

np.absolute(array)

Answer 378

np.log10(1000)

Answer 379

np.sin(x) np.cos(x) np.pi

Answer 380

Group by city then find the size eg home_team = matches.groupby("Home Team Name").size()

Answer 381

pd.DataFrame(dictionary) Indexes automatically given df.index = [list_of_strings]

Answer 382

Use double square brackets df[["column"]]

Answer 383

df[["column1", "column2"]] OR df.loc[:, ["column1", "column2"]]

Answer 384

loc and iloc

Answer 385

df[ ["row1","row2"], ["col1","col2"]]

Answer 386

np.logical_and(array > 1, array < 5) array[np.logical_and(array > 1, array < 5)] np.logical_and() np.logical_or() np.logical_not()

Answer 387

for index, var in enumerate(seq): expression

Answer 388

for key, value in dictionary.items(): expression

Answer 389

To get every element of an array, you can use a NumPy function called nditer (ND iter) for val in np.nditer(array): print(val)

Answer 390

Prints out the column names

Answer 391

In pandas, you need to explicitly say that you want to iterate over the rows. Generates label on row and actual information. for label, row in np.iterrows(dataframe)|: print(label) print(row) dataframe.loc[label, "country_name_length"] = len(row["country"]) Can also select a specific column eg print(row["column_name"] or (as shown, can create new column) But this is inefficient - use .apply eg dataframe["country_name_length"] = dataframe["country"}.apply(len)

Answer 392

Use .apply(function) eg dataframe["country_name_length"] = dataframe["country"}.apply(len)

Answer 393

Allows you to apply a function on a particular column in an element-wise fashion.

Answer 394

Using a seed - generate pseudo random numbers np.random.seed(123) np.random.rand()

Answer 395

np.random.randint(0,2) This simulates a coin toss

Answer 396

np.random.randint(1,7)

Answer 397

x = max(0, calculated_value) this ensures x never goes below zero

Answer 398

np.transpose(array)

Answer 399

Use of docstrings - placed inside triple double quotation marks def function(paramters): """ """

Answer 400

use keyword global global name

Answer 401

nonlocal keyword

Answer 402

**kwargs This turns the identifier keyword-pairs into a dictionary within the function body Then, within the function body, we print all the key value pairs stored in the dictionary kwargs for key, value in kwargs.items():

Answer 403

We need to use map() to apply the lambda function to all elements of the sequence result = map(lambda x,y: x+y) It returns a map object, convert to list using list(result)

Answer 404

result = filter(lambda x: len(x) > 6, list)

Answer 405

eg if we don't want our function to work for a particular set of values - such as don't want to square root negative numbers using an if statement, we can raise a value error for cases in which the user passes the function a negative number if x < 0: raise ValueError("X must be non-negative")

Answer 406

COUNT (DISTINCT "column_name")

Answer 407

The describe() method computes some summary statistics for numerical columns like mean and median

Answer 408

df.values - a 2D NumPy array df.columns - column labels df.index - row labels

Answer 409

df.sort_values([col1, col2], ascending=[True,False])

Answer 410

Need double square brackets df[["col1", "col2"]]

Answer 411

The dates are in quotes, written as year, month then day This is the international standard date format

Answer 412

Using .isin() dogs["colour"] .isin(["Black", "Brown"])

Answer 413

Aggregate .agg() def function(column): return column.quantile(0.3) df["column"].agg(function) Can be used on multiple columns - pass in list ["col1","col2"] Agg itself can also take a list of functions to apply at the same time Can use .agg for the IQR

Answer 414

Calling .cumsum() on a column returns not just one number, but a number for each row of the data frame df["column"].cumsum() Can also have .cummax(), .cummin(), .cumprod() These all return an entire column of a dataframe, rather than a single number

Answer 415

use .drop_duplicates() eg df.drop_duplicates(subset=["col1", "col2"]

Answer 416

To count the dogs of each breed, we subset the breed column and use the value_counts() method Can do .value_counts(sort=True)

Answer 417

df["column"].value_counts(normalize=True)

Answer 418

dogs.groupby("colour")["weight"].mean()

Answer 419

Pass in multiple summary statistics at once to calculate df["column"].agg([np.min, np.max, np.sum])

Answer 420

A way of calculating grouped summary statistics .pivot_table() df.pivot_table(values="col", index="colour") o The values argument is the column that you want to summarise o The index column is a column that you want to group by Automatically calculates the mean, if you want another statistic, use aggfunc df.pivot_table(values="col", index="colour". aggfunc=np.median) To group by more than one variable, pass in columns df.pivot_table(values="col", index="colour", columns="breed', fill_value=0, margins=True)

Answer 421

df.set_index("column") can include multiple columns df.set_index(["col1", "col2"])

Answer 422

df.reset_index() to get rid of it completely df.reset_index(drop=True)

Answer 423

.loc df.loc[[item1, item2]]

Answer 424

Outer -df.loc[[item1, item2]] - with a list Inner - df.loc[[(oteritem1, inneritem1), (outerritem2, inneritem2]] - with a tuple

Answer 425

.sort_index() for multiple indexes - By default, it sorts all index levels from outer to inner, in ascending order, can control this;: df.sort_index(level = [inner, outer], ascending=[True, False])

Answer 426

Selects consecutive elements from objects

Answer 427

df["columns"].dt.year /.dt.month etc

Answer 428

eg df["column"].hist() avg_weight_by_breed.plot(kind=bar)

Answer 429

pass in rot=45

Answer 430

Represented by NaN df.isna().any() - tells you if there are any missing values in each column df.isna().sum() - tells you how many missing values are in each column

Answer 431

Drop - df.dropna() Fill with 0 - df.fillna(0)

Answer 432

df.to_csv("new filename.csv")

Answer 433

journalsBSA.iloc[journalsBSA["Rank"].idxmin()].loc["Rank"] correct - journalsBSA.loc[journalsBSA["Rank"].idxmin(), "Title"]

Answer 434

Change the axis limits - ax.set_ylim()

Answer 435

determine the y values based on the predicted model and compare to actual values in table MSE = np.mean( (predicted_y - df["column"])**2)

Answer 436

phys_groups.size().sort_values(ascending=False)

Answer 437

In the world of databases, rows are referred to as records Columns are referred to as fields

Answer 438

SELECT DISTINCT column1, column2 FROM dataframe

Answer 439

return the unique combinations of multiple field values

Answer 440

A view is a virtual table that is the result of a saved SQL SELECT statement Views are considered virtual tables There is no result set when creating a view Then this table can be queried

Lecture Notes Flashcards

(484 cards)