4.11.1 Big Data

Question 1

Define big data.

Answer 1

A catch all term for data that won’t fit in the usual containers.

Question 2

What do the three v’s do?

Answer 2

Describe big data.

Question 3

What are the 3 v’s?

Answer 3

Volume.
Velocity.
Variety.

Question 4

Define volume.

Answer 4

Too much data for it all to fit on a conventional hard drive or server. Data has to be stored over multiple serves, each of which is composed over many hard drives.

Question 5

In terms of volume, why must data be stored over multiple servers?

Answer 5

As relational databases don’t scale well over multiple machines.

Question 6

Define velocity.

Answer 6

Data in the servers created and modified rapidly. Servers must respond to frequently changing data within a matter of milliseconds.

Question 7

Define variety.

Answer 7

Data held on the servers consists of many different data types - from binary files to multimedia files.

Question 8

In terms of big data, what is the biggest problem?

Answer 8

Unstructured nature gives cause for difficulty when analysing the data. Conventional databases are not suited to store big data as it is required that it confirms to a column and row structure. Do not scale well over multiple servers.

Question 9

What needs to happen when storing big data over multiple servers?

Answer 9

The processing associated with using the data must be amongst multiple machines.

Question 10

Why is storing big data over multiple machines incredibly difficult with conventional programming paradigms?

Answer 10

As all machines would have to be synchronised so no data is overwritten or damaged.

Question 11

Why is functional programming used with big data?

Answer 11

Solves the problem of programming over multiple machines.
Stateless - no side effects.
Uses immutable data structures.
Supports higher order functions.
Attributes make it easier to write and correct efficient, distributed code than with any procedural programming.

Question 12

How can we represent data that doesn’t conform to the typical column and row format?

Answer 12

With the fact based model

Question 13

In terms of the fact based model (FBM) how is data stored?

Answer 13

As a fact.

Question 14

(FBM) What are the benefits of facts?

Answer 14

Immutable and cannot be overwritten, reducing the risk of loosing data due to human error.

Question 15

(FBM) what is stored with each fact?

Answer 15

A time stamp - indicating the data and time each piece of information was recorded.

Question 16

Why are timestamps used?

Answer 16

Multiple different values could be held for the same attribute - computers can discern most reason values.

Question 17

Define a graph scheme (Big Data and Graphs (BDG)).

Answer 17

Uses nodes and edges to graphically represent the structure of a dataset.

Question 18

Define an edge.

Answer 18

Relationships between entities with a brief description of it.

Question 19

Where are the properties? (BDG)

Answer 19

Listed within the entities.

Question 20

How often are timestamps used and why.

Answer 20

Rarely, as it is assumed that most nodes contain the most recent information available.

Question 21

What are the alternative representations of properties? (BDG)

Answer 21

Inside rectangles joined to entities with a dashed line, not representing a relationship, just the properties that belong to said entity.

Question 22

(Functional programming and Big Data (FPBD) when do we use functional programming in big data?

Answer 22

When working with data which needs to be distributed over multiple servers (volume).

Question 23

(FPBD) does functional programming have side effects?

Answer 23

No, it will not change any values or affect the program elsewhere.

Question 24

(FPBD) What is stateless news?

Answer 24

When the current state of the variable, regardless of the order call of functions, does not rely on variables from other function.

Question 25

(FPBD) why do we use stateless news?

Answer 25

It is easier to write correct code so we can predict the behaviour of the program.

Question 26

(FPBD) What is a benefit of functional programming.

Answer 26

Supports higher order functions - takes one or more functions as an input and outputs a function.

Question 27

(FPBD) What does FPBD not allow for?

Answer 27

Variable assignments - a created variable cannot be modified and is an immutable object.

Question 28

(FPBD) What is a benefit of variable assignment?

Answer 28

Makes parallel processing across multiple servers easier.