big data, functions, and more paper 2 stuff Flashcards
(41 cards)
1
Q
a database is stored at an office. staff use a client server database system. describe an example of a problem that could occur if no systems were in place to manage concurrent access to the database
A
- two users edit the same data simultaneously
- one user writes the data then the other user writes the data
- only one users update is lost
2
Q
- the performance of a system is unsatisfactory. the time delay between a client sending a query to the server and the client receiving the results is long
- explain how the performance of the system might be improved, include the following factors:
- the hardware of the server
- the design of the computer network
- the database and software running on the server
A
- Server hardware
- replace the processor with one that had more cores
- replace processor with one that has more cache memory
- use a processor with a bigger word size
- faster clock speed
- replace HDD’s with SSDs
- use the harvard architecture
- Replace the motherboard with one which has buses which run at a faster clock speed
- Replace the motherboard with one which has more lines in the data bus
- Use the Harvard architecture
- Distribute the processing across multiple servers - Network
- Replace the network cable with cable that has a higher bandwidth
- Replace any wireless/WiFi connections with wired ones
- Replace the network cards with ones that can transmit data at a higher bitrate
- Consider the overall network design eg how the network is divided into subnets A. split
the network into subnets
- Use a star topology (instead of a bus)
- Consider using a more efficient protocol for the data across the network
- Add additional wireless access points - Database and Software
- Use a more efficient technique for controlling concurrent access to the database
- Replace the database software with software that uses more efficient algorithms for
tasks
- Rewrite the database software in a language that is suitable for concurrent execution //
- Ensure the software is compiled rather than executed by an interpreter
- normalise the database design
- Use a non-relational database system
- Distribute the data across multiple servers
- Try to reduce the amount of other (unrelated) software that might be running on the
database server at the same time
- run some tasks at quiet times/overnight
- Purge / archive data that is no longer necessary
3
Q
explain some of the challenges that face legislators in the digital age
A
- information can be combined/transferred in ways that were not previously possible
- technology evolves quickly so it’s difficult for the law to keep up with changes
- different laws in different countries
- some crimes may be committed by states rather than individuals
- methods such as encryption makes it harder to monitor criminal activity
- individuals may have access to large amounts of sensitive information that may be of public interest
4
Q
what is a programming paradigm
A
a style of computer programming
5
Q
what is procedural programming
A
- supported by languages like Python/Pascal
- have a series of instructions that tell the computer what to do with the input in order to solve the problem
6
Q
what is structured programming
A
a type of procedural programming approach using clear control structures: sequence, selection (if/else), and iteration (loops).
Improves code readability and maintenance by avoiding “goto” statements.
Breaks programs into smaller, manageable blocks (functions or procedures).
eg python
7
Q
what is oop
A
- supported by languages like Java, Python, Delphi
- type of programming that organises software design around data or objects
8
Q
what is declarative programming
A
- eg sql
- you write statements that describe the problem to be solved
- You describe what the program should do, not how to do it.
9
Q
what is functional programming
A
- supported by Haskell, Python, Java, C++ and more
- statements are written as a series of functions which accept data as arguments and return an output
10
Q
what is a function
A
- a mapping from a set of inputs
11
Q
what is the domain
A
the set of input values
12
Q
what is the codomain
A
set of output values
13
Q
the domain or codomain have to be…
A
part of a data type
14
Q
isEqual :: int -> int -> bool
A
int = input time
isequal = function name
bool = what gets returned
isEqual :: x y = x == y
return True if x = y
15
Q
what is a first class object
A
an object which may:
- appear in expressions
- be assigned to a variable
- be assigned as an argument
- be returned in a function call
16
Q
differences between functional
and procedural
A
Functional: Based on mathematical functions and immutability.
Procedural: Based on step-by-step instructions (procedures).
Functional: Avoids changing state; uses pure functions.
Procedural: Changes program state using variables.
17
Q
what is functional composition
A
when we combine two functions to get a new function
18
Q
partial function application
A
- process + one of the variables
eg (add 6) 4
19
Q
what are high order functions
A
- any function that takes a function as an argument or returns a function as a result or does both
20
Q
what is map
A
- a higher order function that takes a list and the function to be applied to the elements in the list
- makes a list by applying the function to each element in old list
eg map (max 3) [1,2,3,4,5] = [3,3,3,4,5]
21
Q
what is filter
A
- another high order function which takes a predicate(to define a Booleon condition) and a list
- it returns the elements within a list that satisfies the Bool condition
eg
filter (==5) [2,5,7,2,5] = [5,5]
22
Q
what is the fold function
A
- refuse a list to a single value using recursion, by applying an operator
eg to add all the elements in a list, function will be foldl (+) 0[2,3,4,5]
23
Q
foldl foldr
A
foldl = fold left
eg, foldl (/) 100[2,5] = (100/2) / 5 = 5
foldr = fold right
eg foldr (/) 100[2,5] = 2/(5/100) = 40
24
Q
heads and tails of a list
A
- heads is the first item in the list
- tails is the rest of the list
eg [1,2,3,4] - 1 is the head
- [2,3,4] is the tail
25
why is fold high order function
- the operations are functions
26
what is big data
data that is collected on such a large scale that it cannot easily be analysed
27
characteristics of big data
- velocity , the data is generated at high velocity
- high VOLUME of data to process, cannot all be stored on one server
- variety , big data comes in many forms, structured, unstructured, text, video, image
28
features of functional programming that make it easier to write
(variables are immutable which means that) the state variables cannot be changed (after they are created)
(Functions are stateless which means that) functions do not have side-
effects
higher order functions can take a function as an argument and apply it to every element in a list
The order of execution can be determined at run-time
29
map and fold/reduce are efficiently parallelised, what does this mean
many processors can work simultaneously on parts of a dataset without affecting other parts
30
what do the circles represent in big data
entities
31
what do the squares represent in big data
fields associated with entities
32
what are facts based models
- models that captures a single piece of information, eg sensor location
33
how can a function be partially applied to an argument (add for example)
- the function is applied to one of the arguments
- the output of this function is a new function
- eg (add 4) 6
34
Describe what Big Data is, using examples
- data collected on a large scale
characteristics:
- Variety of different forms of information, eg :Email messages, Videos
- high volume of data, eg large medical datasets for diagnosis
- The data is must be processed ay high velocity, eg :Thousands of items to process per
second.
35
challenges that could arise with big data
- Data cannot be stored on one server
* Not possible to process data quickly enough with one computer.
* Data cannot be represented in a table
* Some forms of data / unstructured data are difficult to analyse.
36
how challenges with big data can be overcome
* Distributed database systems distributed across multiple servers.
* Use of functional programming.
* parallelising the execution of programs.
* input split into parts then mapper executed on each part then all results combined by reducers
* Functional programming makes it easier to write distributable code
* Functional programming makes it easier to write correct code
* Use of many thousands of commodity servers.
* Use of servers with multiple CPUs / cores / drives.
* Machine learning can identify patterns / the value in the data // use of predictive
data models.
* Use of languages such as XML or JSON to describe semi-structured data.
37
what do the circles in the graph represent in big data
the nodes holding the PROPERTIES that are atomic
38
what do the edges in big data graphs represent
the interactions/relationships between the nodes
39
advantages of vonn neumann over harvard
Simpler design: Uses a single memory and bus for both data and instructions, making it easier to build.
Lower cost: Fewer hardware components are needed compared to Harvard's separate memory units.
Flexible memory use: Memory can be dynamically allocated for data or instructions as needed.
Easier programming: Uniform memory space simplifies coding and compiling.
Widely adopted: Most general-purpose computers and processors use Von Neumann architecture.
40
imperative language
Program is a sequence of instructions that are followed in order;
Instructions describe how to carry out a task // each instruction is a step;
Instructions can change the state of the program
41
Describe what Uniform Resource Locators (URLs) are used for in a
RESTful application.
Each resource is represented by a URL;
Entering a URL causes the server to (use CRUD to) retrieve (the relevant) data;
URLs are sent between the client and the server using HTTP;
