UNDERSTANDING BIG DATA AND NOSQL

Question 1

Generally refers to either data that exceed the ability of database management tools used to capture, store, and analyze data (McKinsey, 2011), or to next-generation technologies and architectures designed to extract value from large-scale data at low cost and support the rapid collection, discovery, and analysis of data (IDC, 2011).

Answer 1

Big data

Question 2

The characteristics of big data can be explained by the three elements (3V)

Answer 2

volume, velocity, and variety

Question 3

Characteristics of big data
Refers to a volume of data of tens of terabytes, petabytes, or more, thus exceeding the processing limit of commonly used software when collecting, storing, and processing data.

Answer 3

Volume

Question 4

Characteristics of big data
* ‘Big data’ is created very quickly
* Data collection, processing, storage and analysis need to be pricessed in real time

Answer 4

Velocity

Question 5

Characteristics of big data
* Diverse kinds of data
* Big data can be classified into structured, semi structured, and unstructured data

Answer 5

Variety

Question 6

6V of big data

Answer 6

1. Volume
2. Variety
3. Velocity
4. Veracity
5. Visualization
6. Value

Question 7

Data to be stored in a fixed field

Answer 7

Structured data

Question 8

Data not stored in a fixed field, but which contain metadata or schema, such as XML or HTML.

Answer 8

Semi-structured data

Question 9

• Data not to be stored in a fixed field
• Document, picture, video, and audio data, etc.

Answer 9

Unstructured

Question 10

A technology that can collect data from all devices and systems

Answer 10

Collection
Crawling, ETL, CEP, etc

Question 11

A technology that can store and process collected large-scale data using a distributed processing system.

Answer 11

Storage/processing
Distributed file system, NoSQL, MapReduce processing

Question 12

A method of analysis that can assist companies and the public with using big data in business and daily life.

Answer 12

Analysis
Natural language processing, machine learning, data mining algorithms

Question 13

A technology that can visualize analyzed results effectively.

Answer 13

Visualization
R, graphs, drawing, etc

Question 14

Web ___ copies the entire web page after collecting the URLs to be collected, or collects data with a specific tag only after analyzing the HTML code.

Answer 14

crawling

Question 15

Collects data using the SQL function of the DBMS.

Answer 15

Collection using the DBMS
Oracle, MariaDB, MS SQL, Tibero, etc.

Question 16

Collects data when a certain condition is met

Answer 16

Collection using sensors
CQL, Kafka

Question 17

Collects data using port that can transfer files.

Answer 17

FTP collection

Question 18

Collects data by reading HTML tags

Answer 18

HTTP collection
Scraper

Question 19

A file system that allows access to files on multiple host computers which are shared over a computer network.

Answer 19

Distributed File System (DFS)
GFS (Google File System), HDFS (Hadoop Distributed File System), etc.

Question 20

A new type of data storage/retrieval
system that uses a less restrictive consistency model (BASE characteristics) than the traditional relational database.

Answer 20

NoSQL (Not Only SQL)
Hbase, Cassandra, Mongodb, CouchBase, Redis, Neo4J, etc.

Question 21

A technology that processes a large amount of data in a distributed parallel computing environment.

Answer 21

Distributed parallel processing
MapReduce

Question 22

A file system architecture for storing and processing large-scale and unstructured data in a distributed environment. It has the following characteristics.

Answer 22

Distributed File System (DFS)

Question 23

• A programming model designed for the parallel distributed processing of big data using inexpensive machines. This model can process large amounts of data in parallel using a program composed of a map procedure and a reduce method.
• Allows the analysis of large-scale data by processing data that has been distributed and stored in multiple machines.

Answer 23

MapReduce

Question 24

This technique provides insights by effectively transferring numbers, statistics, and valuable meanings, by classifying data for the user’s easy understanding, and by analyzing large-scale data.

Answer 24

Visualization technology

Question 25

Big data visualization methods (5)

Answer 25

1. Time visualization 1. Distribution visualization 1. Relationship visualization 1. Comparison visualization 1. Spatial visualization

Question 26

* Shows the passage of time. * Continuous, segmented

Answer 26

Time Visualization

Question 27

* Shows the relationship between the whole and the part, and the ratio * Pie, treemap

Answer 27

Distribution visualization

Question 28

* Shows the relationship between two or more variable * Bubble chart, histogram

Answer 28

Relationship visualization

Question 29

* Shows spaces and shadows intuitively * Heatmap, Stars

Answer 29

Comparison visualization

Question 30

* Shows information by mapping it on the map. * Including POI data

Answer 30

Spatial visualization

Question 31

Refers to the process of “discovering meaningful patterns from big data.”

Answer 31

Big data analytics

Question 32

**Classification of big data analytics** The primary purpose is to find patterns that describe the given data.

Answer 32

**Descriptive modeling** ... *Applied technique* Association rule, clustering, database, segmentation, visualization, etc

Question 33

**Classification of big data analytics** A model is created based on the given data, and is used to predict new input data.

Answer 33

**Predictive modeling** ... *Applied technique* Classification, Regression, time series analysis, Neural network, SVM

Question 34

**Classification of big data analytics** When the target is determined

Answer 34

**Supervised data** ... *Applied technique* Decision Tree, Neural network, Case-based reasoning

Question 35

**Classification of big data analytics** When there is no target. The correlation or similarity between data is analyzed with the focus on input variables

Answer 35

**Unsupervised data** .... *Applied technique* Association rule discovery, market basket K-means clustering

Question 36

Main methods of big data analysis

Answer 36

1. Logistic regression 1. Decision tree analysis 1. Neural network analysis 1. Text mining 1. SNA (Social Network Analysis) 2. Opinion Mining 3. Natural Language Processing (NLP)

Question 37

A statistical technique used to predict the possibility of an event’s occurrence (probability of occurrence) using a linear combination of independent variables.

Answer 37

Logistic regression

Question 38

A method of quantitative analysis that classifies an interested group into several subgroups or performs Prediction by drawing a decision tree chart.

Answer 38

Decision tree analysis

Question 39

A method of analysis that handles a problem with parallel/distributed/probabilistic calculation based on the idea that digital information is a network of nerve cells, rather than a method of processing digital information based on a deterministic binary computational model using the human brain itself as the model.

Answer 39

Neural network analysis

Question 40

A technology that extracts and processes useful information by applying natural language processing technology and document processing technology to unstructured/semi structured data.

Answer 40

Text mining

Question 41

An analysis methodology that analyzes and visualizes the relationship between objects - such as people, groups, organizations, computers and data - and the characteristics and structure of the network.

Answer 41

SNA (Social Network Analysis)

Question 42

* A technology that quickly analyzes the information the user wants and intelligently infers meaningful information from a large number of unstructured reviews such as SNS and replies. * It is used effectively for corporate marketing policies or public opinion analysis by extracting the hot topics of the social network service and analyzing the flow in real time.

Answer 42

Opinion mining

Question 43

* An artificial intelligence technology that understands, creates, and analyzes human language using computers. * The process of understanding natural language by analyzing human language mechanically to convert it into a form that can be understood by computers.

Answer 43

Natural Language Processing (NLP)

Question 44

* A ____ is an expert who can collect, organize, investigate, analyze, and visualize data. * The ____ provides information necessary for corporate/organizational decision-making by collecting, analyzing, and discovering the value of data, using various platform foundations and analysis infrastructures related to big data.

Answer 44

data scientist

Question 45

Capabilities of the data scientist

Answer 45

**Management** * - Business * - Data management * - Data analysis * - Change management **Technology –capability** * - Understanding statistical analysis tools * - Programming language * - RDBMS technology * - Distributed computing * - Mathematical knowledge

Question 46

Provides other means of processing than the tabular relations used in relational databases.

Answer 46

NoSQL database

Question 47

A non-relational distributed data repository that can be expanded horizontally, such as data replication and distributed storage on multiple servers focusing on the write speed for processing unstructured and ultra-high-capacity data.

Answer 47

NoSQL database

Question 48

**CHARACTERISTICS OF NOSQL** Provides a loose data structure that allows data to be processed at the petabyte level.

Answer 48

Processing of large-scale data

Question 49

**CHARACTERISTICS OF NOSQL** - Saves data relatively freely without a predefined schema. - Saves (data) in a simplified form such as the key value, graph, and document structure.

Answer 49

Use of flexible schemas

Question 50

**CHARACTERISTICS OF NOSQL** Supports scale-out, data replication, and distributed storage using multiple servers composed of Pc-level commercial hardware.

Answer 50

Inexpensive cluster configuration

Question 51

**CHARACTERISTICS OF NOSQL** - No query language like SQL in existing relational databases is provided. - A simple interface is provided by calling a simple API or HTPP.

Answer 51

Simple CLI (Call Level Interface)

Question 52

**CHARACTERISTICS OF NOSQL** NoSQL loads data by automatically dividing data items into the cluster environment.

Answer 52

High availability

Question 53

**CHARACTERISTICS OF NOSQL** While the relational DBMS focuses on ensuring logical structure and ACID, NoSQL makes the application process some of the integrity works instead of assigning them all to the DBMS.

Answer 53

Allow as much integrity is needed

Question 54

**CHARACTERISTICS OF NOSQL** The methods of saving data are largely divided into column, value, document, and chart, using a function that allows data storage and access using the key values, without the fixed data schema for data modeling.

Answer 54

Schema-less

Question 55

**CHARACTERISTICS OF NOSQL** NoSQL has a structure that allows expansion of the system’s scale and performance and distribution of the I/O load more easily, so that large-scale data can be created, updated, and queried, while not causing downtime for any clients and application systems that access the system, even if the system fails partially.

Answer 55

Elasticity

Question 56

**CHARACTERISTICS OF NOSQL** NoSQL provides query language, related processing technology, and API that can efficiently search and process data according to the characteristics of data even in a system composed of tens or thousand of servers.

Answer 56

Query

Question 57

**CHARACTERISTICS OF NOSQL** NoSQL has a structure in which memory-based caching technology is very important, and which can provide a high-performance response speed even for large-scale queries and be consistently applied to development and operation.

Answer 57

Caching

Question 58

**CHARACTERISTICS OF NOSQL** Partitioning allows a gradual node increase.

Answer 58

High scalability

Question 59

**CHARACTERISTICS OF NOSQL** There is no single point of failure, and data are available even though a certain node is down because they are replicated.

Answer 59

High availability

Question 60

**CHARACTERISTICS OF NOSQL** The result should be quickly returned based on memory instead of disk, which can be achieved by using the non-blocking write and low complexity algorithm.

Answer 60

High performance

Question 61

**CHARACTERISTICS OF NOSQL** Each write operation needs to be atomic.

Answer 61

Atomicity

Question 62

**CHARACTERISTICS OF NOSQL** Strong consistency is not needed, but the resulting consistency is sufficient (Read-Your-Writes).

Answer 62

Consistency

Question 63

**CHARACTERISTICS OF NOSQL** Data should be kept on a disk, not just in a volatile memory only.

Answer 63

Persistence

Question 64

**CHARACTERISTICS OF NOSQL** When a node is added or deleted, data should be automatically loaded without the need for data distribution or manual mediation, and there should be no constraints, such as distributed file system or shared storage, or any need for special hardware. Hardware should be operable in heterogeneous hardware.

Answer 64

Deployment

Question 65

**CHARACTERISTICS OF NOSQL** Data of various types such as key-value pairs, hierarchical data, and graphs should be modeled conveniently.

Answer 65

Modeling flexibility

Question 66

**CHARACTERISTICS OF NOSQL** Multiple GET that obtains a set of values for the provided key from a query, and queries that obtain data based on a specific range of keys, are needed.

Answer 66

Query flexibility

Question 67

**Description of NoSQL’s BASE properties** - Emphasis is placed on availability and the use of optimistic locking and queue. - Availability is ensured even with multiple failures and copies are stored in multiple storages.

Answer 67

Basically Available

Question 68

**Description of NoSQL’s BASE properties** - Node status is determined by the information transmitted from outside. - Updates between distributed nodes are updated when data reach the node.

Answer 68

Soft-State

Question 69

**Description of NoSQL’s BASE properties** The property of maintaining consistency optimally even though consistency is lost temporarily.

Answer 69

Eventually Consistent

Question 70

**Types of NoSQL** - The most basic NoSQL database that provides simple and fast Get, Put and Delete functions based on the key value. - Dynamo, Redis, MemcacheDB, etc.

Answer 70

Key-value based

Question 71

**Types of NoSQL** - An NoSQL database that expresses the entry attribute of the relational database as a node and the relationship as the edge between nodes, such as Neo4J, Flock DB, etc.

Answer 71

Graph based

Question 72

**Types of NoSQL** - An NoSQL database that stores data in rows in the column family, which corresponds to the table in the relational database. - Cassandra, Hbase, SimpleDB, etc.

Answer 72

Column family based

Question 73

**Types of NoSQL** - An NoSQL database that stores documents such as XML, JSON, BSON, etc. in the value part of the key-value database, such as NoSQL DB, MongoDB, CouchDB, etc.

Answer 73

Document based

Question 74

A data modeling technique that derives logical connection points using the data composition method with generalized notation and execution procedures

Answer 74

ACID-based data modelling

Question 75

Data modeling is closer to file structural design, rather than to general data modelingin which a data set that can be easily processed by the program is created

Answer 75

BASE-based data modeling

Question 76

This theory asserts that as it is impossible for a distributed data store to simultaneously satisfy all of data consistency, availability, and partition tolerance, only two should be strategically selected.

Answer 76

CAP theorem (Consistency, Availability, and Partition Tolerance)

Question 77

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** All nodes should show the same data at the same time (Each user should always view the same data)

Answer 77

Consistency

Question 78

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** Even if some nodes are down, it should not affect the other nodes. (All users should always be able to read and write data.)

Answer 78

Availability

Question 79

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** Even if some messages are lost, the system should operate normally. (The system should work properly in a physically distributed network environment.)

Answer 79

Partition Tolerance

Question 80

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** An exceptionally reliable type in which message loss can be prevented even if the system is down. Essential when a transaction is required. Example: General RDBMS.

Answer 80

C + A

Question 81

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** A performance type in which all nodes must perform well together. Examples: Google's BigTable, HyperTable, HBase.

Answer 81

C + P

Question 82

**BASE ATTRIBUTES OF NOSQL- CHARACTERISTICS OF THE NOSQL DATA MODEL** Essential for asynchronous store operations. Examples: Dynamo, Apache Cassandra, CouchDB, Oracle Coherence.

Answer 82

A + P

Question 83

A _____ refers to the network between the components constituting a given society.

Answer 83

social network