Data Storage and Processing Flashcards
(130 cards)
1
Q
What challenge do most existing IoT solutions face?
A
Most IoT solutions are tailored to specific verticals, leading to separate data silos, which makes it difficult to capture the full potential of IoT across multiple domains.
2
Q
Why is handling IoT data from different domains challenging?
A
IoT data come from various structures, sources, and descriptions, making it complex to integrate and process them properly across different domains.
3
Q
What is required to ensure interoperability of IoT devices?
A
IoT data must be stored in different network databases, shared among multiple nodes, analyzed by various tools, and interpreted by different machines to ensure interoperability.
4
Q
What is the Semantic Web, and how does it help with IoT data?
A
The Semantic Web, or linked data web, provides reasoning engines and tools to analyze and link IoT data meaningfully across various domains.
5
Q
What role does complex event processing play in IoT data analysis?
A
Complex event processing searches for dependencies and patterns in streaming IoT data, creating real-time insights to help businesses identify opportunities and threats early.
6
Q
Why is a single server insufficient for handling IoT data?
A
IoT data are often too large for a single server or database to handle, requiring distributed processing approaches like MapReduce.
7
Q
How does the MapReduce programming model help manage IoT data?
A
MapReduce distributes datasets across multiple databases to process the data separately and then recombines the results, making it possible to handle large volumes of structured and unstructured IoT data.
8
Q
How did the web evolve from its initial phase to the Semantic Web (Web 3.0)?
A
The web started as a collection of documents linked to each other and gradually evolved into the Semantic Web, where documents and pieces of data are meaningfully connected.
9
Q
What was unclear about the relationships between documents in the early phases of the web?
A
In the early phases, relationships between documents were unclear because they were not linked to specific pieces of data.
10
Q
What does the Semantic Web enable for users and machines?
A
The Semantic Web provides meaningful links between data, allowing users (both humans and machines) to explore and understand connections between pieces of information.
11
Q
What is linked data, and what does it create?
A
Linked data refers to semantically linking and integrating pieces of information across domains, creating a global web that connects data on topics like books, companies, and social media.
12
Q
How do machines use linked data in the Semantic Web?
A
Machines can connect distributed data sources, process new data as they appear on the web, and produce integrated results, enhancing applications like data browsers and search engines.
13
Q
What does a generic linked data browser allow users to do?
A
A generic linked data browser lets users browse a data source and travel along links to related sources, enhancing data exploration.
14
Q
What capability do linked data search engines provide?
A
Linked data search engines allow expressive query capabilities over aggregated data by crawling the global web of linked data.
15
Q
What is linked data?
A
Linked data refers to machine-readable, well-defined information published on the web that can be connected to external datasets from various sources.
16
Q
What format is used in linked data technologies to connect information?
A
Linked data technologies use the Resource Description Framework (RDF) format to create a web of data by linking different things.
17
Q
What kinds of data sources can linked data technologies connect?
A
Linked data can connect data sources ranging from geographically distributed database to heterogeneous systems that cannot interoperate at the data level.
18
Q
Who specified the rules for publishing data as part of the global web of data?
A
Tim Berners-Lee, the inventor of the World Wide Web, specified the rules for publishing data as part of the global web of data.
19
Q
What are the four linked data principles as specified by Tim Berners-Lee?
A
- Use Uniform Resource Identifiers (URIs) as names for things.
- Use HTTP URIs to help people look up the things’ names.
- Use RDF and SPARQL standards to provide useful data.
- Include links to other URIs to help people discover more things.
20
Q
Which two fundamental web technologies are relied on by the first two linked data principles?
A
The first two linked data principles rely on Uniform Resource Identifiers (URIs) and Hypertext Transfer Protocol (HTTP).
21
Q
How does RDF enhance linked data?
A
RDF supports a generic, graph-based data model that structures and links data describing things in the world, enhancing linked data.
22
Q
What does the Resource Description Framework (RDF) syntax encode and represent?
A
RDF encodes and represents web resources and data in a structure known as triples.
23
Q
What are the three components of an RDF triple?
A
- Subject: A resource identified by a URI.
- Predicate: A URI specifying the relationship between the subject and object.
- Object: A resource or literal (a basic string value) identified by a URI, related to the subject.
24
Q
What does the predicate represent in an RDF triple?
A
The predicate specifies the relationship between the subject and the object, represented by a URI.
25
What can the object in an RDF triple be?
The object can be either a resource or a literal (basic string value) identified by a URI.
26
How are subjects and objects in RDF triples similar to hypertext links?
Like hypertext links that connect documents, subjects and objects in RDF triples link items in various datasets, contributing to the web of data.
27
Give an example of an RDF triple relationship.
An example is "Berlin" (subject) and "Germany" (object) being related through the predicate "is the capital of," showing that Berlin is the capital of Germany.
28
What type of relationship exists in RDF between subject and object resources?
RDF defines a unidirectional relationship from the subject to the object resource.
29
Can a resource in RDF be used in multiple triples? If yes, in what roles?
Yes, a resource can be used in various triples with different roles: as a subject, predicate, or object.
30
What do multiple connections between RDF triples create?
Multiple connections between RDF triples create a connected graph of data.
31
In an RDF graph, how are resources and predicates represented?
* Resources are represented as nodes.
* Predicates (relationships between nodes) are depicted with lines connecting the nodes.
32
What is the significance of the connected graph in RDF?
The connected graph allows for multiple relationships between data points, enabling more complex and meaningful data linkages across different datasets.
33
What is a major benefit of using centralized systems for RDF datasets?
Benefit: No communication overhead between different nodes, as all data storage and queries are processed on a single machine.
34
What limits the capabilities of centralized systems in handling RDF datasets?
Limitation: The system is restricted by the memory and computational capacity of the single node.
35
How do distributed systems improve over centralized systems for RDF datasets?
Improvement: Distributed systems offer larger memory and computational power by utilizing multiple machines.
36
What are the potential drawbacks of distributed systems when processing RDF data?
Drawback 1: Expensive communication between machines.
Drawback 2: Intermediate data shuffling during complex queries can degrade system performance.
37
What is the DBPedia project, and what does it aim to do?
DBPedia Project: It extracts the structured content of Wikipedia and makes it available in RDF. It allows users to semantically query properties, relationships, and link to related datasets.
38
How does the DBPedia project improve user experience in applications?
Improvement: Applications can exploit information from other datasets to enhance the user experience by linking related information in RDF triples.
39
Why is RDF Schema (RDFS) used in conjunction with RDF?
RDF Schema (RDFS) is used to define classes of resources in RDF, enabling the categorization of things into hierarchical classes, which RDF alone does not support.
40
What is a resource in RDFS, and how is it classified?
A resource in RDFS is an instance of a certain class, and each class can have subclasses with additional descriptions.
41
Does RDF Schema (RDFS) specify how applications should use the class descriptions?
No, RDFS does not specify how an application should use the descriptions of resources in the classes.
42
How does linked data facilitate data abstraction in IoT?
Linked data uses common identifiers like International Resource Identifiers (IRIs), which integrate common data structures from various IoT sensors, enhancing data abstraction.
43
What role do machines play in interpreting linked data in IoT?
Machines can interpret data descriptions by extracting the origin, attributes, and understanding the relationships between the data and other related information.
44
What is the main purpose of the Internet of Things (IoT)?
The main purpose of IoT is to interpret the semantic data captured from various sources and sensors and transform it into actionable knowledge.
45
Why are IoT data considered useless?
IoT data are considered useless if they cannot be understood or interpreted, as they must provide meaningful insights to be actionable.
46
What challenges arise from the heterogeneous nature of IoT data?
The heterogeneous nature of IoT data presents challenges in ensuring interoperability among IoT devices due to the support for different protocols and data formats.
47
How does the Semantic Web contribute to IoT?
The Semantic Web provides analytical tools and best practices that facilitate data reasoning, help satisfy interoperability requirements, and enable effective integration and analysis of different sources of IoT data.
48
What is the relationship between the Internet of Things and the Semantic Web?
The relationship between IoT and the Semantic Web results in global interoperability between devices, enabling the generation of new services through effective data integration and analysis.
49
What are the key open approaches developed by the Semantic Web community for data analytics?
The key open approaches include sharing and reusing open data through linked data, linked vocabularies, and linked services.
50
How are semantic IoT data stored and managed in the Semantic Web?
Semantic IoT data are stored and managed in RDF databases as RDF graphs.
51
What language is used for querying and reasoning over RDF graphs?
SPARQL is used for querying and reasoning over the stored RDF graphs.
52
What is the role of semantic technologies in data analytics?
Semantic technologies help derive meaning from collected data, transforming it into actionable information.
53
What are some well-known methods and technologies employed by the Semantic Web to process IoT data?
Some well-known methods include linking data, real-time and linked stream processing, logic-based approaches, machine learning, distributed semantic reasoning, and cross-domain recommender systems.
54
What advantage does linking data provide in the context of the Semantic Web?
Linking data allows for meaningful connections not just between documents, but also between machine-readable and interpretable datasets, enhancing data interoperability and insight extraction.
55
What extension has been added to SPARQL to handle stream sensor data?
An extension called linked stream data has been added to SPARQL to help handle stream sensor data.
56
What does linked stream data allow SPARQL to do?
Linked stream data allows SPARQL to enrich stream sensor data with linked open data, which is freely used and distributed.
57
What mechanisms does the Semantic Web provide to ensure the consistency of IoT data?
The Semantic Web provides mechanisms to develop rules, check data consistency, and ensure that IoT data are logically valid.
58
What is the purpose of the Linked Edit Rules (LER) approach?
The Linked Edit Rules (LER) approach checks the consistency of data to ensure its validity, such as verifying that relative humidity values cannot be negative.
59
How is logic-based reasoning utilized in the analysis of IoT data?
Logic-based reasoning is used to analyze simple sensor data, such as temperature or humidity, and is characterized by being fast and easy to implement.
60
When are machine learning techniques and data mining approaches applied in the context of IoT?
Machine learning techniques and data mining approaches are applied to reason about complex semantic IoT data (e.g., electrocardiography [ECG] signals) where logic-based reasoning alone is insufficient.
61
Where is time-sensitive data processed in many IoT network architectures?
Time-sensitive data are processed at the edge of the network to reduce latency and save bandwidth.
62
What new challenges are introduced by processing data at the edge of the network?
New challenges include the need for reasoning at every layer of the IoT data management and computational stack (i.e., cloud, fog, and edge) and the difficulty for smart nodes to understand and interpret data from heterogeneous IoT sensors.
63
How does distributed reasoning improve reasoning latency in large datasets?
Distributed reasoning can improve reasoning latency by allowing processing to occur at the sensors and edge devices, thus reducing the amount of data sent to centralized locations.
64
What are the advantages of distributed reasoning over centralized reasoning?
21. Distributed reasoning is advantageous when:
* Data are distributed both logically and physically.
* Communication costs are negligible compared to problem solution costs.
* There is collaboration between the system's components to solve problems.
65
When is distributed reasoning particularly beneficial?
22. Distributed reasoning is beneficial when:
* Data are dynamic with ambiguous content.
* Data size exceeds the computational capacity of IoT devices.
* Sharing data and reasoning tasks can yield comprehensive intelligence.
66
What impact does distributed reasoning have on the performance of knowledge systems?
Distributed reasoning can improve knowledge system performance by splitting large computational tasks into sub-tasks that can be solved more efficiently.
67
What is the primary focus of traditional recommender systems?
Traditional recommender systems focus on a single vertical and assist users in finding their topic of interest among vast amounts of data in a specific domain.
68
What do Cross-Domain Recommender Systems (CDRS) use to provide recommendations?
CDRS use data and knowledge gained from multiple source domains to provide recommendations in a target domain, assuming there is information overlap between items and users across different domains.
69
How are cross-domain recommendation approaches classified?
26. Cross-domain recommendation approaches are classified into two categories based on how knowledge is exploited:
* Knowledge linking approach: User preferences are merged, and recommendations from both domains are combined.
* Knowledge sharing approach: The source domain transfers its data to the target domain for producing recommendations.
70
What is the purpose of Prefix.cc?
Prefix.cc simplifies the RDF development process by looking up URI prefixes.
71
What is the role of rdf-vocab?
rdf-vocab is an open-source project used by RDF developers to look up and search for linked data vocabularies.
72
What does the W3C RDF Validator do?
The W3C RDF Validator is an online service that checks and visualizes RDF documents.
73
What are some examples of data reasoners used in IoT analytical applications?
Examples of data reasoners include:
* CEL Description Logic (DL)
* Euler
* FaCT++
* HermiT Reasoner
* Java Expert System Shell (JESS)
* Jena Eyeball (a command-line semantics validator).
74
How can data reasoners be classified?
Data reasoners can be classified based on their linkage and discovery mechanisms or their usability.
75
What is complex event processing (CEP)?
CEP is a set of techniques used to aggregate, process, and analyze large amounts of streaming data to generate real-time insights from those events as they happen, even before the data is stored in databases.
76
When does CEP generate real-time insights from data?
CEP generates real-time insights as the events happen, before storing the data in databases.
77
How are insights generated in CEP?
Insights are generated by searching for dependencies and complex patterns in the incoming raw data.
78
What is the purpose of CEP in businesses and organizations?
CEP helps businesses and organizations identify opportunities and threats, enabling systems and applications to respond in real-time as quickly as possible.
79
How does CEP identify meaningful events?
CEP identifies meaningful events by continuously processing raw data and finding correlations between other events before the data is stored in databases.
80
What does CEP search for in the incoming raw data?
CEP searches for dependencies and complex patterns in the raw data to generate insights.
81
Why is it important for CEP to process data before it is stored in databases?
Processing data before storing it allows for real-time insights and enables faster responses to opportunities and threats.
82
What kinds of responses can CEP insights trigger in systems and applications?
CEP insights help systems and applications respond to opportunities and threats in real-time and as quickly as possible.
83
Why are "complex event processing" and "stream processing" used interchangeably?
They are used interchangeably because they rely on the same underlying technologies.
84
What is the main focus of complex event processing (CEP)?
CEP is focused on searching for complex patterns and dependencies between different events to identify a particular event.
85
What does stream processing primarily focus on?
Stream processing focuses on aggregating data in time windows and responding to a single event, often using time series data.
86
How does stream processing handle data from a surveillance camera?
It collects and processes the images (data) captured by the camera and responds to that event by analyzing the time series data.
87
What is Apache Kafka, and how is it related to stream processing?
Apache Kafka is an open-source stream processing engine, known for its streaming analytics, mission-critical applications, and data integration.
88
What are some applications of CEP in business monitoring?
CEP is used to monitor business processes and resources, helping businesses identify opportunities and problems at early stages.
89
How is CEP applied in sensor networks?
CEP is used in sensor networks to measure physical parameters, such as temperature, for predictive maintenance in industrial and manufacturing facilities.
90
What role does CEP play in predictive maintenance for industrial facilities?
CEP analyzes patterns from IoT devices to predict when equipment may need to be shut down or repaired.
91
How is RFID data used in smart retail stores with the help of CEP?
CEP processes RFID data to help management optimize store layout and inventory tracking.
92
How is complex event processing (CEP) applied in the stock market?
CEP is used to derive useful data about the stock market by analyzing real-time data streams and detecting complex patterns.
93
What are some well-known tools for complex event processing?
Some well-known tools include Hadoop/MapReduce, Amazon Kinesis Analytics, and Microsoft Azure Stream Analytics. LinkedIn uses Apache Samza, and Twitter uses Apache Storm.
94
How does CEP process data to generate high-level business information?
CEP detects complex patterns from real-time data streams, transforming low-level data into high-level business information.
95
Why is CEP considered a time-sensitive task?
CEP is time-sensitive because it requires ultra-low latency (typically less than a few milliseconds) to handle real-time data effectively.
96
What is an example of a connected vehicle IoT use case involving CEP?
An example is a vehicle where ice sensors warn of a slippery road, the weather forecast shows a high chance of precipitation, and other sensors in the car indicate unusual conditions. CEP processes all these events to provide critical information to the driver, connected cars, and road units.
97
Why is ultra-low latency important in CEP?
Ultra-low latency is crucial in CEP because it enables the system to process and respond to real-time events almost instantly, which is necessary in scenarios like connected vehicles.
98
What type of data is CEP typically handling in real-time?
CEP handles low-level data streams that are transformed into meaningful, high-level business information.
99
What role does CEP play in connected vehicle systems?
CEP processes data from various sensors (e.g., ice sensors, brakes, steering) to provide crucial information to drivers, other vehicles, and road infrastructure in real-time.
100
What is big data in the context of IoT?
Big data refers to the large volumes of structured and unstructured data produced by billions of connected IoT devices and sensors. This data is often too complex to be processed using conventional tools.
101
What are the five characteristics of big data, also known as the 5Vs?
2. The 5Vs are:
* Volume: Size of the data.
* Velocity: Speed at which data is generated.
* Variety: Types of data (structured, unstructured, semi-structured).
* Veracity: Trustworthiness and accuracy of the data.
* Value: Usefulness of the data in gaining insights.
102
What role does Apache Hadoop play in processing big data?
Apache Hadoop is an open-source software utility that allows for distributed storage and processing of large datasets using the MapReduce programming model.
103
What is the Map function in the MapReduce model?
The map function breaks down a dataset into key-value pairs and transforms it into a structured set of data. It operates on one key-value pair at a time.
104
What does the Reduce function do in the MapReduce model?
The reduce function combines the output of the map function into smaller sets of data tuples. It groups values associated with the same key and outputs reduced key-value pairs.
105
Explain how the MapReduce model can be compared to a shopping scenario.
In a shopping scenario, the grocery list is split into smaller lists (e.g., bakery, seafood), and multiple shopping carts collect items in parallel. They all meet at the cashier, speeding up the process compared to using a single cart.
106
What is the key limitation of traditional relational databases when dealing with big data?
Traditional relational databases are limited in handling the complexity and volume of unstructured data typically found in big data because they rely on tabular relations and SQL for querying structured data.
107
What are NoSQL databases, and how do they differ from traditional relational databases?
NoSQL databases are non-relational, highly scalable databases designed to store and retrieve unstructured data. They don't use tabular relations, and they support various models such as key-value, document, column-oriented, and graph models.
108
What are key-value store databases, and what are their advantages and limitations?
Key-value store databases store data as key-value pairs, with the key being a unique identifier and the value a large data field. They offer high performance but do not support complex queries, as only keys can be queried, not the values.
109
What is a column-oriented database, and how does it structure data?
Column-oriented databases store data in columns instead of rows. They use a key space that contains column families, each having rows and columns.
110
How do document-oriented databases store and retrieve data?
Document-oriented databases store data as JSON documents. They allow fast querying and are flexible, making them suitable for use cases such as IoT applications in healthcare.
111
What is the primary use case for graph-oriented databases?
Graph-oriented databases are used to store graph-based data, such as social network information. They focus on the relationships between data points, storing the data as originally produced without a predefined structure.
112
What is the main benefit of using the MapReduce programming model in data processing?
The main benefit of the MapReduce model is its ability to scale data processing across multiple machines, making it efficient for handling large datasets in distributed environments.
113
What does veracity refer to in the context of big data?
Veracity refers to the uncertainty, accuracy, and trustworthiness of the data in big data analytics.
114
What is the significance of value in the 5Vs of big data?
Value refers to the ability of data to provide useful insights or information that can be acted upon.
115
What tools are typically required to process big data?
Tools like Apache Hadoop are needed to process big data using distributed computing over several machines and servers..
116
What is the function of the Map stage in the MapReduce model?
The Map stage breaks down a dataset into key-value pairs, converting unstructured data into structured data.
117
What is the main challenge that IoT devices create in terms of data?
IoT devices produce large volumes of structured and unstructured data, often exceeding the processing capabilities of traditional databases.
118
Why do traditional relational databases struggle with big data?
Relational databases are designed for structured data with predefined relationships and cannot handle the complexity and volume of unstructured big data.
119
What are NoSQL databases, and why are they important for big data?
NoSQL databases are non-relational databases that are highly scalable and can store unstructured data. They are important because they can handle the complexities of big data without requiring tabular relations.
120
What are the four types of NoSQL databases?
* Key-value store: Stores data as key-value pairs.
* Column-oriented: Stores data in columns rather than rows.
* Document-oriented: Stores data as JSON documents.
* Graph-oriented: Stores data in graph format, focusing on relationships.
121
Describe key-value store databases and their limitations.
Key-value store databases store data as key-value pairs and offer high performance. However, they do not support querying or searching values, only the keys.
122
How do column-oriented databases structure their data?
Column-oriented databases store sparse tabular data in columns instead of rows, organized by key spaces that contain column families, rows, and columns.
123
What is a document-oriented database, and why is it suitable for IoT use cases?
A document-oriented database stores data as JSON documents. It provides flexibility and fast queries, making it ideal for IoT applications like healthcare that need real-time data access.
124
What is a graph-oriented database used for?
Graph-oriented databases are used to store graph-based data, such as social networks, where relationships between data are as important as the data itself.
125
Why are traditional data storage and analytical tools insufficient for handling data in the Semantic Web of Things?
Traditional tools cannot handle the massive, heterogeneous data produced by IoT devices, which requires special tools and techniques like semantic technologies and NoSQL databases.
126
What role do semantic technologies play in the Semantic Web of Things?
Semantic technologies, such as linking data, machine learning, distributed semantic reasoning, and cross-domain recommender systems, help derive meaning from the large amounts of data collected from IoT devices.
127
What is complex event processing (CEP)?
Complex event processing is a set of techniques used to aggregate, process, and analyze large amounts of streaming data to provide real-time insights as events occur.
128
How does complex event processing benefit businesses?
CEP helps businesses uncover patterns, identify opportunities, and detect potential threats in their early stages by analyzing real-time data streams.
129
Why are relational databases unsuitable for storing IoT data?
Relational databases struggle to handle complex, semi-structured, and unstructured data typical in IoT environments, making them unsuitable for big data storage.
130
How do NoSQL databases store IoT data, and what programming model do they rely on?
NoSQL databases store IoT data in documents, key-value pairs, or graph models and rely on the MapReduce programming model for processing large datasets.
