Big Data Lecture 06 Wide Column Stores Flashcards
What are the issues with RDBMS (relation database management systems) and how to solve them?
<ul><li>We hit the limit of storage of one machine --> scale up!</li><li>It is too slow --> cluster, replicate: scale out! (Difficult, very high maintainance costs.)</li><li>Solution: HBase - distributed database system!</li></ul>
How do wide-column stores compare to other methods? What is the only downside?
It marks all the columns green in this!<br></br><img></img><br></br>Small size per item, at around 10MB for optimal performance.<br></br><br></br>*Random access means being able to access any data as we wish without reading everything sequentially.
Who was the pioneer of Wide-Column Stores?
Google with its ‘Big Table’.
What is the motivation for making huge tables? How does it improve on RBDMS? Why don’t we replace RBDMS with it?
What is related stays together (stuff is denormalized), this is because the join operations in query time are very expensive.<br></br><br></br>RDBMS is good for updating stuff, which we cannot do with WCS.
How is each row identified?
It has a unique row ID, by which the records are sorted.
What are column families?
Residual of tables that were joined in the database, they are somehow realated, and they are stored together.<br></br><br></br>They must be pre-specified, but we can later add columns within the family.
What are the possible types of values in WCS?
The values are not typed, they are just byte objects. Some utility functions for reading the data are implemented.
What is the optimal size of an object in WCS?
<= 10 MB per cell, but it can be anything (text, image, webpage, JSON…)
What queries can be executed?
<ul><li>get (per rowID),</li><li>put (inserting per rowID, can also overwrite, can be partial, e.g. per column family),</li><li>scan (linearly scan certain rows and columns),</li><li>delete (per rowID).</li></ul>
What do we refer to as key-value model in general?
Any sort of hash map setting where the key points to the value (e.g. using hashtable).<br></br>
What is column oriented storage?
Columns of a database are physically stored on the disc together.
What are example of Wide Column Stores databases?
Google’s Big Table,<br></br>Apache HBase,<br></br>Cassandra.
How are tables stored in WCS?
In regions: cut of both rows (min inclusive and max exclusive) and columns.
What manages files underneath the WCS?
HDFS!
What is the architecture of HBase?
We have HMaster node (does DDL operations), which rules over processes on RegionServers (does DML operations), these all can be running on the same machine. Each RegionServer is an HDFS client.<br></br><br></br><img></img>
What is DDL and DML?
Data Definition Language (e.g. create table with specified columns),<br></br>Data Manipulation Language (e.g. add this row to this column).
How are regions saved?
HMaster assigns them to RegionServers.<br></br><img></img><br></br>If they are too big, they will be split, nodes are also rebalanced if they are too full, and if node fails, HMaster creates a new one.
How are the data persistently stored?
In an HFile, which is a sorted list of key-value pairs, ordered by:<br></br><ul><li>row,</li><li>column,</li><li>version (in reverse).</li></ul><div><img></img><br></br></div><div><img></img><br></br></div>
How are different users of WCS allowed to modify the same data? Relation to CAP theorem?
They are not, data is locked until the users stops modifying.<br></br><br></br>The data has linear versioning, there are no DAGs.<br></br><br></br>The system is not fully available always to everyone, but it is consistent.
How are nodes managed in WCS?
Using ZooKeeper, distriubuted system that manages all the heartbeats, and locks on files (and race conditions).
What makes WCS data access so efficient?
Each RegionServer is a HDFS client, it has a replica of its own data, so it does not have to request it, it can get it fast in the local system.<br></br><br></br>If RegionServer is assigned responsibility, but it does not have the data, it is fine as over a longer time, the data will be replicated onto this node, which will speed up the system again.
How is each value of the table stored in WCS, what are its parts?
It is yet another key value model!<br></br><img></img><br></br>Each record has its own key, and value, those are first (for memory efficiency) preceeded by key length (32 bits) and value length (32 bits).
How does Gamma code work?
The number of 1s terminated by a 0 tell us how many bits to read, and then pre-pend 1 (all non-zero numbers start with 1 in binary).<br></br><img></img>
How are data accessed in an HFile?
Using this beautiful long key, which stores all the information about the record (row, column family, column, versioning time-stamp and deleting marker).<br></br><img></img><br></br>Note, there is no column-qualifier lenght, because the whole key has a fixed length, so it can be inferred.<br></br><br></br>This key refers to the records.<br></br><img></img>
These are stored in HDFS, one region per server.
- Maximum size for one store is reached,
- overall memstore size is reached,
- the write-ahead log is full.
- RDBMS: B+-trees - they minimize seek-time bound by indexing the data.
- WCS: Log-Structrure-Merge trees - optimize by size for throughput.
The MemStore dumps a file, then it always merges whenever we have the file of the same size, so it goes up and up and up.
These files are stored at different level, either memory, disc or on further nodes.
- To create/update/delete a table: talk to HMaster,
- to know which server to talk to: talk to Meta table (cache that locally),
- to get the data, talk to RegionServer.
- Cache - pre-save HBlocks using standard caching criteria (do not use when reading all the data from a certain block, use batch processing instead, also do not use if doing random access),
- key ranges (know for certain that something is inside the file or not),
- Bloom filters, tell you with certainty that something is not in there, but only maybe if it is in there.
- uses tablets instead of regions (discontinuous ranges row ranges),
- it uses universe master to run other smaller HBase subsystems.
2. it shortcircuits DataNodes with HDFS.
