Bigdata Lineage

Question 1

What is Hadoop

Answer 1

• HDFS
• YARN
• MapReduce
• Hadoop Commons

Question 2

What is HBase

Answer 2

Database on top of Hadoop. Random read/write access. Store large tables (1M columns, 1B rows) atop of commodity hardware. Bigtable-like capabilities

Question 3

Hadoop history

Answer 3

2006
Yahoo

Question 4

HBase history

Answer 4

2007
Hadoop subproject

Question 5

What is Dynamo

Answer 5

Dynamo is a paper from Amazon. The paper describes an internal database at Amazon to handle their scale.

Question 6

Where does the term NoSQL comes from

Answer 6

2007. Dynamo paper helped launch the ‘NoSQL’ movement

Question 7

Cassandra history

Answer 7

2008, DataStax. Origin: Facebook. 2 Facebook engineers from Amazon made much of the dev)

Question 8

What is Cassandra

Answer 8

Column-oriented, scalable database

Question 9

What is a column oriented database

Answer 9

Excels in handing time series, «group by»

Question 10

What is Apache Pig

Answer 10

High-level scripting language to generate MapReduce jobs

Question 11

Apache Pig history

Answer 11

2007. Yahoo. Dead in 2017 (latest release 0.17)

Question 12

Why Pig is dead?

Answer 12

Preference SQL:
- Hive
- spark SQL
Performance et écosystème: spark

Question 13

Apache Hive history

Answer 13

2008. Facebook.
      Tend to be replaced by solutions designed for cloud storage such Iceberg, Hudi and Delta Lake

Question 14

What is Apache Hive

Answer 14

Querying tool, SQL-like interface (HiveQL). Creates MapReduce jobs

Question 15

Relation between Pig and Hive

Answer 15

Both provides a high-level solution to create MapReduce jobs. Pig uses a specific language (Pig Latin) while Hive uses a SQL-like language (HiveQL)

Question 16

Lakehouse architecture solutions

Answer 16

Hive, Iceberg, Hudi, Delta Lake

Question 17

Next generation solutions compared with Hive

Answer 17

Iceberg, Hudi and Delta Lake
They offer ACID transactions, schema evolution and data versioning. Designed for cloud storage

Question 18

What came after Hadoop

Answer 18

What slowed down Hadoop:

Cloud storage alternatives to HDFS
New Processing frameworks such as Spark
Docker, K8s
Hadoop Ozone

But Hadoop is still widely used

Question 19

What is Apache Impala

Answer 19

SQL engine for data stored in HDFS
Like spark but before spark - 2013, Cloudera
Process data in memory, in contrast to Hive
Replaced by Spark

Question 20

Impala vs Hive vs Spark

Answer 20

Impala: massively parallel processing in memory just like Spark
Hive: not in memory
Spark: not coupled with HDFS: general purpose