Data Engineering Fundamentals

Question 1

What are the three main types of data?

Answer 1

Structured, Unstructured, and Semi-structured

Question 2

Give an example of structured data.

Answer 2

Database table, CSV file, Excel spreadsheet

Question 3

What is a key characteristic of unstructured data?

Answer 3

It doesn’t have a predefined schema or structure.

Question 4

Give an example of unstructured data.

Answer 4

Image, audio file, email

Question 5

What is semi-structured data?

Answer 5

Data with some organization, like tags or hierarchies, but not as rigid as structured data.

Question 6

What are the three Vs of Big Data?

Properties of Data

Answer 6

Volume, Velocity, Variety

Question 7

What does “Volume” refer to in the context of data?

Answer 7

The amount or size of data.

Question 8

What does “Velocity” refer to in the context of data?

Answer 8

The speed at which data is generated and processed.

Question 9

What does “Variety” refer to in the context of data?

Answer 9

The different types and sources of data.

Question 10

What is a data warehouse optimized for?

Data Warehouses vs. Data Lakes

Answer 10

Complex queries and analysis of structured data.

Question 11

What is a key characteristic of a data lake?

Answer 11

It can store vast amounts of raw data in its native format.

Question 12

What is the schema approach for a data warehouse?

Answer 12

Schema-on-write (schema is defined before data is loaded)

Question 13

What is the schema approach for a data lake?

Answer 13

Schema-on-read (schema is defined when data is read)

Question 14

What is a data lakehouse?

Data Lakehouse and Data Mesh

Answer 14

A hybrid architecture combining features of data lakes and data warehouses.

Question 15

What is a key concept of a data mesh?

Answer 15

Domain-based data management.

Question 16

What does ETL stand for?

ETL Pipelines

Answer 16

Extract, Transform, Load

Question 17

What happens in the “Transform” stage of an ETL pipeline?

Answer 17

Data is cleaned, converted, and prepared for the target system.

Question 18

What is CSV commonly used for?

Data Formats

Answer 18

Storing data in a tabular format, often for spreadsheets or databases.

Question 19

What is JSON commonly used for?

Answer 19

Data interchange, especially in web applications.

Question 20

What is a key advantage of Avro?

Answer 20

It stores data and its schema together.

Question 21

What is Parquet optimized for?

Answer 21

Analytics and efficient querying of large datasets.

Question 22

What is data lineage?

Data Modeling and Lineage

Answer 22

Tracking the flow and transformation of data.

Question 23

What is a star schema?

Answer 23

A data modeling technique often used in data warehouses with fact tables and dimensions.

Question 24

What is the purpose of indexing in a database?

Database Performance Optimization

Answer 24

To speed up data retrieval.

Question 25

What is database partitioning?

Answer 25

Dividing a database into smaller, more manageable parts.

Question 26

What is the goal of stratified sampling? ## Footnote Data Sampling

Answer 26

To ensure representation of different subgroups in a sample.

Question 27

What is data skew? ## Footnote Data Skew

Answer 27

An imbalance of data across partitions in a distributed system.

Question 28

Give an example of a technique to address data skew.

Answer 28

Adaptive partitioning, salting, repartitioning.

Question 29

What is data completeness? ## Footnote Data Validation

Answer 29

Ensuring all required data is present.

Question 30

What is data consistency?

Answer 30

Ensuring data values are consistent across different datasets

Question 31

What does the COUNT() function do in SQL? ## Footnote SQL

Answer 31

Counts the number of rows.

Question 32

What does the GROUP BY clause do in SQL?

Answer 32

Groups rows based on the values in one or more columns.

Question 33

What is the purpose of pivoting in SQL?

Answer 33

To turn row-level data into columnar data.

Question 34

What does git init do? ## Footnote Git

Answer 34

Initializes a new Git repository.

Question 35

What does git add do?

Answer 35

Adds changes to the staging area.

Question 36

What does git commit do?

Answer 36

Records changes to the repository.

Question 37

What does git branch do?

Answer 37

Lists, creates, or deletes branches.

Question 38

What does git checkout do?

Answer 38

Switches to a different branch.

Question 39

What does git merge do?

Answer 39

Combines changes from different branches.

Question 40

What does git push do?

Answer 40

Sends local commits to a remote repository.

Question 41

What does git pull do?

Answer 41

Fetches changes from a remote repository and merges them.

Question 42

What does git stash do?

Answer 42

Temporarily saves changes.

Question 43

What does git rebase do?

Answer 43

Reapplies commits onto a different base branch.