Azure Databricks

Question 1

What is medallion architecture?

Answer 1

A medallion architecture is a data design pattern used to logically organize data in a lakehouse, with the goal of incrementally and progressively improving the structure and quality of data as it flows through each layer of the architecture (from Bronze ⇒ Silver ⇒ Gold layer tables).

Question 2

In medallion architecture, what is the bronze layer?

Answer 2

The landing zone for raw data.

Question 3

In medallion architecture, what is the silver layer?

Answer 3

cleansed and conformed data

In the Silver layer of the lakehouse, the data from the Bronze layer is combined, organized, and cleaned up to create a comprehensive view of the important aspects of the business, such as customers, stores, transactions, and reference tables. This helps to ensure that the Silver layer provides a unified and reliable representation of the enterprise’s key information.

Question 4

T or F: From a data modeling perspective, the Silver Layer has more 3rd-Normal Form like data models

Answer 4

True

Question 5

In medallion architecture, what is the gold layer?

Answer 5

curated business-level tables. Uses more de-normalized and read-optimized datamodels with fewer joins. (flattened data)

Question 6

What are the benefits of a lakehouse architecture? (5 items)

Answer 6

Simple data model

Easy to understand and implement

Enables incremental ETL

Can recreate your tables from raw data at any time

ACID transactions, time travel

Question 7

What is the concept of a data mesh?

Answer 7

Bronze and silver tables can be joined together in a “one-to-many” fashion, meaning that the data in a single upstream table could be used to generate multiple downstream tables.

Question 8

What is databricks?

Answer 8

Databricks offers a powerful workspace that integrates various components and tools to simplify the end-to-end data lifecycle. It provides capabilities for data ingestion, data preparation, data exploration, machine learning, and visualization.

Question 9

What programming languages does databricks support?

Answer 9

Using Jupyter notebooks, The platform supports multiple programming languages, including Python, Scala, R, and SQL,

Question 10

T or F: Databricks has Apache Spark capabilities

Answer 10

T

Question 11

T or F: You can manage spark clusters through APIS with datbricks?

Answer 11

T

Question 12

What do databricks alerts do?

Answer 12

you could set up an alert to monitor certain data streams and then automatically create support tickets if those data streams or queries exceeds certain thresholds.

Question 13

What are apache spark clusters?

Answer 13

Spark is a distributed data processing solution that makes use of clusters to scale processing across multiple compute nodes. Each Spark cluster has a driver node to coordinate processing jobs, and one or more worker nodes on which the processing occurs.

Question 14

What is a databricks file system?

Answer 14

The nodes in a spark cluster have access to a shared, distributed file system in which they can access and operate on data files. The Databricks File System (DBFS) enables you to mount cloud storage and use it to work with and persist file-based data.

Question 15

What is a notebook?

Answer 15

One of the most common ways for data analysts, data scientists, data engineers, and developers to work with Spark is to write code in notebooks.

Question 16

What is a hive metastore?

Answer 16

Hive is an open source technology used to define a relational abstraction layer of tables over file-based data. The tables can then be queried using SQL syntax.

Question 17

What is a delta lake?

Answer 17

Delta Lake builds on the relational table schema abstraction over files in the data lake to add support for SQL semantics commonly found in relational database systems. SQL semantics are ie ddl statements, dml statements, etc

Question 18

How is databricks used as a data workflow tool?

Answer 18

it can be used to flow data from one source to another and manipulate that data along the way. So what that means is that the tools that integrate with Databricks best are the tools that accept the data flow. So for example, Databricks can flow its data to data stores like Azure Synapse Analytics for extra analytics, of course, it can flow its data to something like Azure Data Lake too.

Question 18

How is Azure Active Directory used in Databricks?

Answer 18

Each user or even groups for that matter, can be granted various levels of permissions to your resource or even your data.

Question 19

What is an azure databricks workspace?

Answer 19

workspace is an analytics platform that’s actually based on Apache Spark. Databricks integrate Spark into Azure and makes the interaction with Spark seamless for collaborators like data engineers and data scientists.

Question 19

What is a sql warehouse?

Answer 19

SQL Warehouses are relational compute resources with endpoints that enable client applications to connect to an Azure Databricks workspace and use SQL to work with data in tables.