Distributed Systems

Question 1

What are the five aspects of a distributed system?

Answer 1

Distributed systems should be scalable, reliable, available, efficient, and manageable.

Question 2

What’s does scalability mean with respect to a distributed system?

Answer 2

Scalability is the capability of a system to grow (or shrink) to meet evolving demands.

Question 3

What are two forms of scalability? How do they differ?

Answer 3

Scalability can be horizontal or vertical. Horizontal generally means you add more nodes or servers to distribute and accommodate the load. Vertical generally means increasing the power or resources of an existing node or server.

Question 4

Which is generally easier to implement or support? Horizontal or vertical scaling?

Answer 4

Horizontal is typically easier to add additional components to existing pool (especially in Kubernetes). Vertical can be limited to capacity of a single server (or pod) and can sometimes involve downtime.

Question 5

What are good examples of database technologies that easily scale horizontally? What about vertically?

Answer 5

Cassandra and MongoDB both support easily scaling horizontally. MySQL is something that supports scaling vertically.

Question 6

What does reliability mean within a distributed system?

Answer 6

Reliability means that a system should still operate and respond correctly even in the presence of errors, faults, or one or more failing components.

One failing component should always be able to be replaced by a healthy one.

Question 7

What’s a real-world example of a company supporting reliability?

Answer 7

Amazon and their shopping cart transactions. A transaction should never be aborted due to the machine processing it failing, there should be a replica with that transaction to process it should a failure occur.

Question 8

What’s an important and potentially costly solution to support reliability?

Answer 8

Reliability can be achieved through redundancy by eliminating single points of failure. This can be costly and sometimes as the expense of complexity to support it.

Question 9

What is fault-tolerance?

Answer 9

Fault-tolerance is the concept of allowing a system to continue to function, potentially at a reduced rate, when some of its components fail (either through absorbing the error, retrying, etc.)

Question 10

How do reliability and fault-tolerance differ?

Answer 10

Reliability refers to the entire system and end-to-end correctness. Fault-tolerance focuses on the ability of the system to continue operating when components fail.

Question 11

Perspective-wise, where do reliability and fault-tolerance fall?

Answer 11

Reliability is a more user-centric concept (e.g. does the system meet my needs) whereas fault-tolerance is a more system centric concept (e.g. how are failures handled).

Question 12

How do you measure reliability?

Answer 12

Metrics such as uptime, error rates, and time between failures.

Question 13

How do you measure fault-tolerance?

Answer 13

Metrics such as recovery/failover time (I.e. how quickly a failure can be detected, isolated, and recovered).

Question 14

Explain the relationship between availability and reliability.

Answer 14

If a system is reliable, it is available; however availability does not imply reliability.

Question 15

What is serviceability? How does it play a role in distributed systems?

Answer 15

Serviceability is how easy a system is to maintain (e.g. ease of diagnosing issues, early detection of failures or issues, downtime during maintenance or upgrades).

Question 16

What is a load balancer? What do they do?

Answer 16

A load balancer is a component that spreads traffic to help reduce overall load and responsiveness.

Question 17

How does a load balancer work?

Answer 17

While requests are being distributed, the load balancer will keep track of the statuses (or potentially other metrics) to determine which resources should or shouldn’t have traffic directed to them.

Question 18

What problem is a load balancer designed to solve?

Answer 18

It can mitigate against single points of failure (e.g. if a single web server fails, traffic will be redirected to healthy, responsive ones).

Question 19

What are three levels of a typical application where a load balancer would be appropriate?

Answer 19

In between the user and the web servers (distributes traffic across multiple servers), between web servers and other application components (e.g. services like distributed caches, etc.), between internal services and databases.

Question 20

What are a few benefits of load balancers?

Answer 20

Faster user experience, increased availability, metrics for predictive analytics, fewer or less stressed components or services.

Question 21

What are the two factors that a load balancer uses to distribute traffic?

Answer 21

Health checks and the selected routing algorithm.

Question 22

What are some of the common load balancing algorithms?

Answer 22

Least connection (least busy server), least response time (server responding the fastest), least bandwidth (server serving the least traffic), round robin (routes through servers in order, repeats), weighted round-robin (same but servers may be weighted based on resources or type), hash (specific traffic is routed based on some type of calculated hash, e.g. same ip goes to same server).

Question 23

If a load balancer is unreliable, how can you prevent this?

Answer 23

You can implement a load-balancing group or cluster to function as a load-balanced, load-balancer (active/passive).

Question 24

What is caching? What principal does it take advantage of?

Answer 24

Caching involves use of high speed storage layer that sits between the calling application and the original source of data. It takes advantage of the locality of reference principle.

Question 25

What is the goal of caching?

Answer 25

Caching prevents unnecessary calls to more expensive storage layers for common lookups.

Question 26

What are four common use-cases or types of cache?

Answer 26

In-memory caches, disk caching, database caching, and CDNs.

Question 27

What is in-memory caching? How does it work and what’s a good example of it?

Answer 27

In-memory caching involves storing data within memory on the application server and referencing it directly. Redis or Memcached are common examples of this.

Question 28

What is disk-caching? How does it differ from in-memory caching? What’s a common use-case for it?

Answer 28

Disk-caching is the idea of storing data on disk on the application server instead of in-memory. While not as fast as reading from memory, it’s typically faster than lookups from external sources. Database queries and file system data are good examples of something cached on disk.

Question 29

What is database caching? What are some common forms of it?

Answer 29

Database caching is when commonly accessed data can be cached in the database itself. Result sets and queries (query execution plans) are common examples of what is stored in a database cache.

Question 30

What is client-side caching? What might be stored there?

Answer 30

Client-side caching, as the name implies, involves caching data on the client’s machine (e.g. web app, browser) to prevent calls to the server. This is common used to store static or data that seldom changes such as images, JavaScript, CSS or other client resources.

Question 31

What’s server-side caching? What’s commonly stored there?

Answer 31

Much like client-side is stored on the client, data can be cached on the server as well. Web servers can commonly cache page responses (full or partial), objects, etc.

Question 32

What is CDN caching? What scenarios is it good for?

Answer 32

CDN caching involves storing data on a distribution of servers (typically geographically distributed) to improve data access from remote locations. Images, video, and other static assets are commonly stored. Typically CDNs are reserved or commonly used for global or very large-scale applications.

Question 33

What is DNS caching? What is it good for?

Answer 33

DNS caching involves the Domain Name Server caching IP addresses to avoid frequent lookups via domain. This can be extremely useful to greatly improve performance.

Question 34

Why is cache invalidation important?

Answer 34

If caches aren’t properly evicted or invalidated, users can experience stale data or potentially system consistency issues, especially in systems with a large number of disparate caching mechanisms.

Question 35

What are the three primary schemes for cache-invalidation during writes? How do each work?

Answer 35

Write-through, write-around, and write-back. Write-through which will write to the cache and database simultaneously to avoid stale cache data. It can have higher latency as there are two writes as opposed to one. Write-around will only write to the primary data store (not the cache). This avoids the cache being flooded with writes, but recently accessed data will need to be read directly from the data store. Write-back involves only writing to the cache and later reflecting those writes to the data store. This can be highly performant but risky as crashes could cause data in the cache to be lost.

Question 36

What are the five primary forms of cache invalidation operations?

Answer 36

Purge, refresh, ban, TTL, and stale-while-revalidate. Purge removes content from the cache. Refresh reads from the source, even if content is in the cache and update it. Ban will invalidate the cache based on some type of pattern or criteria. TTL will check the cache and if the data is expired (via the TTL) it will refresh it from the data store. Stale-will-revalidate will serve stale content during an update and refresh the cache asynchronously once finished, typically reserved for CDNs.

Question 37

What are the two most popular cache read strategies? How do they differ?

Answer 37

Read-through means the cache is responsible for reading from the source during misses. Read-aside means the application is responsible for reading, checking the cache, or from the source during misses. Read-through can be simpler to implement but doesn’t provide the same fine-grain control or fault-tolerance as read-aside.

Question 38

What are the six most common cache eviction policies?

Answer 38

First In First Out (FIFO): The cache evicts the first block accessed first without any regard to how often or how many times it was accessed before. Last In First Out (LIFO): The cache evicts the block accessed most recently first without any regard to how often or how many times it was accessed before. Least Recently Used (LRU): Discards the least recently used items first. Most Recently Used (MRU): Discards, in contrast to LRU, the most recently used items first. Least Frequently Used (LFU): Counts how often an item is needed. Those that are used least often are discarded first. Random Replacement (RR): Randomly selects a candidate item and discards it to make space when necessary.