Mesos (CC, Resource Management)

Question 1

Background (Mesos)?
(Share the same cluster, improving utilization - avoid data replication)

Answer 1

Mesos is an open-source platform for cluster management. It was developed as a research project at the University of California, Berkeley, and is now owned by Apache.

It allows different frameworks, such as Hadoop and MapReduce, to share the same cluster, improving cluster utilization and avoiding data replication.

Question 2

Problem (Mesos)?

Answer 2

• There were a large number of cluster computing frameworks available, but resource sharing across frameworks was not possible.
• Each framework was developed independently, making optimal sharing of data and cluster resources difficult.

Question 3

Solution (Mesos)?

Answer 3

• Mesos shares resources in a fine-grained manner across frameworks, using a decentralized approach.
• It delegates control of scheduling to the frameworks.
• Mesos decides how many resources to offer to each framework, and the framework decides which resources to accept and which tasks to run on them

Question 4

Applications/Uses (Mesos)?

Answer 4

• Mesos is used by companies like Apple, Netflix, and eBay.
• Apple rewrote Siri in 2015 to be deployed on Mesos.
• Netflix used Mesos to ensure efficient resource allocation in their microservice-based architecture

Question 5

Strengths and Weaknesses (Mesos)?
(Data locality, Scabable, able to run multiple frameworks of the same framework)
(the paper focused on short tasks - not representative of all workloads, no authentication by default, no fault tolerance for node and scheduler).

Answer 5

Advantages:
Mesos enables efficient utilization of resources by allowing multiple frameworks to run concurrently.

• It improves data locality by allowing frameworks to take turns reading data stored on each machine.
• Mesos is scalable and robust to failures.
• It allows running multiple instances of the same framework.

Weaknesses:
- The original Mesos paper (2011) focused on short tasks, which may not be representative of all workloads.
I
- t weakly incentivizes the use of short tasks.

• Security: Mesos does not provide authentication by default.
• Fault Tolerance is not implemented: Users must implement fault tolerance for node, scheduler, and executor failures