Lecture 7

Question 1

Describe the main problem Mesos framework is trying to address

Answer 1

• Difficult for a single framework to manage resources of clusters

Aim:

• To run multiple frameworks in a single cluster to maximise utilisation + share between frameworks

Question 2

What are the common solutions for sharing clusters?

Answer 2

Statically partition the cluster and run framework per partition

• Low utilisation
• Rigid partitioning which does not match real time changing application demands
• Mismatch between allocated granularity of static partitioning of current application

Question 3

What is Mesos and Goals?

Answer 3

Is a common resource sharing layer which diverse frameworks can run for accessing cluster resources

Goals

• High utilization
• Support diverse frameworks: Has different scheduling needs
• Scalability: to 10,000’s nodes
• Reliability: in face of failures and high availability

Question 4

What are the Design Elements?

Answer 4

• Fine-grained sharing:

Allocation at the level of tasks within a job, improves: utilisation, latency and data locality

• Resource offers:

Simple, scalable application-controlled scheduling mechanism

Question 5

How Mesos use resource offer to allocate resources to different frameworks?

Answer 5

• Simple, scalable application-controlled scheduling mechanism
• Offers different resources to frameworks to pick to launch
• Rejects offers that do not satisfy with the constraints
• The framework has to wait a long time to receive an offer to satisfy the constraint – time consuming

Pros: Keeps Mesos simple, expandable for future frameworks

Cons: Decentralised solutions are not optimal

Question 6

Draw the Mesos Architecture diagram

Answer 6

Question 7

Describe the different responsibilities of scheduling between the Mesos master + the framework schedulers

Answer 7

Mesos Terminology

• Mesos Master: Fine-grained sharing across frameworks
• Mesos Slave: On each node
• Frameworks that run tasks on slave nodes

Question 8

Describe the Mesos Architecture

Answer 8

• Fair sharing based on generalisation of max-min fairness for multiple resources
• Strict priorities
• Mesos revoke (kill) tasks when framework uses up the resources for a while

Question 9

Describe how Omega uses shared state to allocate resources to different frameworks

Answer 9

1. A framework’s scheduler makes a decision
2. Updates the shared the cell state copy in atomic commit
3. One must succeed
4. The scheduler resyncs its local copy

Question 10

Main Advantage of Omega

Answer 10

Fast

Cell state: The master copies the resources and allocates the cluster

Each framework maintains its own scheduler. Each scheduler has its own private, local, frequently updated copy of its cell to make decisions.

Question 11

Disadvantages of Omega

Answer 11

Under certain conditions

• Omega schedulers operation in parallel
• Schedulers do incremental transactions to avoid starvation
• Performance viability of shared-state determined by the frequency which transactions will fail + costs!