Twitter Heron (DIS, Stream Processing)

Question 1

Background (Twitter Heron)?

Answer 1

• Twitter Heron is a stream processing system designed as an alternative to Apache Storm. It was developed to address the limitations of Storm as Twitter’s data processing needs grew.

Question 2

Problem (Twitter Heron)?

Answer 2

• Apache Storm, while serving Twitter for several years, exhibited limitations in scaling, debuggability, performance, and manageability as Twitter’s use cases and data volume increased.
• Storm’s architecture had a single point of failure (Nimbus), made debugging complex, and led to inefficient resource utilization.

Question 3

Solution (Twitter Heron)?

Answer 3

• Heron was designed with improved reliability, fault tolerance, debuggability, scalability, and performance compared to Storm.
• It provides process-level isolation, running each component in a separate JVM process for better resource management and failure containment.
• It eliminates the single point of failure by distributing responsibility across multiple components.

Question 4

Applications/Uses (Twitter Heron)?

Answer 4

Twitter Heron is used within Twitter.

Question 5

Strengths and Weaknesses (Twitter Heron)?

Answer 5

• Reliability and Fault Tolerance: Heron eliminates the single point of failure present in Storm.
• Scalability and Performance: Heron is faster and more resource-efficient than Storm.
• Debuggability: Heron offers better debuggability due to isolated JVM instances and transparent component status.
• Manageability: Heron is easier to manage and visualize compared to Storm.