Evolving the Agile Organization [P2]: Evidence-Based Management - The Evidence-Based Management Guide Flashcards
1
Q
What does measuring value enable?
A
Improvement and Agility
2
Q
Purpose of EBM?
A
- continuously improve customer outcomes, organizational capabilities, and business results under conditions of uncertainty.
- improve performance over time and refine goals based on intentional experimentation and evidence
3
Q
What are the key elements of this model? (Evidence-Based Management)
A
- Strategic goal
- Intermediate goals
- Immediate tactical goals
- Starting state
- Current state
4
Q
Define the Strategic Goal
A
- something important that the org wants to achieve
- very big and far away
- many uncertainties to achieve it - need empiricism to achieve it
5
Q
Define Intermediate Goals
A
- practical targets - path towards Strategic Goal
- path to Intermediate Goals is a bit uncertain but not completely unknown
6
Q
Define Intermediate Tactical Goals
A
critical near-term objectives towards which a team or group of teams will work towards Intermediate Goals
7
Q
Define Starting State
A
where the organisation is relative to the Strategic Goal when it starts its journey
8
Q
Define Current State
A
where the organisation is relative to the Strategic Goal at the present time
9
Q
How does an org progress toward the Strategic Goal?
A
- org runs experiments which involve forming hypothesis that are intend to advance the org toward their current Intermediate Goal.
- as they run these experiments and gather results, they use the evidence they obtain to evaluate their goals and determine their next steps to advance towards these goals
10
Q
How is a step defined?
A
do something towards the Intermediate Goal, inspect the result and adapt their next actions based on feedback
11
Q
What is The Experiment Loop?
A
- Hypothesis - one idea for how to make progress toward Next Target State
- experiment and measure - try out the idea. Measure lag and lead results
- inspect - based on the measures, what did you learn?
- adapt - what might you try next? Do we need to adjust the EBM Goal?
12
Q
When setting goals, what must orgs do?
A
- Define specific measurements that will indicate that the goal is achieved
- Make goals, measures and experiments transparent to encourage org alignment
13
Q
What is a Strategic Goal usually focused on?
A
Achieving a highly desirable but unrealized outcome for a specific group of people that results in improved happiness, safety, security, or well-being of recipients of some product or service
14
Q
What is Unrealized Value in EBM?
A
the satisfaction gap between a beneficiary’s desired outcome and their current experienece
15
Q
Broadly speaking, what are the three categories that measures fall into?
A
- Activities
- Outputs
- Outcomes
16
Q
Explain the Activities measure category
A
- things that people in the org do
- e.g. perform work, meetings, discussions, write code, create reports, conferences etc.
17
Q
Explain the Outputs measure category
A
- things that the org produces
- e.g. product releases (including features), reports, defect reports, product reviews etc
18
Q
Explain the Outcomes measure category
A
- desirable things that a customer or user of a product experiences
- represent some new or improved capabilities that the customer or user was not able to achieve before
- can be negative - in the case where the value a customer/user experiences declines from previous experiences e.g. service no longer available
19
Q
What is a common challenge orgs face around measurement?
A
- hard - measuring outcomes
- easier - measuring activities and outputs
20
Q
What does KVAs stand for?
A
Key Values Areas
21
Q
What are KVAs?
A
- areas that examine
- goals of the org - Unrealised Value
- current state of the org relative to those goals - Current Value
- responsiveness of the org in delivering value - Time-to-Market
- effectiveness of the org in deliver value - Ability-to-Innovated
- ⇒ org can understand where they are where they need to go
22
Q
What question is asked when considering Unrealised Value (uv)?
A
What is the additional potential value that could be achieved?
23
Q
What question is asked when considering Current Value (CV)?
A
What value is currently delivered by the org?
24
Q
What question is asked when considering Ability to Innovated (A2I)?
A
How effective is the org at improving value?
25
What question is asked when considering Time to Market (T2M)?
How long does it take to deliver new value?
26
For each of the KVAs, which are market value and which are organisational capacity?
- Unrealised Value - Market value
- Current Value - Market value
- Time-to-Market - Org cap
- Ability-to-Innovate - Org cap
27
What is the purpose of the Current Value?
- understand value that an org delivers to customers and stakeholders at the present time
- considers only what exists now, not future value
28
What are questions the orgs need to continually re-evaluate for current value?
- How happy are users and customers today? Is their happiness improving or declining?
- How happy are your employees today? Is their happiness improving or declining?
- How happy are your investors and other stakeholders today? Is their happiness improving or declining?
29
What is Unrealised Value?
The potential future value that could be realized if the organization met the needs of all potential customers or users
30
What opportunity is measured by Unrealized Value?
Gap between users’ current experience and their desired experience
31
What questions do the org need to ask to continually re-evaluate for unrealized value?
- Can any additional value be created by our org in their market or other markets?
- Is it worth the effort and risk to pursue these untapped opportunities?
- Should further investments be made to capture additional Unrealized Value?
32
Why do we look at T2M?
- ***“responsiveness”***
- To minimise the amount of time it takes for the org to deliver value
33
What happens if aren’t actively managing T2M?
ability to sustainably deliver value in the future is unknown
34
What does T2M refer to?
Time to market
35
What are questions that the org need to continually re-evaluated for T2M?
- How far can the org learn from new experiences and info?
- How fast can you adapt based on the info?
- How fast can you test new ideas with customers?
36
What does improving T2M help improve?
freq at which an org can potentially change CV
37
What is A2I?
- ***“effectiveness”***
- The effectiveness of an org to deliver new capabilities that might better meet customer needs
38
What does A2I refer to?
Ability to Innovate (A2I)
39
What is the purpose of looking at the A2I?
to maximise the org’s ability to deliver new capabilities and innovative solutions?
40
What questions should an org ask to continually re-evaluate their A2I?
- What prevents the org from deliver new value?
- What prevents customers or user from benefiting from that innovation?
41
What is the benefit of improving A2I?
helps org become more effective in ensuring that the work it does improves the value that its products or services deliver to customers or uses
42
How is a Feature defined?
distinguishing characteristics of a product
43
How is a requirement defined?
- something that someone thinks would be desirable in a product
- feature description is one kind of requirement
44
Why do we need hypothesizes and experiments?
your understanding of customer needs will likely different to what they actually need, so need to gather evidence and adapt
45
How is a hypothesis defined?
- proposed explanation for some observation that has not yet been proven (or disproven).
- In the context of requirements, it is a belief that doing something will lead to something else, such as delivering feature X will lead to outcome Y
46
How is an experiment defined?
a test that is designed to prove or reject some hypothesis
47
Every feature and every requirement really represent what?
a hypothesis about value
48
Do we need to build an entire feature to realised its value (or lack of?)
no - may be sufficient for a team to simply build enough of it to validate critical assumptions that would prove or disprove its value
49
What aspects from EBM are implicit parts of an agile approach
Explicitly forming hypotheses, measuring results, and inspecting and adapting goals based on those results
50
[Current Value (CV)]
What does Revenue per Employee measure?
The ratio (gross revenue / # of employees) is a key competitive
indicator within an industry. This varies significantly by industry.
51
[Current Value (CV)]
What does Product Cost Ratio
measure?
Total expenses and costs for the product(s)/system(s) being
measured, including operational costs compared to revenue.
52
[Current Value (CV)]
What does Employee Satisfaction
measure?
Some form of sentiment analysis to help gauge employee
engagement, energy, and enthusiasm.
53
[Current Value (CV)]
What does Customer Satisfaction
measure?
Some form of sentiment analysis to help gauge customer
engagement and happiness with the product.
54
[Current Value (CV)]
What does Customer Usage Index
measure?
Measurement of usage, by feature, to help infer the degree to
which customers find the product useful and whether actual usage
meets expectations on how long users should be taking with a
feature.
55
[Unrealized Value (UV)]
What does Market Share
measure?
The relative percentage of the market not controlled by the
product; the potential market share that the product might achieve
if it better met customer needs.
56
[Unrealized Value (UV)]
What does Customer or User
Satisfaction Gap
measure?
The difference between a customer or user’s desired experience
and their current experience.
57
[Unrealized Value (UV)]
What does Desired Customer
Experience or
satisfaction measure?
A measure that indicates the experience that the customer would
like to have
58
[Time-to-Market (T2M)]
What does Build and Integration
Frequency measure?
The number of integrated and tested builds per time period. For a
team that is releasing frequently or continuously, this measure is
superseded by actual release measures
59
[Time-to-Market (T2M)]
What does Release Frequency measure?
The number of releases per time period, e.g. continuously, daily,
weekly, monthly, quarterly, etc. This helps reflect the time needed
to satisfy the customer with new and competitive products.
60
[Time-to-Market (T2M)]
What does Release Stabilization
Period measure?
The time spent correcting product problems between the point the
developers say it is ready to release and the point where it is
actually released to customers. This helps represent the impact of
poor development practices and underlying design and code
base.
61
[Time-to-Market (T2M)]
What does Mean Time to Repair measure?
The average amount of time it takes from when an error is
detected and when it is fixed. This helps reveal the efficiency of
an organization to fix an error.
62
[Time-to-Market (T2M)]
What does Customer Cycle Time measure?
The amount of time from when work starts on a release until the
point where it is actually released. This measure helps reflect an
organization’s ability to reach its customer
63
[Time-to-Market (T2M)]
What does Lead Time measure?
The amount of time from when an idea is proposed, or a
hypothesis is formed until a customer can benefit from that idea.
This measure may vary based on customer and product. It is a
contributing factor for customer satisfaction.
64
[Time-to-Market (T2M)]
What does Lead Time for Changes measure?
The amount of time to go from code-committed to code
successfully running in production.
65
[Time-to-Market (T2M)]
What does Deployment Frequency measure?
The number of times that the organization deployed (released) a
new version of the product to customers/users.
66
[Time-to-Market (T2M)]
What does Time to Restore Service measure?
The amount of time between the start of a service outage and the
restoration of full availability of the service.
67
[Time-to-Market (T2M)]
What does Time-to-Learn measure?
The total time needed to sketch an idea or improvement, build it,
deliver it to users, and learn from their usage.
68
[Time-to-Market (T2M)]
What does Time to remove
Impediment measure?
The average amount of time from when an impediment is raised
until when it is resolved. It is a contributing factor to lead time and
employee satisfaction.
69
[Time-to-Market (T2M)]
What does Time to Pivot measure?
A measure of true business agility that presents the elapsed time
between when an organization receives feedback or new
information and when it responds to that feedback; for example,
the time between when it finds out that a competitor has delivered
a new market-winning feature to when the organization responds
with matching or exceeding new capabilities that measurably
improve customer experience.
70
[Ability to Innovate (A2I)]
What does Innovation Rate measure?
The percentage of effort or cost spent on new product capabilities, divided
by total product effort or cost. This provides insight into the capacity of the
organization to deliver new product capabilities.
71
[Ability to Innovate (A2I)]
What does Defect Trends measure?
Measurement of change in defects since last measurement. A defect is
anything that reduces the value of the product to a customer, user, or to the
organization itself. Defects are generally things that don’t work as intended.
72
[Ability to Innovate (A2I)]
What does On-Product Index measure?
The percentage of time teams spend working on product and value.
73
[Ability to Innovate (A2I)]
What does Installed Version
Index measure?
The number of versions of a product that are currently being supported.
This reflects the effort the organization spends supporting and maintaining
older versions of software.
74
[Ability to Innovate (A2I)]
What does Technical Debt measure?
A concept in programming that reflects the extra development and testing
work that arises when “quick and dirty” solutions result in later remediation.
It creates an undesirable impact on the delivery of value and an avoidable
increase in waste and risk.
75
[Ability to Innovate (A2I)]
What does Production
Incident Count measure?
The number of times in a given period that the Development Team was
interrupted to fix a problem in an installed product. The number and
frequency of Production Incidents can help indicate the stability of the
product.
76
[Ability to Innovate (A2I)]
What does Active Product
(Code) Branches measure?
The number of different versions (or variants) of a product or service.
Provides insight into the potential impact of change and the resulting
complexity of work.
77
[Ability to Innovate (A2I)]
What does Time Spent Merging Code Between Branches measure?
The amount of time spent applying changes across different versions of a product or service. Provides insight into the potential impact of change and the resulting complexity of work.
78
[Ability to Innovate (A2I)]
What does Time Spent
Context-Switching
measure?
Examples include time lost to interruptions caused by meetings or calls,
time spent switching between tasks, and time lost when team members are
interrupted to help people outside the team can give simple insight into the
magnitude of the problem.
79
[Ability to Innovate (A2I)]
What does Change Failure
Rate measure?
The percentage of released product changes that result in degraded
service and require remediation
