{ "@context": "https://schema.org", "@type": "Organization", "name": "Brainscape", "url": "https://www.brainscape.com/", "logo": "https://www.brainscape.com/pks/images/cms/public-views/shared/Brainscape-logo-c4e172b280b4616f7fda.svg", "sameAs": [ "https://www.facebook.com/Brainscape", "https://x.com/brainscape", "https://www.linkedin.com/company/brainscape", "https://www.instagram.com/brainscape/", "https://www.tiktok.com/@brainscapeu", "https://www.pinterest.com/brainscape/", "https://www.youtube.com/@BrainscapeNY" ], "contactPoint": { "@type": "ContactPoint", "telephone": "(929) 334-4005", "contactType": "customer service", "availableLanguage": ["English"] }, "founder": { "@type": "Person", "name": "Andrew Cohen" }, "description": "Brainscape’s spaced repetition system is proven to DOUBLE learning results! Find, make, and study flashcards online or in our mobile app. Serious learners only.", "address": { "@type": "PostalAddress", "streetAddress": "159 W 25th St, Ste 517", "addressLocality": "New York", "addressRegion": "NY", "postalCode": "10001", "addressCountry": "USA" } }

Unit I - Module I Cost Estimating Basics Flashcards

(104 cards)

1
Q

Which of the following are the three major types of cost estimates?

A. Analysis of Alternatives, Budget Estimate, Independent Cost Estimate

B. Budget Estimate, Life Cycle Cost Estimate, Independent Cost Estimate

C. Independent Cost Estimate, Life Cycle Cost Estimate, Rough-Order-of-Magnitude

D. Budget Estimate, Independent Cost Estimate, Economic

A

B. Budget Estimate, Life Cycle Cost Estimate, Independent Cost Estimate

The three major types of cost estimates are Life Cycle Cost Estimate (LCCE),
Independent Cost Estimate (ICE), and Budget Estimate. Other types include
Rough Order of Magnitude (ROM), Estimate At Complete (EAC), Independent
Cost Assessment (ICA), Analysis of Alternatives (AoA), and Economic
Analysis (EA). “Economic Evaluation” is not a term commonly used to
describe a type of cost estimate.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

True or False. A Cost Analysis Requirements Description (CARD) ensures that cost projections are based on a common definition of the system and the acquisition program.

A

True.

The purpose of a CARD is to collect, integrate, and coordinate technical, programmatic, and
schedule information necessary to estimate program costs.

A CARD ensures that cost projections developed by the analyst are based
on a common definition of the system and the acquisition program.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

A Work Breakdown Structure…

A. Provides a financial reporting framework

B. Relates the work scope elements to each other and to the end products)

C. Evolves as system/program evolves

D. All of the above

A

D. All of the above

A Work Breakdown Structure (WBS) provides a common frame of reference for relating
job tasks to each other and relating project costs at the summary level of detail.
The WBS may evolve with the program.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

True or False. Data collection should always begin after determining the estimating methodology.

A

False

In our standard process, we have data collection occurring after the development of the WBS and establishment of the program/system baseline.

The following steps
include data analysis and methodology selection. That being said, it is often a good
idea to have in mind potential methodologies before commencing data collection.

As we stress throughout CEBoK, data collection and data analysis should be
considered an iterative process.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Cost estimating and analysis at its best is an interdisciplinary mix which includes which of the following?

A. Economics, Sociology, and Mathematics

B. Statistics, Accounting, and Political Science

C. Probability, Marketing, and History

D. Engineering, Operations Research, and Statistics

A

D. Engineering, Operations Research, and Statistics

Some of the disciplines useful in cost estimating and analysis are Engineering, Mathematics, (including Probability and Statistics), Economics, Operations Research, Budgeting Accounting, Marketing, and Computer Science. Sociological, political, and historical considerations are a bit too far afield to have a direct bearing.

“Historical” in the cost estimating sense refers to the data that serves as the basis for our estimates.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

True or False.

A Budget Estimate is also known as a Total Ownership Cost (TOC) estimate.

A

False.

TOC Estimate is a synonym for the more commonly used Life Cycle Cost Estimate (LCCE).

This type of estimate includes Research and Development (R&D), Production, Operations
and Support (O&S), and Disposal. A Budget Estimate generally spans a shorter time period (five or six years) that does not encompass all of these phases.

It also probably only corresponds to one or two line items in the budget, whereas a TOC Estimate or LCCE
encompasses all costs, regardless of funding source.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

True or False.

Training and logistics plans should always be described in the Program/System Baseline documentation.

A

True.

The Program/System Baseline documentation should include all of the information found in a CARD (Cost Analysis Requirements Description), spanning system technical description,
work breakdown structure (WBS), predecessor system description, manpower requirements, risk, operational concept, deployment, logistics support concept, training, acquisition schedule, acquisition strategy, system test and evaluation (ST&E), and environmental
impacts analysis.

The training plan should include types of user training or certification, types of training or certification needed for system maintenance or administration personnel, training provider and location,
seat requirements, and frequency of training.

Logistics and supportability considerations are very important, because O&S is often the majority of life cycle cost (LCC), but because they are further in the future, they may not receive as much attention as they should.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

The cost estimating process includes which of the following?

A. Data Collection and Analysis
B. Accounting and Financial Management
C. Results and Report Generation
D. Model Development and Validation
E. All of the above
F. A, B, and D only
G. A, C, and D only

A

G. A, C, and D only

True. Data collection and analysis, model development and validation, and results and report generation, in that order, are all steps in the cost estimating process.

Accounting and financial management is a related discipline.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

True or False. Cost estimating can be applied with cookbook precision

A

False.

Amongst the limitations of cost estimating are that it cannot be applied with cookbook
precision; produce results that are better than input data; predict political impacts;
substitute for sound judgment, management, or control; or make final decisions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

High quality cost estimates have which of the following characteristics?

A. Traceability
B. Credibility
C. Replicability and Auditability
D. Accuracy
E. Comprehensiveness
F. All of the above
G. None of the above

A

F. All of the above

All the qualities listed, as well as timeliness, are characteristics of high quality estimates.
All the others contribute to the credibility of the estimate, which is the single most important quality of a good cost estimate, without which the benefits will be much harder to realize.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

True or False. Cost analysis is required whenever resources are allocated.

A

True.

Since resources are not limitless, we must estimate and plan for the consumption of finite resources, such as labor, capital, and equipment, available to the program or job.
Cost estimating is required to manage changing requirements efficiently under tight constraints and to quantify the resource impact of alternatives to support management decisions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

A Life Cycle Cost Estimate (LCCE)…

A. Is often confused with an operations and support estimate

B. Compares the costs and benefits associated with each alternative course of action

C. Is a “cradle to grave” estimate

D. A and C only

E. B and C only

F. A and B only

G. None of the above

A

D. A and C only

A Life Cycle Cost Estimate (LCCE) is a “cradle to grave” estimate encompassing
Research and Development (R&D), Production, Operations and Support (O&S), and Disposal.

Because the emphasis of an LCCE is to include the oft-neglected O&S portion, it can be misconstrued as an O&S estimate. Choice B describes an Economic Analysis (EA).

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

True or False.

A well-documented estimate withstands intense scrutiny and increases the credibility of the estimate

A

True.

Robust and detailed documentation permits reviewers to follow estimate logic
from assumptions to results. At a minimum, it should provide reviewers with: the purpose and scope of the estimate; ground rules and assumptions; program background and system
description; schedule; and cost estimating methodology.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

True or False.

Scatter plots provide the analyst with a good idea of the relationships and trends present in the data.

A

True.

Scatter plots develop visual depiction of the relationships in the data.
They provide the analyst with uncanny insight, a wealth of important information, and give a good idea of the relationships and trends present in the data.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Which of the following is not a benefit of cost estimating and analysis?

A. Can help choose between design alternatives

B. Can capture cost impacts of new technology or processes

C. Can transform a poor data set into a high quality estimate

D. Can establish and defend budgets

E. Can quantify impacts of program risks

A

C. Can transform a poor data set into a high quality estimate

Cost estimating, when done well, can provide all the benefits noted, but it cannot produce results that are better than the input data. Thus it suffers from the same “garbage in, garbage out”
phenomen that many processes do.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Which of the following are cost estimating techniques or methodologies?

A. Engineering build-up
B. Analogy
C. Extrapolation from actuals
D. Expert opinion
E. Parametric
F. All of the above
G. None of the above

A

F. All of the above

The primary cost estimating techniques are analogy, parametric, and engineering build-up, and extrapolation from actuals, encompassing averages, learning curve projections, and earned value management (EVM) estimates at complete (EACs), is also an important technique.
Keep in mind that expert opinion is not in and of itself a valid technique, but judgment and expertise are applied when using the other techniques listed above.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

True or False.

Cross-checking an estimate with historical data for similar programs or other estimating methodologies or models does not assist in validating the estimate.

A

False

Estimates should always be based on historical data, but cross-checking them
against standard factors, rules of thumb, and the like increases the validity of and confidence in the estimate.

Remember that no estimate is ever right; the best we can hope for is to be in the
right ballpark – that’s why it’s called an estimate!

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Which of the following would not be negatively impacted by an overly short turn-around time requirement for an estimate?

A. Ability to obtain needed data

B. Ability to coordinate with other organizations

C. Ability to perform regression analysis

D. Quality and detail of the estimate

A

C. Ability to perform regression analysis

Limited time to perform an estimate inevitably impacts the quality and detail of the estimate.

There may be inadequate time to coordinate with other organizations and obtain needed data.

If applicable data were available, in a continuously-maintained database, say, then the actual regression analysis could be performed fairly quickly using any of various tools available, including Microsoft Excel.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

True or False. Cost estimates can be used to evaluate proposals for cost reasonableness.

A

True

For a government agency, accurate estimates assess the reasonableness of contractor proposals and program budgets and help the program office effectively defend budgets to oversight organizations
such as OSD, OMB, and the ODNI, and even to Congress.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

What is the biggest concern that would prompt an agency to do an ICE?

A. What is the cost and requirement impact of different solutions?

B. Is the project affordable?

C. Is the estimate the same when unbiased?

D. Is money available when needed?

E. What solutions would meet mission requirements?

A

C. Is the estimate the same when unbiased?

While project affordability is paramount, the best way to ensure this is to conduct an
independent cost estimate (ICE), free from the bias of the program office. Program risks especially can be mistreated if the the original estimating party is not detached from advocacy. Headquarters may
want to do an objective estimate to make sure all program risks are captured and accurately assessed.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Cost estimating is the process of

A

collecting and analyzing historical data and applying quantitative models, techniques, tools, and databases to predict the future dollar amount of an item, product, program or task to facilitate the successful completion of a program, project, or process.

Often, the cost estimator will need to adjust for new materials, new technology, a new software programming language, or new resource allocations. Still, the estimate must never become divorced from historical experience or it risks losing credibility.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

Cost estimating:

A

translates system/functional requirements associated with programs, projects, proposals, or processes into budget requirements,

determines and communicates a realistic view of the likely cost outcome of a system or program, and

forms the basis of the plan to develop, field, and support the system or program.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

For commercial entities (i.e., contractors), this plan usually begins in the proposal stages where the cost estimate serves

A

as an important basis for a new business proposal which includes the management bid/no-bid decision.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
Q

For government entities, an Independent Cost Estimate (ICE) or the program cost estimate will often serve

A

as the basis for a program’s budget.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
25
The three primary reasons for cost estimating are:
1. Budgeting: ICEs provide the basis for improving program budgets to give programs a chance for success and ensure effective utilization of valuable agency resources. If a program runs over budget, other programs' resources will be diverted to cover the additional costs, which can cause a detrimental ripple effect through other programs. Some programs that run far over budget will be terminated, leaving a gap in capabilities. Cost estimating's success in supporting the budgeting process should be measured by a track record showing how long each ICE-based budget proves adequate on a year-by-year and cumulative basis. The track record should determine what fiscal years relative to the ICE the program first ran a deficit and first incurred a debt (i.e., net cumulative shortfall). 2. Planning: After establishing the budget, planning lays the foundation for successful project execution. Estimating supports the development of Basis Of Estimate (BOE) documentation which establishes the integrated baseline. Earned Value Management (EVM) tracks cost and performance against this baseline as measured by performance indices. More information on Contracts and EVM is available in Modules 14 and 15, respectively. 3. Trade-offs: Trade-off studies use a baseline cost estimate to explore the relative cost of changing requirements. Trade-offs aim to achieve the proper balance of maximizing utility and minimizing cost. Often the cost impact of trades is cursory. Overall progress toward cost goals must be measured by periodic Life Cycle Cost Estimates (LCCEs), preferably based on a closed engineering design
26
Risk assessment plays a big part in budgeting, planning, and trade-offs. Without assessing risks, programs will likely develop
unrealistically low budgets that result in immediate shortfalls, establish overly aggressive baselines that result in immediate performance deficiencies, and produce unrealistically optimistic trade-offs that result in faulty decisions. These mistakes will lead program managers to pursue unaffordable or impractical options.
27
Applications As part of a total systems analysis, cost estimating helps decision makers:
make decisions on program viability, structure, and resource requirements; establish and defend budgets; conduct Analysis of Alternatives (AoA); create new business proposals and perform source selection; conduct in-process reviews of major projects; perform design trade-offs; assess technology changes; comply with public law; and satisfy oversight requirements.
28
Cost estimating contributes to a sound management approach and should be done whether it is required or not.
For government projects, it may be a requirement of public law, and both the public and private sectors have oversight organizations whose requirements must be met.
29
Cost estimating helps the project team
make better decisions and, with the appropriate rigor, helps to defend those decisions to others outside the program.
30
Special purpose organizations exist within the government and industry to perform ICEs and validate or adjust the estimates of other organizations. These are known as cost oversight organizations. There are other organizations in government that oversee
for fiscal responsibility at a broader level including the Office of Management and Budget (OMB), the Government Accountability Office (GAO), and the United States Congress.
31
Government program offices and industry project teams use cost estimates to
justify obtaining resources and to help allocate resources internally.
32
Budgeting, contracting, and purchasing organizations use cost analysis to make sure
the costs/prices of goods and services budgeted, contracted, and purchased are reasonable, consistent, and well-negotiated.
33
manufacturers need to know their operating costs in order to manage
and reduce them
34
Cost estimating and analysis require an interdisciplinary mix of many different skills and domains of expertise. Most importantly, they require the use of mathematics, from basic arithmetic to algebraic functions to rather sophisticated probability and statistics.
Cost estimators require an understanding of economics, business, and accounting because cost estimating involves the application of economic ideas like inflation and obtains much of its data from cost accounting systems. Cost estimates are also used to justify the allocation of resources, which means estimators must understand the budgeting process Since cost estimating frequently seeks to optimize all the aspects of system development, estimators should have a familiarity with ideas from Operations Research (OR), the management sciences, and Industrial Engineering (IE). IE can also be applied to detailed production analysis in a manufacturing environment Furthermore, familiarity with the product system, software, process, etc., helps to identify cost drivers and understand the data needed to develop cost estimates Functional expertise, mechanical engineering, systems engineering, physics, software development, and logistics experience aid in the analysis and help analysts communicate with the engineers working on the project. A little bit of salesmanship helps defend the estimate on the merits of its methodology and documentation and ensure success with program leadership and external organizations. Finally, marketing insight helps develop price proposals that go after new work.
35
Overview of Cost Estimating and Analysis This cost estimating overview discusses:
the context of when to estimate, the benefits of cost estimating and analysis, the qualities or characteristics of a good cost estimate, limitations of cost estimating and analysis, and concerns and issues that arise during the cost estimating process.
36
Overview of Cost Estimating and Analysis Context Cost estimating is a critical element in the acquisition process which revolves around the cost of an item or job and the availability of resources and funding. A reasonable, supportable, and responsive budget supports the efficient and timely execution of a program or task, and the budget is founded on competent estimates.
Analysts must estimate and plan for the consumption of finite resources like labor, capital, and equipment allocated to the program or job. Realistic estimates of the projected cost of an activity determine adequate resource allocation, which leads to a greater probability of a successful project
37
Overview of Cost Estimating and Analysis Context Budget estimate justifications are needed
for white papers, analysis of budget cut impacts, responses to budget reductions, etc. Similar, though perhaps not as detailed, requirements exist to support the budgeting process within commercial organizations.
37
Overview of Cost Estimating and Analysis Context Within the government, major programs require a validated cost estimate to continue through milestone reviews and acquisition reviews. DoD Instruction 5000.2 "Operation of the Defense Acquisition System" and DoD Instruction 5000.4-M "Cost Analysis Guidance and Procedures", provided in References, mandate that programs develop both a POE and a DoD Component Cost Analysis (CCA) to support acquisition milestone reviews
This mandate is similar to the scrutiny required to continue through a stage-gate development process in industry. As a program matures technically and schedules change, cost estimates should be periodically updated to account for these changes. Updating the estimate provides a level of management insight into the current program status that helps maintain effective control of the program and balance resources and the budget. As programs progress, the requirements for the program may change due to external pressures from customers and resource sponsors or due to internal systems engineering decisions. Cost estimating must manage changing requirements efficiently under tight constraints and quantify the resource impact of alternatives to support management decision making.
38
Overview of Cost Estimating and Analysis Context Different phases in the system life cycle require different types of estimates.
During the pre-systems acquisition phase, AoAs or trade studies are useful. AoAs are studies comparing technical, cost, and performance characteristics of multiple approaches and are used to select among alternatives before committing to a particular project. During systems acquisition, a source selection process is performed to compare competitors’ approaches to satisfying requirements. After system development, a cost study can then be performed before full-scale production. Sometimes multiple ICEs will be performed for budgeting purposes or program cost tracking.
39
Overview of Cost Estimating and Analysis Benefits and Qualities Ultimately, the cost estimate translates into a budget input. Cost estimating helps formulate budgets by
integrating the requirements and budgeting processes. Furthermore, most estimating documentation usually translates easily to budget documentation. For a government agency, accurate estimates assess the reasonableness of contractor proposals and program budgets and help the program office effectively defend budgets to oversight organizations such as OSD, OMB, and Congress. They also assist in quickly and accurately determining the impacts of budget cuts on program baselines and the program/system functionality
40
Overview of Cost Estimating and Analysis Benefits and Qualities Manufacturers require cost estimates to accomplish a task before it has started, ensuring maximum productivity and profitability.
An organization’s success in completing projects within a reasonable budget directly links to future business opportunities. Accurate estimates of the cost of a system or program provide the framework for selling the system or program at a reasonable profit and avoid two potential situations: buying in and pricing out of the market. Buying in refers to advertising a cost significantly under the actual amount of effort for the purpose of winning a piece of work as the lowest bidder. As the system is developed, it becomes apparent the estimate was too low and to recover, either the customer or manufacturer is faced with paying for the overrun, which can lead to lost profitability or bankruptcy. Pricing out of the market refers to estimating too high, meaning the manufacturer can no longer remain competitive. Accurate estimating prevents the occurrence of these situations, ensuring a solid reputation which enhances future business potential.
41
Overview of Cost Estimating and Analysis Benefits and Qualities Cost estimating uses established, repeatable methods that allow estimates to be clearly traced, replicated, and updated.
These methods contribute to the credibility of an estimate. Accurate cost estimates help management avoid cost growth and schedule slips. Depending on the detail of the cost estimate, cost drivers can be easily identified for cost improvement initiatives. In program management, cost estimating facilitates managing schedule and technical risks by objectively quantifying the impact of these risks (see Module 9 Cost and Schedule Risk Analysis).
42
Overview of Cost Estimating and Analysis Benefits and Qualities Cost estimates form the basis for evaluating competing systems or initiatives via a
cost/benefit analysis or AoA. Comparative studies evaluate candidate solutions to a problem from a technical, performance, and cost standpoint before the decision maker has committed to a particular solution. These types of studies are discussed in Module 13 Economic Analysis.
43
Overview of Cost Estimating and Analysis Benefits and Qualities Cost estimating should capture the cost impacts of both design decisions and business decisions.
In the former case, trade studies are conducted throughout the development of a system when multiple design solutions exist and cost is often a critical selection criterion. This approach is formalized in disciplines such as Cost As an Independent Variable (CAIV), Design To Cost (DTC), and target costing. Estimates can also evaluate the impact of new ways of doing business (e.g., in-sourcing vs. outsourcing, Commercial Off-The-Shelf (COTS) vs. custom software
44
Overview of Cost Estimating and Analysis Benefits and Qualities In this module, a commercial airplane example demonstrates the various considerations of an acquisition program. There are several key benefits of doing cost estimates at various phases of the acquisition process
Before a solution is decided upon, alternatives should be explored. For any solution, risks and potential problems should be identified so a realistic cost can be used for decisions. The result will be the best value that balances performance and cost. The estimate should yield a budget for planning out program expenditures by year, primarily for development. The outcome of all these benefits is a competitive advantage for industry or a successful, affordable mission for government agencies.
45
Overview of Cost Estimating and Analysis Benefits and Qualities The characteristics of high-quality cost estimates are
Accuracy: Estimates are unbiased, neither unrealistically low nor overly conservative, but based on an assessment of most likely costs. Underlying data have been normalized for technical baseline and for inflation using appropriate guidance. Cost Estimating Relationships (CERs) result from regression analyses with good curve fits and minimal error bands, making them valid predictors of cost. Time phasing of the estimate is logical and reliable. Comprehensiveness: The estimate uses a Work Breakdown Structure (WBS) at an appropriate level of detail to ensure those cost elements neither omit nor double-count elements. All cost-driving Ground Rules and Assumptions (GR&A) are detailed in the documentation of the cost estimate. Replicability: The estimate presents a WBS fully traceable to the system specification. The estimate thoroughly documents source data, significance, and goodness of fit statistics for CERs, detailed calculations and results, and explanations for why a particular method or reference was chosen. Auditability: A reviewer can follow the estimate process, repeat the calculations, and arrive at the same answer. Credibility: The estimate must be accurate, replicable, and auditable in order to be credible. Credibility is the single most important quality of a good cost estimate. Timeliness: Even a high-quality estimate cannot provide value if not delivered to decision makers on time.
46
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Analysts develop cost estimating methodologies with an imperfect understanding of the technical merits and limitations of the system.
Applying historical data in an estimate is always subject to interpretation. Because of future uncertainties, there are limitations in determining the degree to which reality varies from the plan.
47
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Realistically, the cost analysis process should:
conform to the situation/program; produce results of the same caliber as the input data; disregard political and non-cost impacts; utilize sound judgment, management, or control; and support but not dictate the final decisions.
48
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Decision Support Cost estimates support critical decision making, but analysts may find it challenging to quantify the cost impacts of various alternatives under consideration due to
difficulty in establishing CERs that are linked to the right design parameters, new technologies, or non-traditional approaches. Also, cost analysis cannot inherently exchange variables that cannot be directly related to cost. Such trade-offs require decision analysis techniques from operations research
49
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Quality The quality of an estimate relies on good data (input) and good documentation (output). Defining and obtaining good data presents one of the most notoriously difficult problems in cost analysis.
Analysts may only have access to secondary data which have been adjusted one or more times from the original raw data. Inadequately documented estimates lose credibility and compromise preparedness for future estimates. Without good documentation, crucial information, data, experience, and estimating approaches may be lost along with the tacit knowledge of analysts who could retire or leave the organization.
50
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Coordination Because system costs frequently cut across organizational boundaries
the data collection and estimate validation processes may involve a lot of coordination requiring considerable time and resources.
51
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Consistency A cost estimate must be internally consistent throughout a program’s lifespan and track well to previous estimates
An estimate should use consistent content and assumptions along with estimates of the program developed by other organizations (if these estimates are to be compared). Because two estimates may not directly compare in practice, analysts may need to develop techniques to map costs between different structures or to adjust costs based on differing assumptions. Inconsistent historical data between or within sources further creates barriers to consistency
52
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Resource and Schedule Constraints Analysts must deliver cost estimates to decision makers in a timely manner or the estimates will be overcome by events.
Analysts usually resource constrain estimates, but this merits careful planning up front. The schedule must consider the availability and quality of data required for developing the estimate so that the quality of the estimate will not suffer. If an estimate is of lower quality, its utility to key decision makers will be limited. Time constraints may also impact quality, coordination, and consistency.
53
Overview of Cost Estimating and Analysis Challenges to Cost Estimating Security/access There may be reasons to restrict access to both the data used to create the cost estimate and the completed estimate product
Data may be considered proprietary or classified. Precautions to prevent misuse of data may slow the cost estimating process.
54
Cost Estimating Products The cost estimating process involves four basic cost products:
the technical and schedule baseline description, which gives us inputs to the cost estimating process; the cost estimate itself, in all its forms; the cost model used for the estimate; and the accompanying estimate documentation.
55
Cost Estimating Products The cost estimating process involves four basic cost products: Technical and Schedule Baseline Description The technical and schedule baseline description provides a detailed technical and schedule description of a program/system to be estimated. The details facilitate identification of any area or issue that could have a significant cost impact and which therefore must be addressed by the cost estimator. It also accommodates the use of various estimating methodologies.
The technical and schedule baseline description collects, integrates, and coordinates all the technical, programmatic, and schedule information necessary to estimate the costs of a program. The engineers and technical team develop the various components of the technical and schedule baseline description, and the cost analyst uses the technical and schedule baseline description to ensure that cost projections are based on a common definition of the system and the acquisition program. With the basis of the estimates explicitly documented, analysts can easily update the estimate and provide a verifiable trace to a new cost baseline when key assumptions change during the course of the project lifetime. The credibility of the cost estimate suffers if the analyst cannot clearly explain the technical, programmatic, and schedule assumptions underlying each of the cost estimates.
56
Cost Estimating Products The cost estimating process involves four basic cost products: Technical and Schedule Baseline Description A Cost Analysis Requirements Description (CARD) is a specific case of a technical and schedule baseline description required by major DoD programs (see Resources for DoD Policy Guidance).
CARD-like document should always be developed for an estimate for the reasons given above, regardless of whether it is for a government estimate or not. Other government agencies have similar CARD-like documents such as the Office of the Director of National Intelligence's (ODNI) Intelligence Capability Baseline Description (ICBD), National Aeronautics and Space Administration's (NASA) Cost Analysis Data Requirements (CADRe) and Department of Homeland Security’s Cost Estimating Baseline Document (CEBD). A well-done technical and schedule baseline description can bring benefits not only to cost estimating but to the entire program.
57
Cost Estimating Products The cost estimating process involves four basic cost products: Technical and Schedule Baseline Description The technical and schedule baseline description features programmatic information such as program plan and purpose as well as mandates or directives governing the program. A DoD CARD also includes[3]
System technical description: This includes key performance characteristics, replaced system(s), the entity responsible for the system, the system user, descriptions of hardware and software components including interactions, technical protocols or standards used, system architecture and equipment configurations, Key Performance Parameters (KPPs), operational profile, reliability analysis, security requirements, and test and evaluation concepts/plans. WBS: The determination of how sufficient the WBS is for cost estimating purposes and whether the WBS accurately reflects the system baseline should be included with the WBS description. Description of predecessor systems: Provide the reasons for replacing the legacy system, a detailed description of the legacy system hardware and software components, all technical protocols or standards used, KPPs, a maintenance/logistics description, a training description, and a legacy phase-out plan. Manpower requirements: Include all requirements associated with the system, how these compare to any legacy system, applicable grade or salary levels, skill level requirements, and location of personnel. Technical/performance, schedule, and cost risk: The program manager should include an assessment of risk. Operational Concept: This includes program management details, how and where the system will be operated, and platforms it will be installed on. Deployment details: Provide information regarding the platform/site deployment schedule, special preparations required, user preparations/training, the roll-out plan or Initial Operating Capability (IOC) schedule, the standard platform/site configurations, and the transition plan between the legacy and current system. Logistics support details: Provide the maintenance and sparing plan and handling of upgrades. Training plan: This includes types of user training or certification, types of training or certification needed for system maintenance or administration personnel, training provider and location, seat requirements, and frequency of training. Contractor development and production costs. Acquisition schedule and milestones. Acquisition Strategy: Include stated objectives of the program, any secondary program goals, details of vendor support, and contracting details. System Test and Evaluation Plan: Provide the number of required tests, test schedule, testing entry/exit criteria, test bed locations and configuration, contingency plans. Environmental impacts: This should include a mitigation plan and disposal concept. Changes from the prior CARD: Ensure the versions are defined and summarized including change pages with track changes and margin bars so the reader can easily understand the differences without having to compare page by page.
58
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate The cost estimate uses established methodologies to incorporate the analysis of individual cost elements in order to develop project estimated costs of a program or system from data and experience together with the corresponding documentation
There are various types of cost estimates; each should have an associated cost model and documentation. The quality of the documentation influences the quality of the estimate. Different types of cost estimates support different types of decisions. For example, a program in very early development would not need a 5-year budget look-ahead. Instead, decision makers need to evaluate the trade space to decide what technologies to use. Alternately, a nearly-complete program would have little use for a ballpark cost figure when only future costs are necessary. Estimates should become more detailed and accurate as the program progresses. A good understanding of the types of cost estimates available ensures the estimate remains relevant to the decision required. When developing a cost estimate, the analyst should consider past and current data relative to the system being estimated. The data and methods available dictate the approach to developing the estimate.
59
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates A Life Cycle Cost Estimate (LCCE) is the prototypical cost estimate that includes Research and Development (R&D), production, Operations and Support (O&S), and disposal costs (commercial definitions are similar).
A LCCE is often confused with O&S cost estimates, but the LCCE actually covers all costs associated with the program for its complete duration. Analysts compile LCCEs as early in the life cycle as possible, even in pre-acquisition and proposal stages, since the opportunity to minimize the cost of ownership diminishes rapidly as the design and development of a system proceeds. Milestone decisions generally require LCCEs as supporting documentation. The estimate covers costs regardless of funding source, spanning all relevant organizations and appropriation types and is also known as a Total Ownership Cost (TOC) estimate, which is also discussed further in Module 16 Cost Management.
60
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates An Independent Cost Estimate is a LCCE developed by an analyst outside the program or project office that tests the reasonableness of the program or project estimate. The ICE uses the same information employed by the program or project estimate as a technical and schedule baseline description or similar baseline documentation
However, the ICE employs an estimating methodology different from the one used to develop the program or project estimate. It can be characterized as an objective assessment of what the program's most likely cost will be and provides an unbiased test of the reasonableness of the LCCE. The ICE provides an independent source to critically examine the LCCE and identify potential budgetary excesses and shortfalls. The ICE also identifies the cost risks associated with the project so that they may be addressed and provides supporting documentation at critical milestone reviews. Independence in cost estimating provides a critically important, unbiased look at a program
61
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates A Budget Estimate refers to estimates developed strictly to obtain authorization for funding.
In theory, a budget quality estimate has a wider margin for error than other types of estimates. The fundamental purposes of budget estimates are: to ensure project feasibility and attainable performance levels; to develop a reliable project cost estimate consistent with realistic schedules; and to establish baseline project definitions, schedules, and costs. A budget estimate ideally includes a risk assessment; not all organizations require it, though it is a GAO best practice.
61
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates A Rough Order of Magnitude (ROM) estimate is used when very little specific information is known about the project.
The design effort for the program or system has not begun, and the estimate is often based on ratios or factored historical information. A ROM is primarily used for feasibility studies and selection from among alternatives
62
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates An Estimate At Completion (EAC) is used for in-process programs and for ICEs when there is sufficient cost history.
It typically relies on the techniques of EVM, which is the subject of Module 15. EACs have become more important in recent years because of the need to certify programs to satisfy the requirements of Nunn-McCurdy Act of 2010 and Weapon Systems Acquisition Reform Act (WSRA) of 2009. Links to these acts can be found in References.
63
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates An Independent Cost Assessment (ICA) is an independent estimate with risk incorporated that started with an external estimate
(usually from the contractor or program office) and incorporates adjustments based on a more objective view of the program. These assessments may fall anywhere in the spectrum from a full-blown ICE to a cursory sufficiency review
64
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates An Economic Analysis (EA) describes a specific mission requirement and lists specific alternative courses of action that will satisfy the requirement.
An EA is a framework for systematically investigating problems of choice. The EA examines and compares the costs and benefits associated with each alternative course of action. It is also known as cost-benefit analysis or Business Case Analysis (BCA). Module 13 Economic Analysis is devoted to this topic.
65
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates The Analysis of Alternatives (AoA) evaluates the costs and benefits (i.e., operational effectiveness or utility) of different courses of action to meet recognized needs.
AoAs aid and document decision-making by showing the advantages and disadvantages of the alternatives being considered. Whereas an EA is more of a BCA where benefits can be quantified monetarily, an AoA usually involves performance characteristics as well. These types of analysis are discussed in Module 13 Economic Analysis.
66
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Estimate Types of Estimates In the case of the commercial aircraft example:
A LCCE would determine a full program cost including all development, production, and maintenance of the craft. For example, automation of tasks can have a high development effort but significantly reduce the cost to build and maintain the system. Therefore, a good understanding of life cycle costs informs decision making. It includes aircraft R&D, acquisition, maintenance, spare parts, and disposal. An ICE would assess a program office estimate for bias. Program risks may understate costs if the original estimating party is not detached from advocacy. Headquarters may want to do an objective estimate to make sure all program risks are captured. An ICA would evaluate the wing manufacturer’s estimate for bias and feasibility. Reviewing the prime contractor's estimate for reasonableness and impartiality will determine if a sole-source contract is the best option. A budget estimate would ensure the affordability of the portfolio. If another project is draining resources, the cost and schedule of this project could be impacted. Earlier stages of a program are sometimes given a certain amount of money to develop a technology or proof of concept. As the program moves towards Authority To Proceed (ATP), more emphasis is placed on generating a firm budget for future development and production. ROM estimates would determine whether the cost of this next-generation plane is in the right ballpark. It can only really be compared to other ROM estimates, since a quick-look estimate is not as accurate as a full-scale estimate. An AoA would help down-select from early trades with cost and requirement impacts of solutions. The analyst might explore various designs: blimp, jet, and propeller. For short flights, the propeller could probably meet mission requirements at a much lower cost, while the jet is only effective at long ranges. Blimps could even be in the trade space, but the amount of money necessary to invest might make them stack up poorly against the other options. An EAC would be useful when the jet design is already being built. A good assessment of spend rates and progress could identify serious problems. The jet has high-risk engines, and the best way to evaluate that impact is a good unbiased look at the program. An EA would provide a business analysis of different solutions to the problem. Is there a market for building a high-altitude near-supersonic plane? What should the passenger capacity be?
67
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Model A cost model is the collection of cost estimating methodologies associated with a cost element or work breakdown structure that best reflects the cost of a future system given three important considerations:
the level of detail of technical/programmatic definition of the future system, the level of detail of relevant existing cost estimating methodologies, and the level of detail of historical data available to develop new cost estimating methodologies.
68
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Model Inputs Rates for labor, materials, tooling, scrap, etc., need to be provided
Audited or negotiated direct and indirect rates are usually applied. A variety of factors affect rates, from union demands to manufacturing shifts from one plant to another. Analysts should collaborate with their rate development team.
69
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Model Model Structure Depending on the system's maturity and the scope, time, and resources given for the estimate, the analyst may develop a model structure at varying levels of detail
Analysts may build custom models, use commercially available software, or both. The model should apply some combination of accepted estimating techniques such as parametric, analogy, and engineering build-up
70
Cost Estimating Products The cost estimating process involves four basic cost products: Cost Model Outputs Depending on the time period covered by the estimate and the requirements and assumptions of the estimate, it may be both time-phased
Indicating which costs are to be incurred in which years, and escalated, representing either the year that dollars are spent, a base year, or a budget year. The estimate should include risk analysis to provide a range of possible outcomes to assist management in decision-making.
71
Cost Estimating Products The cost estimating process involves four basic cost products: Estimate Documentation The purpose of the cost estimate states the reason for developing the estimate. This includes whether it is an initial or updated estimate, the specific tasking and relevant correspondence, and the cost estimating team composition.
The scope of the cost estimate identifies phases (e.g., development, production, operations and support), specific time periods covered by the estimate, and systems and processes encompassed by the estimate. The scope also indicates specific areas not addressed by the estimate and reasons why. The ground rules and assumptions list all technical and programmatic conditions that formed the basis for the estimate to assist management in understanding these conditions. When conditions are dictated to the estimating team, they become the ground rules by which the estimate will be conducted. In the absence of a dictated ground rule, the estimator establishes assumptions which allow the estimate to proceed. In exercising this privilege, the estimator must avoid arbitrary assumptions by utilizing expert judgments rendered by experienced program and technical personnel. The program background and system description characterizes significant program and system aspects and status in terms of work accomplished to date, current position, work remaining, detailed technical and programmatic descriptions, major components, performance parameters, support concepts, contract types, acquisition strategy, and any other information that will assist the reader in fully understanding the system. The program schedule includes the master schedule for development, production and deployment, and a detailed delivery schedule. Finally, the cost estimating methodology describes the estimation methods. Reviewers use the cost estimating methodology to follow the logic from assumptions to the conclusion and to replicate or update the estimate at a later time. Specifically, BOE documentation for proposals must support government reviews and audits.
72
Cost Estimating Products The cost estimating process involves four basic cost products: Estimate Documentation The estimate documentation details the primary data, techniques, and rationale for each element of the cost estimate. Analysts should document:
any data used, its source, and normalization procedures (see Module 4 Data Collection and Normalization); the basis, development, and/or source of all CERs and factors used, including a description of how they were applied, their relevance to the program being estimated, and any regression statistics (see Module 3 Parametric Estimating); how the labor hours were developed, and direct and indirect labor rates as industry averages or organization-specific; the methodology used to develop cost improvement curves (if applicable) and descriptions of the curves in terms of slope, source, and relevance to the cost elements and the program being estimated including unique aspects of curve analysis (see Module 7 Learning Curve Analysis); all hardware and materials required and the basis of their costs (e.g., vendor quotes, catalog prices, etc.); all pre-existing cost models used and their relevance to the estimate, along with complete details regarding any parametric inputs and outputs and any calibration performed; the specific inflation indices and computations used as well as the approach used to time-phase the estimate (see Module 5 Inflation and Index Numbers); the logic and rationale that led to specific conclusions (i.e., expert judgments); the risk analysis conducted and details on how confidence was established (see Module 9 Cost and Schedule Risk Analysis); a conclusion expressing the cost estimating team’s determination regarding the reasonableness of the program office estimate; appendices containing any other information pertinent to the cost estimate; and a reference list.
73
Cost Estimating Process and Methods This section will address the seven basic steps of a more general cost estimating process:
Develop a Work Breakdown Structure (WBS) Develop a program/system baseline Collect data Analyze data Develop cost estimating methodology for each cost element/work package Build and validate the cost model Generate results and other required output, such as reports Specific process steps vary between different organizations and may also include risk and uncertainty analysis, but general guidelines apply. The estimate should include a clear baseline and a framework for how to estimate costs. Data needs to be gathered, analyzed, and methods derived. Finally, the analyst should generate results and reports for the decision-maker.
74
Cost Estimating Process and Methods Work Breakdown Structure Development A WBS establishes a common frame of reference for relating job tasks to each other and for relating project costs at the summary level of detail. It provides a consistent and visible framework for specifying the objectives, labor, materials, and contracts of the program/system. A standardized WBS satisfies mandatory cost reporting requirements for future estimates, described further in Module 4 Data Collection and Normalization. The key WBS functions are to
display and define the program/system being developed or produced using a product-oriented family tree, describe program components at the level necessary to capture all elements of cost, and relate the elements of work to each other and the end product.
75
Cost Estimating Process and Methods Work Breakdown Structure Development it is important that the WBS encompass the system-level activities such as
system engineering, specific builds, integration, fielding, and support of the system throughout its life cycle until it is removed from the inventory. The technical and schedule baseline description or similar documentation provides this program plan. The WBS does not have to be the same across systems or program phases but should be consistent. A WBS can be tailored for each system or program for the purpose of capturing idiosyncrasies. Even when elements of the WBS vary by phase, it is advantageous to establish the master WBS for the life cycle of the program, including each phase WBS, as early on as possible The WBS is sometimes referred to as the Cost Element Structure (CES). While often interchanged, the term WBS typically relates to a program/system from an engineering standpoint while the term CES specifically refers to the cost estimating process. A properly developed CES should encompass all the elements of the WBS
76
Cost Estimating Process and Methods Work Breakdown Structure Development In the context of EVM
a WBS is the result of project/program planning that establishes the physical work packages or elements and the activities within those packages that completely define a project. It organizes the physical work packages into levels that can be rolled up into control accounts. Module 15 Earned Value Management has more about the WBS from the perspective of planning work packages and tracking project cost and schedule performance. The WBS used for EVM, however, may not capture engineering effort performed by government system engineering or system integration personnel. Since the WBS divides the program/system into work packages, it can also interrelate the schedule and costs. It separates elements of the program/system into component parts, making the relationships of the parts clear and the relationship of the tasks to be completed - to each other and to the end product - clear. The work packages or their activities can also be used as schedule activities enabling resource loading of a schedule, resource budgeting against time, and the development of a variety of cost budgets plotted against time.
77
Cost Estimating Process and Methods Work Breakdown Structure Development A WBS also provides a consistent financial reporting framework. The WBS:
assists in tracking the resource allocations, cost estimates, expenditures, the status of efforts, and cost and technical performance; forms the basis for communication throughout the acquisition process and unifies the planning, scheduling, cost estimating, budgeting, contracting, configuration management, and performance reporting disciplines; permits managers to evaluate progress in terms of contract performance; summarizes data for successive levels of management and provides the appropriate information on the actual costs, current status, and projected performance of the elements for which they are responsible; and keeps program status constantly visible so that the program manager, in cooperation with the contractor, can identify and implement changes necessary to assure desired performance.
78
Cost Estimating Process and Methods Work Breakdown Structure Development A WBS should be developed early in the conceptual stages of the program to provide a structure
to the design requirements and functional architecture of the program/system before the physical architecture is able to be defined. It is not until later phases that the system is described by its specifications. The WBS evolves with the program through an iterative analysis of the program objectives, functional design criteria, program scope, technical performance requirements, proposed methods of performance (including acquisition strategy, drawings, process flow charts), and other technical documentation. As development progresses, analysts define the detailed technical objectives and assign specific work tasks to each element, including the resources, materials, and processes required to attain the objectives. The linkage between the specification requirements, the WBS, the statement of work, and the schedule provides specific insights into the relationship between cost, schedule, and performance. This relationship allows all related items to be tracked to the same WBS element. Therefore, the levels of the WBS should map to these requirements and conform to its product-oriented family tree
79
Cost Estimating Process and Methods Work Breakdown Structure Development When developing a WBS, engineering efforts throughout the life cycle aid in defining the description of the system and its related levels.
As the program/system matures, engineering efforts focus on system level performance requirements, specifically validating critical technologies and processes and developing top-level specifications. Through this process, the work breakdown structure better defines the system in terms of its specifications and the configuration items that comprise the system. Once the system concept is determined, then major subsystems and configuration items can be identified and lower level functions defined. Detailed design activities are ongoing until the total system definition is complete. The levels of the WBS are directly linked with the detailed configuration of the system.
80
Cost Estimating Process and Methods Work Breakdown Structure Development During production, the system is produced as defined throughout the previous phases. Production includes the actual fabrication, modification, purchase, or some combination thereof, of hardware/software/firmware. The engineering efforts actively maintain the configuration of the system being produced. The WBS is defined to the level appropriate for contract management and maintenance. When major modifications occur, the same WBS can be used. If the changes are substantial, a new work breakdown structure can be developed. As part of developing a WBS, the program manager will also develop a WBS dictionary. The WBS dictionary:
lists and defines the work breakdown structure elements, shows the hierarchical relationship of the elements, describes each work breakdown structure element and the resources and processes required to produce it, and provides a link to the detailed technical definition documents. The analyst should revise the WBS dictionary often to incorporate changes so that the dictionary always reflects the current status of the program. Document all changes for future reference. See the Resources section for an example of a WBS dictionary.
81
Cost Estimating Process and Methods Program/System Baseline Development The program/system baseline resembles a technical and schedule baseline description or similar document. It includes ground rules and assumptions and provides the basis for the cost estimate
It enables the translation of program requirements into a reliable and defensible program cost estimate. It also describes the system’s important features including required resources (such as hardware and personnel), program quantities, operational concepts, and milestones. The program/system baseline corresponds to the EVM baseline. At the beginning of each cost estimate, estimators determine the programmatic and technical scope. The programmatic scope indicates the window of time covered by the estimate, and whether it includes the entire life cycle or a specific period. The estimator also identifies the phases (R&D, production, O&S, disposal) covered by the estimate. The technical scope includes all requirements for the program including a detailed description of work to be performed, out-of-scope work, deliverables, any constraints or special conditions, sequences of events and any interdependencies, and milestones. The baseline should include key performance characteristics, hardware/software components, architecture, configurations, reliability and security requirements, etc.
82
Cost Estimating Process and Methods Program/System Baseline Development Document the impact on the project baseline of any applicable contractual changes. The baseline definition process includes:
analogous systems' technical details, the reason for replacing legacy hardware/software, and phase-out plan; technical/performance, schedule, and risk as determined by management and/or engineers (they relate to the technological maturity or Technology Readiness Levels (TRLs) discussed in Advanced Topics); manpower requirements including skills, salaries, and location; operational concept including program management details, operation specifics, and platforms; and deployment details including sites, schedules, special preparations, site configurations, and transition plans.
83
Cost Estimating Process and Methods Program/System Baseline Development The components that define the program/system baseline are:
logistics support including maintenance, sparing, upgrade plans, and any ancillary costs; framework for reporting actual costs and software counts, training plan detailing types of training/certification, providers, locations, and frequency; the system test and evaluation plan which includes the test schedule, number of tests, criteria, locations, subsystem tests, and subcontractor test review; environmental impacts such as mitigation plans, disposal concept, environmentally acceptable alternatives, environmental conditions expected to be encountered, hazardous materials that may be encountered or generated during the subsystem’s development, and the quantities of each hazardous material used or generated over the subsystem’s lifetime. The program/system baseline needs to include the information previously discussed as part of the technical and schedule baseline description. In addition to the current WBS, it should include any previously developed WBS.
84
Cost Estimating Process and Methods Data Collection and Data Analysis Data is the critical raw material for cost estimating. Data provides credibility and defensibility to all estimates, without which the estimate would be looked upon as merely a guess or, at best, the analyst's opinion. Data provides information on cost trends over a variety of related programs; insight into cost structures; and the information used to develop CERs, parametric estimates, and models. In addition, properly analyzed data allows for assessments of the statistical accuracy and fidelity of an estimate. These two crucial steps of the cost estimating process are discussed further in Module 4 Data Collection and Normalization and Module 5 Index Numbers/Inflation; and Module 6 Basic Data Analysis Principles and Module 8 Regression Analysis, respectively. Data collection (obtaining and normalizing data) and data analysis (methods and process) go hand in hand. The two steps of the process are somewhat iterative; data analysis may uncover the need for further data collection. There are four steps in the data collection process:
Develop an understanding of the total picture of the estimating task Establish a cost element/work breakdown structure and boundaries around the estimate Understand the estimating techniques to be employed and their associated data collection needs Develop a data collection plan
85
Cost Estimating Process and Methods Data Collection and Data Analysis Before examining the data, the cost analyst must thoroughly understand what needs to be estimated. The analyst must understand the intended use of the estimate and the scope of the estimate, both of which should have been made apparent in the program/system baseline definition. A cost element structure must be developed based on the earlier developed WBS. Existing data in different structures should be mapped to the new structure. The analyst must understand the data requirements for the estimate. To understand data needs, the analyst must have an idea of the estimating technique(s) that will be employed in the estimate. The next step is to identify potential data sources for each component of the estimate. Keep in mind that different estimating techniques may require different data. Next, the cost analyst must develop and execute a data collection plan. The plan should discuss capturing primary or secondary cost, technical, and programmatic data and ensure every cost element is covered, encompassing the full scope of the estimate. Primary and backup data sources should be identified, and attempts should be made to identify data sources for cross-checks. To keep the estimating effort on schedule, a projected data collection timeline should be included with the plan. Finally, collect the data. Selecting and collecting appropriate data for the estimate requires sound analytic judgment. The estimating process benefits from organized and structured data. For data to be useful, the data needs to be consistent and comparable to other data used in the estimate. The process to do this is called normalization. Types of data normalization include:
by cost units; by sizing units; by special groupings such as mission application, by platform operating environment, and by recurring/non-recurring costs; by development state; and of end terms for homogeneity.
86
Cost Estimating Process and Methods Cost Element Methodology The three basic costing techniques or methodologies available when developing a cost estimate are analogy, parametric and build up
The analogy costing technique compares the cost of an item to be estimated to a similar item. A parametric estimate costing technique is a mathematical relationship based on historical data that relates cost to one or more technical, performance, cost, or programmatic parameters. Engineering build-up costing technique involves estimating costs at the lowest definable level, frequently making use of IE techniques such as time standards. These three primary methods can be used at any level desired, as long as data is available at that level. The estimator must determine the most appropriate technique for the task at hand based on the available data and phase of the program/system. Multiple techniques may be employed in an estimate, but each technique has strengths and weaknesses. Applicability of any one technique varies during a program’s life cycle and for different levels of fidelity required for an estimate.
87
Cost Estimating Process and Methods Cost Element Methodology The following are examples of how an analyst might use these techniques:
Analogy Scale a historical program's cost to the current cost estimate by a similar parameter. The Model X100 jet engines have only been used once before, so we cannot easily estimate the cost. However, we can scale based on comparing parameters of the new system versus the analogy. Since the analogy is a similar program (same X-series), we will look at a weight-based scale. The weight of the new engine is double the old engine, therefore we can double the cost Parametric Use past programs to find a relationship of the cost to a parameter (e.g., weight). Then estimate the cost using anticipated parameters. In figure 1.6, we plotted the cost versus variable 1 for several comparable programs. We ran a trend line through the results to use as a prediction tool. Engineering build-up We know an Air Conditioning (AC) system costs on a plane because we received quotes from a Heating, Ventilation and Air Conditioning (HVAC) vendor for all ductwork and an AC blower off the shelf. Add up all sub-system costs at the lowest discernible level. All costs, from nuts and bolts to top-level, must be accounted for. For an HVAC system, we need the costs of ductwork and blower separately. We would also need any other parts necessary: maybe 10,000 rivets to secure ductwork or even all the necessary switches to control operation. Other methods or costing techniques include expert opinion and extrapolation from actuals. Expert opinion utilizes the subjective opinion of Subject Matter Experts (SMEs) to corroborate or adjust cost estimates. On the other end of the spectrum, extrapolation from actuals uses data from prototypes or complete or incomplete units to project the cost of future units. When using different methods for different cost elements, make sure not to double count. A high-level CER may already capture detail in a lower-level build-up.
88
Cost Estimating Process and Methods Model Development and Validation The estimator should determine the best techniques to develop the cost estimates for each of the WBS elements
Remember that when developing the cost model, more than one cost estimating technique may be used. For example, if a system involves a key piece of state-of-the-art equipment which has never been built before, a detailed engineering cost estimate would not be possible since the system description is minimal and historical data does not exist. Therefore, an analogy would be applied in the cost estimates when historical cost data exist for one or more items that are similar to those proposed. Parametric cost estimates can be applied when relationships between cost and system characteristics can be authenticated.
89
Cost Estimating Process and Methods Model Development and Validation After developing the cost model, it must be validated.
Testing the estimate ensures reasonableness and completeness. The analyst should test key cost elements for sensitivity to the cost-estimating techniques used and to key ground rules and assumptions. Testing should include crosschecking the results with historical data from similar programs/systems, applying a different estimating methodology, and/or applying a different cost model (proprietary models versus custom model, e.g., in Excel). The estimate can also be compared to industry benchmarks. These must be based on publicly available data since proprietary data from another company may not be used. Finally, the analyst should conduct a cost risk assessment. Documenting the cost estimate during development lends efficiency to its preparation and quality to its content. It is extremely difficult to recapture the rationale and judgments applied to the cost estimate after the fact.
90
Cost Estimating Process and Methods Results and Report Generation Cost estimate documentation provides written justification for the cost estimate. All steps in the development of a cost estimate must be documented, including definitions, ground rules, and assumptions. In addition to the identification of the methods employed for each cost element, the documentation should address the rationale for that selection. The documentation should be viewed as a substantive and professional effort. A well-documented cost estimate withstands scrutiny. Following rigorous documentation and estimation procedures increases the credibility of the estimate. A general guideline is that the final product should provide sufficient information on how the estimate was developed so that other analysts could reproduce the estimate. When developing the cost estimate results and report, the level of detail required depends on the audience and estimate effort. The report must be:
crisp and complete; easily and quickly comprehended by audiences unfamiliar with the estimate; thorough enough to address the important details of the estimate; and demonstrative of the competence that underlies the estimate results.
91
Cost Estimating Process and Methods Results and Report Generation At this stage, we can conduct other analyses and incorporate them into the final cost estimate, including CAIV, sensitivity, and risk analyses.
CAIV entails setting aggressive yet realistic cost objectives when defining operational requirements and acquiring systems and managing achievement of these objectives. Essentially, a program is made up of three parameters: schedule, performance, and cost adjusted for risks present in all three parameters. In a simplistic view, two of these variables must depend on the third. CAIV makes performance and schedules a function of available resources, namely cost. CAIV will be covered in greater detail in Module 16 Cost Management. Sensitivity analysis assesses the cost of changing significant input parameters. Sensitivity analysis identifies key assumptions and variables within the estimate and determines how changing them effects costs. Its value lies in the additional information and understanding it brings to bear on the decision. For decision makers facing an investment decision, sensitivity analysis helps determine how changes in costs or benefits (e.g., due to forecast errors) affect the estimate. Risk analysis normally increases the cost estimate. The impact of risk is added to the initial point estimate resulting in a range or distribution of costs. Today’s systems possess increasing technical complexity. This increased technical risk increases the risk of schedule delays and cost overruns. Other areas of risk and uncertainty to explore after developing a cost estimate include pending negotiations, schedule risk, uncertain performance requirements, appropriateness of analogies, level of knowledge about support concepts, and critical assumptions. This is discussed in further detail in Module 9 Cost Risk Analysis.
92
After reviewing module 1, you should:
Understand the definition and application of cost estimating Recognize the benefits of cost estimating Understand the challenges of cost estimating Know the differences between various types of cost estimating products Know the process and methods of cost estimat
92
Testable Topics Motivations for Cost Estimating Applications of Cost Estimating Benefits of Cost Estimating Characteristics of High-Quality Cost Estimates Limitations of Cost Estimating Cost Estimating Challenges Types of Cost Estimates Elements of a Cost Estimate Cost Estimating Process Work Breakdown Structure/Cost Element Structure Cost Products (Technical Baseline)
Non Testable. Cost Estimating Organizations Cost Estimating in the Budget Process Cost Estimating the Acquisition Process Cost Estimating Disciplines Personality Traits of Cost Estimators Resources (Regulatory, Agencies, etc.) Advanced Topics
93
WHAT IS COST ESTIMATING AND HOW IS IT APPLIED?
Cost estimating is the process of collecting and analyzing historical data and applying quantitative models, techniques, tools, and databases to predict the future dollar amount of an item, product, program or task to facilitate the successful completion of a program, project, or process Estimates that are divorced from historical experience lose their credibility It begins during the proposal phase (contractors) or development of the ICE (gov’t) Cost estimating helps the project team make better decisions and, with the appropriate rigor, helps to defend those decisions to others outside the program.
94
There are three primary reasons for cost estimating: Budgeting, planning, and assessing tradeoffs
1.Budgeting: ICEs provide the basis for improving program budgets to give programs a chance for success and ensure effective utilization of valuable agency resources. If a program runs over budget, other programs' resources will be diverted to cover the additional costs, which can cause a detrimental ripple effect through other programs. Some programs that run far over budget will be terminated, leaving a gap in capabilities. Cost estimating's success in supporting the budgeting process should be measured by a track record showing how long each ICE-based budget proves adequate on a year-by-year and cumulative basis. The track record should determine what fiscal years relative to the ICE the program first ran a deficit and first incurred a debt (i.e., net cumulative shortfall). 2.Planning: After establishing the budget, planning lays the foundation for successful project execution. Estimating supports the development of Basis Of Estimate (BOE) documentation which establishes the integrated baseline. Earned Value Management (EVM) tracks cost and performance against this baseline as measured by performance indices. More information on Contracts and EVM is available in Modules 14 and 15, respectively. 3.Trade-offs: Trade-off studies use a baseline cost estimate to explore the relative cost of changing requirements. Trade-offs aim to achieve the proper balance of maximizing utility and minimizing cost. Often the cost impact of trades is cursory. Overall progress toward cost goals must be measured by periodic Life Cycle Cost Estimates (LCCEs), preferably based on a closed engineering design. Risk assessment plays a big part in budgeting, planning, and trade-offs. Without assessing risks, programs will likely develop unrealistically low budgets that result in immediate shortfalls, establish overly aggressive baselines that result in immediate performance deficiencies, and produce unrealistically optimistic trade-offs that result in faulty decisions. These mistakes will lead program
95
Cost estimating happens at all levels of government. In general
higher levels of government are less involved in the direct formulation of estimates, and estimates at higher levels are broader in scope.
96
BENEFITS AND CHALLENGES OF COST ESTIMATING
Benefits  Helps formulate budgets  Helps defend budgets to oversight organizations  Maximize productivity and profitability  Accurate estimates deter cost growth and schedule delays Challenges  Limitations in determining how much estimate varies from reality  Some variables do not inherently translate to cost  Quality of estimate relies on quality and availability of data  Lots of coordination  Consistency throughout lifecycle  Resource/Schedule constraints  Access to data
97
THE CHARACTERISTICS OF HIGH-QUALITY ESTIMATES
Accuracy: Estimates are unbiased, neither unrealistically low nor overly conservative, but based on an assessment of most likely costs. Underlying data have been normalized for technical baseline and for inflation using appropriate guidance. Cost Estimating Relationships (CERs) result from regression analyses with good curve fits and minimal error bands, making them valid predictors of cost. Time phasing of the estimate is logical and reliable. *Comprehensiveness: The estimate uses a Work Breakdown Structure (WBS) at an appropriate level of detail to ensure those cost elements neither omit nor double-count elements. All cost-driving Ground Rules and Assumptions (GR&A) are detailed in the documentation of the cost estimate. *Replicability: The estimate presents a WBS fully traceable to the system specification. The estimate thoroughly documents source data, significance, and goodness of fit statistics for CERs, detailed calculations and results, and explanations for why a particular method or reference was chosen. *Auditability: A reviewer can follow the estimate process, repeat the calculations, and arrive at the same answer. *Credibility: The estimate must be accurate, replicable, and auditable in order to be credible. Credibility is the single most important quality of a good cost estimate. *Timeliness: Even a high-quality estimate cannot provide value if not delivered to decision makers on time.
98
COST ESTIMATING PRODUCTS
The technical and schedule baseline description provides a detailed technical and schedule description of a program/system to be estimated. The details facilitate identification of any area or issue that could have a significant cost impact and which therefore must be addressed by the cost estimator. It also accommodates the use of various estimating methodologies. A Cost Analysis Requirements Description (CARD) is a specific case of a technical and schedule baseline description required by major DoD programs The cost estimate uses established methodologies to incorporate the analysis of individual cost elements in order to develop project estimated costs of a program or system from data and experience together with the corresponding documentation. A cost model is the collection of cost estimating methodologies associated with a cost element or work breakdown structure that best reflects the cost of a future system given three important considerations: *the level of detail of technical/programmatic definition of the future system, *the level of detail of relevant existing cost estimating methodologies, and *the level of detail of historical data available to develop new cost estimating methodologies. The estimate documentation details the primary data, techniques, and rationale for each element of the cost estimate
99
TYPES OF ESTIMATES Life Cycle Cost Estimate (LCCE)  Independent Cost Estimate (ICE)  Budget Estimate  Rough Order of Magnitude (ROM)  Estimate at Completion (EAC)  Independent Cost Assessment (ICA)  Economic Analysis (EA)  Analysis of Alternatives (AoA)
A Budget Estimate refers to estimates developed strictly to obtain authorization for funding. In theory, a budget quality estimate has a wider margin for error than other types of estimates. The fundamental purposes of budget estimates are: *to ensure project feasibility and attainable performance levels; *to develop a reliable project cost estimate consistent with realistic schedules; and *to establish baseline project definitions, schedules, and costs. A budget estimate ideally includes a risk assessment; not all organizations require it, though it is a GAO best practice. A Rough Order of Magnitude (ROM) estimate is used when very little specific information is known about the project. The design effort for the program or system has not begun, and the estimate is often based on ratios or factored historical information. A ROM is primarily used for feasibility studies and selection from among alternatives. An Estimate At Completion (EAC) is used for in-process programs and for ICEs when there is sufficient cost history. It typically relies on the techniques of EVM, which is the subject of Module 15. EACs have become more important in recent years because of the need to certify programs to satisfy the requirements of Nunn-McCurdy Act of 2010 and Weapon Systems Acquisition Reform Act (WSRA) of 2009. Links to these acts can be found in References. An Independent Cost Assessment (ICA) is an independent estimate with risk incorporated that started with an external estimate (usually from the contractor or program office) and incorporates adjustments based on a more objective view of the program. These assessments may fall anywhere in the spectrum from a full-blown ICE to a cursory sufficiency review. An Economic Analysis (EA) describes a specific mission requirement and lists specific alternative courses of action that will satisfy the requirement. An EA is a framework for systematically investigating problems of choice. The EA examines and compares the costs and benefits associated with each alternative course of action. It is also known as cost-benefit analysis or Business Case Analysis (BCA). Module 13 Economic Analysis is devoted to this topic. The Analysis of Alternatives (AoA) evaluates the costs and benefits (i.e., operational effectiveness or utility) of different courses of action to meet recognized needs. AoAs aid and document decision- making by showing the advantages and disadvantages of the alternatives being considered. Whereas an EA is more of a BCA where benefits can be quantified monetarily, an AoA usually
100
The cost estimating process involves the following steps.
1. Develop a Work Breakdown Structure (WBS) 2. Develop a program/system baseline 3. Collect Data 4. Analyze Data 5. Develop cost estimating methodology for each cost element/work package 6. Build and validate cost model 7. Generate results and required outputs
101