Dau test and evaluation level 1

Fundamentals Of Test Evaluation

Assignment for Operational Test Director:
You are the Operational Test Director (OTD). From that perspective, briefly (two or three sentences for each question) answer the following three questions:
• Question #1: From your OT perspective what are the 3 most important conclusions you draw from your analysis of the data from the first 10 SRAW firings?
• Question #2: Based' on the IOT&E data so far, what are your top three reasons for placing the SR.AW into deficiency status?
• Question #3: What options might you have considered other than placing the SRAW in deficiency status?
Assignment for Program Manager:
You are the Program Manager (PM). From that perspective, briefly (two or three sentences for each question) answer the following three questions:
• Question #1: From y our perspective as an advocate of the system, what are the 3 most important conclusions you draw from your analysis of the data from the first 10 SR.AW firings? Cite specific firing results to support y our answers.
• Question #2: The OTD believes there have been too many failures already; do you agree? Why or why not? Use specific shots as examples to support your assertion.
• Question #3: Based on extensive DT&E results, you recommended that SR.AW progress to IOT&E. Does the data collected thus far in IOT&E reflect a problem with the SRAW, or with the test plan and conduct? Provide a short justification for your answer using specific test conditions and test data (shot numbers).

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Lesson Overview

The four stages of testing are test planning, test conduct, analysis & evaluation and test reporting . In this lesson you will learn about the third stage, analysis & evaluation.

This lesson discusses and compares the terms "test", "evaluation", and "analysis", and describes some key evaluation principles and strategies.

Specifically, this lesson:

• Defines "analysis" and "evaluation";

• Identifies the principles and strategies associated with analyzing data and evaluating systems;

• Outlines the role played by "quality" requirements ;

• Covers key elementary statistical concepts .

Upon completion of this lesson, you should be able to apply the principles and strategies associated with evaluating systems and analyzing test data.

......_ I Pagelof 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Test versus Evaluation

To understand the analysis and evaluation, stage of the testing you first should understand the difference between "test" and "evaluation".

Tes ting is the process of exercising a component or system in order to obtain performance data.

Evaluation is the process of interpreting and analyzing test data to define system performance results for decision makers.

The data that is processed during evaluation can come from many different sources, including:

• Current tests ;

• Previous tests ;

• Modeling and simulation;

• Analysis ;

• Inspection;

• Demonstration;

• Expert opinion.

Whatever the source of the data, it must be analyzed first before information useful to decision- makers can be generated.

....rl I Page2of 24 ~ Back i ) Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Test versus Evaluation

To understand the analysis and evaluation, stage of the testing you first should understand the difference between "test" and "evaluation".

Tes ting is the process of exercising a component or system in order to obtain performance data.

Evaluation is the process of interpreting and analyzing test data to define system performance results for decision makers.

The data that is processed during evaluation can come from many different sources, including:

• Current tests ; Test ing

• Previous tests ; More formally, " test" is defined in the DAU Glossary as:

• Modeling and simulation; any program or procedure that is designed to obtain, verify, or provide data for the evaluation of any of the

• Analysis ; following: ( 1) progress in accomplishing developmental

• Inspection; objectives; (2) the performance, operational capabili ty and suitability of systems, subsystems, components, and

• Demonstration; equipment items; and (3) the vulnerability and lethali ty of systems, subsystems, components, and equipment items. • Expert opinion.

Whatever the source of the data, it must be analyzed first before information useful to decision-makers can be generated.

....rl I Page2of24 ~ Back i ) Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PR INT I HELP

Knowledge Review

Which of the following is NOT an example of testing?

Bench testing of components

lJ Firing of a cannon

Ship shock trials

./ Comparing results to simulation

Check Answer

Comparing results to simulation is an evaluation func tion all the o ther answers were examples o f testing functions.

...... I P• gelof24 ..... Back fiii Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PR INT I HELP

Analysis

By itself, test data is rarely useful to decision-makers. It must be turned into information. A vital step towards producing useful information from raw test data is analysis. Analysis involves more than just re­ stating the data.

Analysis is the process of using various mathematical or graphical techniques to structure and interpret test data in order to identify patterns, comparisons, or cause/effect.

Analysis involves asking several questions:

• Is the data reasonable?

• Has enough data been gathered?

• Is the qualjtv of the data sufficient?

The use of statistics is a key tool in the overall analysis process.

1'111111111111 I Page 4of24 ~ Back W Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Analysis

By itself, test data is rarely useful to decision-makers. It must be turned into information. A vital step towards producing useful information from raw test data is analysis . Analysis involves more than just re­ stating the data.

Analysis is the process of using various mathematical or graphical techniques to structure and interpret test data in order to identify patterns, comparisons, or cause/ effect.

Analysis involves asking several questions:

• Is the data reasonable?

• Has enough data been gathered?

• Is the quality o f the data sufficient?

The use of statistics is a key tool in the overall analysis process.

I s t he dat a reasonable?

This question reflects the comparison of the pre-test predictions with the actual test results to verify the test was run properly and the data falls within the expected range .

~ I Page4of 24 ~ Back liiii ) Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Analysis

By itself, test data is rarely useful to decision-makers. It must be turned into information. A vital step towards producing useful information from raw test data is analysis . Analysis involves more than just re­ stating the data.

Analysis is the process of using various mathematical or graphical techniques to structure and interpret test data in order to identify patterns, comparisons, or cause/ effect.

Analysis involves asking several questions:

• Is the data reasonable?

• Has enough data been gathered?

• Is the quality o f the data sufficient?

The use of statistics is a key tool in the overall analysis process.

Has enough dat a been gat hered?

This question is also asked during the post-test conduct phase of the T&E process to see if the sample size of good data supports analysis at the desired confidence level.

~ I Page4of 24 ~ Back liiii ) Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Analysis

By itself, test data is rarely useful to decision-makers. It must be turned into information. A vital step towards producing useful information from raw test data is analysis . Analysis involves more than just re­ stating the data.

Analysis is the process of using various mathematical or graphical techniques to structure and interpret test data in order to identify patterns, comparisons, or cause/ effect.

Analysis involves asking several questions:

• Is the data reasonable?

• Has enough data been gathered?

• Is the quality o f the data sufficient?

The use of statistics is a key tool in the overall analysis process.

Is the quality of the da ta s ufficient?

This question is also asked during the post-test conduct phase of the T&E process to verify that the data authentication process certifies the data as acceptable for analysis .

~ I Page4of 24 ~ Back liiii ) Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Review of Some Basic Statistics Concepts

Statistics is the collection and interpretation of quantitative data and the use of mathematics to estimate population parameters. In testing, the analyses of sample data, using statistical techniques, looks for what happened, how well the test article performed vs. some criteria, and makes predictions about future performance. The use of statistics for test design and analysis requires extensive training , beyond the scope of this course . However, testers should nevertheless understand some key concepts and terms, which have been simplified and summarized below and in the following screens.

Basic statistics Concepts and Terms:

Descriptiv e statistics : describes data in formats such as bar charts, graphs, histograms, curves, and frequency distributions. Examples are means, modes, median, range, and standard deviation.

Inferential statistics : draws conclusions from sample data to estimate performance of the entire population.

....rl I Page S of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Review of Some Basic St atistics Concepts

Sta tis tics is the collec tion and interpretation of quantitative data and the use of mathematics to estimate population parameters. In testing, the analyses of sample data, using statistical techniques, looks for what happened, how well the test article performed vs. some criteria, and makes predictions about future performance. The use of statistics for test design and analysis requires extensive training , beyond the scope of this course . However, testers should nevertheless understand some key concepts and terms, which have been simplified and summarized below and in the following screens.

Basic statistics ConcJ -----------------------... Long Description

Descriptiv e statistics : tograms, curves, and frequency distributions. A large circle with the word population surrounds a deviation.

smaller circle with word sample.

Inferential statistics : ~m!"'!!!l'l'l'l!'lml~i!"'l~lll'"~'"!lll!''!lll~~l"!mmlft!-,m!l'l'lmfhce of the en tire population.

....rl I PageSof 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Review of Some Basic Statistics Concepts

Statistics is the collection and interpretation of quantitative data and the use of mathematics to estimate population parameters. In testing, the analyses of sample data, using statistical techniques, looks for what happened, how well the test article performed vs. some criteria, and makes predictions about future performance. The use of statistics for test design and analysis requires extensive training , beyond the scope of this course . However, testers should nevertheless understand some key concepts and terms, which have been simplified and summarized below and in the following screens.

Basic statistics Concept~ Extensive Training

Descriptive statistics: de frequency dis tribu tions. Ex.

Refreshers on basic statistic concep ts are available as a Con tinuous Learning Module (CLE

istograms, curves, and rd deviation.

Inferential statistics: dra 035), accessible via the DAU Continuous Learning Center as well as other sources. See References ance of the entire

population. for details .

....rl I PageSof24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Review of Some Basic Statistics Concepts, Cont.

Probability is a measure of how likely it is that an event will occur expressed as a percentage. Example: 100% ( 1.0) =event always occurs ; while 0% =event never occurs. Most probabilities of test interest fall somewhere between these two extremes.

Frequency Distribution: is a listing of the values that a variable takes in a sample from a larger population sample. T ypically these values are plotted in some manner. Common ones include:

Norm al Distribution: encountered for many naturally-occurring phenomena. Often called the "bell curve" because of the shape of its probability density curve.

Binomial Dist ribution: encountered when outcomes are measured in terms of event success or failure with only two possible outcomes (e.g., in testing, SRAW launch success or failure ) .

Exponent ial Dist ribution: is often used to model the time between independent events that happen at a constant average rate . For example, measures of success or failures over time/ distance/ cycles/ etc. In testing, it is used heavily for reliability calculations.

Confidence Level is the probability that an interval (example: SRAW maximum range = 800 ± 50 meters) will contain some percentage of the total population. For example, in this case a value of 80% means that we are 80% confident that the test data (SRAW sample value) accurately represents that the SRAW will have a maximum range between 750 and 850 meters.

....rl I Page6of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Sample Size and Confidence Levels

"Sample size" refers the number of times a specific test event is run . I t is a critical parameter in the testing process. Obviously, the more times a specific test event is performed, the higher the confidence level will likely be in terms of accuracy when test results are evaluated.

However, while increasing test sample sizes lowers the risk of inaccurate evaluations, it also directly increases testing costs and lengthens schedules. There is a point of decreasing returns where increasing sample size does not appreciably reduce risks . Part of good test planning involves using statistics to aid in the selection of an appropriate sample size for later testing that achieves a reasonable balance between evaluation accuracy and testing affordability.

Variation in test result data is a natural consequence the testing process. Excessive variation in test results, especially "outliers" (data far outside the expected curve of results ), indicates potential system performance anomalies (design flaws) or a poorly run test event.

The smaller the variance in the test data, the more confident we can be in the conclusions .

....rl I Page7of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Sample Size and Confidence Levels

"Sample size" refers the number of times a specific test event is run . It is a critical parameter in the testing process. Obviously, the more times a specific test event is performed, the higher the confidence level will likely be in terms of accuracy when test results are evaluated.

However, while increasing test sample sizes lowers the risk of inaccurate evaluations, it also directly increases testing costs and lengthens schedules. There is a point of decreasing returns where increasing sample size does not appreciably reduce risks . Part of good test planning involves using statistics to aid in the selection of an appropr sonable balance between evaluation accuracy and te Decreasing Returns

Variation in test result date As a practical example, check out the trusty ssive variation in test results, especially "outliers" performance anomalies (de

monograph of the binomial distribution available in Resources. You'll see that, for a given reliability level (say 90% ), for one failure,

~i cates potential system

The smaller the variance in increasing the sample size from 40 to 70 only conclusions. changes the confidence level from 0.95 to 0.995, an increase of just 0.045 but at the expense of nearly twice as many test articles!

....rl I Page7of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Basic Analytic Techniques

A variety of analy tic techniques are used during the statistical analysis of test data. Using these techniques effectively for test design and later for data analysis requires significant levels of mathematical training as well as practical experience. Some of the more common techniques used for statistical inference are summarized in simplified form below:

Hypot hesis Test ing: Setting up and testing a hypotheses is an essential part of statistical inference. In order to formulate such a test, usually some theory has been put forward, either because it is believed to be true or because it is to be used as a basis for argument, but has not been proved. For example, claiming that the SRAW warhead arming distance is 16 meters.

As part of Hypothesis Testing, two hypotheses are set up. One is typically referred to as the "null" hypothesis (e.g., SRAW will arm in less than 16 meters); the other is the "alternative" hypotheses (e.g., SRAW will not arm is less than 16 meters) . Sophisticated mathematical techniques are applied to determine degree and risks of rejection or acceptance of these hypotheses.

T Test : the T-test assesses whether the means of two groups are statistically different from each other and is used in hypothesis testing under certain circumstances. This test is appropriate whenever you want to analy tically compare the means of two groups. For example, if the difference between the mean values of measured noise levels from testing on two different production lots of the SRAW are statistically significant. The T test is used most often on small sample sizes .

....rl I Page8of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Basic Analytic Techniques, Cont.

ANOVA: Analysis of variance is a collection of statistical models, and their associated procedures, in which the observed variance is partitioned into components due to different explanatory variables. ANOVA is used in place of the T test when an experiment or test involves more than two samples.

Factorial Analysis: Is a technique to determine effects of many test factors, such as operator, production lot, weather, etc. simultaneously during a test event in the most optimum way . This technique is one that is used in test design to get the maximum amount of valid data from the minimum amount of tests . Done properly, a single test can produce data that can be used to assess multiple criteria.

Regression Analysis: Involves plotting data in a manner so that mathematical analysis can be performed on it to determine "best fit" curves that account for the relationships between variables. For instance, performing a regression analysis of data related to SRAW accuracy at 800 meters as a function of gunner engagement angles. A graphic relationship between the two, based on actual test data that depicts all possible engagement angles would be developed. Accuracy for SRAW engagement angles not explicitly tested can then be estimated by interpolation or extrapolation from the graphic .

Interpolation is a moderately risky method of estimating untested values within the data set. Ex trapolation is estimating values by extending the regression line beyond the data set and is high risk because grossly inaccurate predictions may result.

....rl I Page9of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Before Testing: Scienti fic Test and Analysis Techniques

DASD(DT&E), in collaboration with the Commander Air Education and Training Command, established the Scientific Test and Analysis Techniques (STAT) in T&E Center of Excellence (COE) in April 2012 under the stewardship of the Air Force Institute of Technology (AFJT) . The COE attained Full Operational Capabili ty in July 2012.

During development of the Test & Evaluation Master Plan (TEMP), the COE works with acquisition program managers and the program's Chief Developmental Tester to improve test effectiveness and ensure efficient use of scarce resources . Utilizing a combination of rigorous scientific methods and lessons learned, the COE determines where test designs can be improved and efficiencies gained, and then applies this knowledge to the program's T&E strategy development.

The use of STAT does not ensure the success of a program, but rather allows programs to make better use of available test resources, allows for an understanding of information that can be obtained within the current test budget, and allows the understanding of the impact if more or less test resources were available. The use of STAT information provides the program leadership a better understanding to manage risk by ensuring effective testing highlights the current technological maturity, suitability, and operational effectiveness of the program.

....rl I Page 10 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Evaluation

Once the test data has been analyzed, evaluation can begin. The evaluation process involves more than just interpreting analyzed test data. I t also involves drawing conclusions and making recommendations .

Drawing conclusions:

Evaluating data means making deductions about system performance and making inferences about potential capabili ty. These inferences are generally associated with specific confidence levels that provide a measure of how closely the data matches reality, and how far beyond the actual test data you are inferring performance. Evaluation should also address each test objective, measure of performance, and measures of effectiveness, as well as issues related to the T&E processes that were used to obtain the data.

Making recommendations:

Evaluating data also means making recommendations on relevant issues, significant findings, and any thing else that might affect decisions and associated consequences. The evaluation should also take into account the user, by making recommendations regarding the impact on operations and maintenance. I f appropriate, the evaluation should also recommend any additional testing that may be necessary.

....rl I Page 11 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRINT I HELP

Analysis versus Evaluation

Many people in the T&E community erroneously use the terms analysis and evaluation interchangeably, but they are different.

Analysis can be thought of as a mechanical manipulation (hand or computer) and examination of ac tual data.

Evaluat ion takes the results o f the analysis and applies addi tional reasoning (mental manipulation) to arrive at conclusions and recommendations.

1'111111111111 1 Paige12of24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PR INT I HELP

Analysis versus Evaluation

Many people in the T&E community erroneously use the terms analysis and evaluation interchangeably, but they are different.

Analysis can be thought of as a mechanical manipulation (hand or computer) and examination of ac tual data.

Evaluation takes the results of the analysis and applies addi tional reasoning (mental manipulation) to arrive at conclusions a

Long Description

The animation shows a businessman lost in thought with question marks flashing over his head. Flying out from either side of his head are the words, 'Analysis' and ' Ev aluation' .

1'111111111111 1 Paige12of24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PR INT I HELP

Knowledge Review

Testing, Analysis and Evaluation are closely linked but distinct stages. Which of the following statements would be most closely linked to 'Testing"?

~ Weight = 20 lbs. and Force = 100 lbs.

lJ Accelera tion due to the launch motor is insufficient

Mass =Force/Acceleration = 100/ 32 = 3.1 slugs

Force = (20/ 32) x 32 = 20 lbs.

Check Answer

Weight - 20 lbs. a nd Force = 100 lbs. would be closely linked to "Testing."

1'111111111111 1 Paige13 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Evaluation: Role of Quality Requirements

As you've learned elsewhere in this course, the first stage of the testing process is "Test Planning". Test Planning consists of identifying information needs and using them for various plans and pre-test activities. These information needs are determined by capabilities as translated into requirements for the system and measures of effectiveness, suitability and performance.

Various Systems Engineering Technical Processes develop and refine system requirements . As part of this, steps are taken to ensure that "quality requirements" are developed. Quality criteria for a requirement includes: completeness, consistency, appropriateness, non-ambiguity, achievability, and verifiabili ty. "Verifiabili ty" is an especially important criterion, which if not addressed properly, will cause evaluation problems later on in this stage of the test process. Aspects of verifiabili ty which are particularly important for evaluation include requirements that are:

• Measurable: some aspect of the requirement can be quantified;

• Observable: requirement satisfaction can be determined easily;

• Objective: no confusion exists as to what is meant;

• Realist ic: no "pie-in-the-sky" wish list.

Because of the impact on the success of the testing process, a "best practice" is to involve T&E personnel early-on in development to assess requirements as they are written for their verifiabili ty. This will pay big dividends later on during analysis and evaluation.

~ I Page14of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Evaluation: Role of Quality Requirements

As you've learned elsewhere in this course, the first stage of the testing process is "Test Planning". Test Planning consists of identifying information needs and using them for various plans and pre- test activities. These information needs are determined by capabilities as translated into requirements for the system and measures of effectiveness, suitability and performance.

Various Systems Engineering Technical Processes develop and refine system requirements . As part of this, steps are taken to ensure that "quality requirements" are developed. Quality criteria for a requirement includes: completeness, consistency, appropriateness, non- ambiguity, achievability, and verifiabili ty. "Verifiabili ty" is an especially im ortant criterion, which if not addressed ro erl , will cause evaluation problems later on in this stag re particularly important

Technical Processesfor evaluation include require

Are used to design and realize the system • Measurable: some as products. Consist of: Stakeholder Requirements • Observable: requirem Definition, Requirements Analysis, Architecture

Design, Implementation, Integration, Verification, • Objectiv e : no confusi Validation and T ransition .

• Realistic: no "pie- in - the- s 1st.

Because of the impact on the success of the testing process, a "best practice" is to involve T&E personnel early- on in development to assess requirements as they are written for their verifiabili ty. This will pay big dividends later on during analysis and evaluation.

~ I Page14of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PR INT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• Not heavy

• More user· frjendly than current system

• Shall be desjgoed JAW MIL-0 -14567

• 100% majntajnable

• No false alarms

• Must use proven technology

• Interfaces wjth other systems must be effective

1'111111111111 1 Paige 15of24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy Not heavy

• More user-friendly than current system

This requirement is not specific enough. • Shall be designed JAW MJL-0-14567

• 100% maintainable

• No false alarms

• Must use proven technology

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy

• More user-friendly than current system

• Shall be designed JAW MJL-0-14567

• 100% maintainable

• No false alarms

• Must use proven technology

More user-friendly t han current syst e m

This requirement is ambiguous. How much better? In what characteristic? What is the current system?

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy Shall be desig ned IAW MIL-Q-14567

• More user-friendly than current system

This requirement specifies how to design and • Shall be designed JAW MJL-0-14567 build it rather than specifying the requirement.

• 100% maintainable

• No false alarms

• Must use proven technology

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy 100•/o maintainab le

• More user-friendly than current system

This requirement is unrealistic. • Shall be designed JAW MJL-0-14567

• 100% maintainable

• No false alarms

• Must use proven technology

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy No false alarm s

• More user-friendly than current system

This requirement is unrealistic. • Shall be designed JAW MJL-0-14567

• 100% maintainable

• No false alarms

• Must use proven technology

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I HELP

Requirements Problems

Invalid requirements can cause many problems in system design and other T&E activities. Here is a list of examples of invalid requirements . Think about why each of these might cause a potential problem then select on each one for an explanation.

• No t heavy Must use proven t echnology

• More user-friendly than current system

This requirement is ambiguous. What is the • Shall be designed JAW MJL-0-14567 definition of proven technology?

• 100% maintainable

• No false alarms

• Must use proven technology

• Interfaces with o ther systems must be e ffec tive

....rl I Page 15 of 24 ~ Back Next

TST102 Fundamentals of Test Evaluation

Lesson 19 - Analysis and Evaluation RESOURCES I PRIMT I H