軟件工程課件：15-QualityControl

1、Top 10 Software Failures 2012US Department of Justice site hackedRoyal Bank of Scotland (RBS) software glitchTwitter crashes Linkedin passwords leakedGoDaddy downOver 10,000 email IDs has been hacked Knight capital software glitchApple maps errors Click Frenzy failsChicago Election website crashes a

2、nd US election glitch1/top-10-software-failures-2012/The Big Cost of Software Bugs over the last 50 yearsIntels Miscalculation, 1994A flow in Intel Corps popular Pentium processor in 1994. Replacement of customer chips cost Intel $475 million. Y2K problem$296.7 billion was spent worldwide from 1995

3、to 2001 to mitigate the damage. Algorithm mistake in investment model, Axa Rosenberg Group, 2009Cost clients $217 million in losses Bad Brake for Toyota, 2010Recall more than 400,000 hybrid vehicles, estimated $3 billion losses.2/slideshow/2012-08-03/the-big-cost-of-software-bugs.htmlThe Cost of Sof

4、tware Bugs3Software bugs are costing the U.S. economy an estimated $59.5 billion each year, with more than half of the cost borne by end users and the remainder by developers and vendors, according to a new federal study. ComputerWorld 2002Worldwide cost of IT failure (revisited): $3 trillion. M. Kr

5、igsman, ZDNet 2012According to recent Cambridge University research, the global cost of debugging software has risen to $312 billion annually. The research found that, on average, software developers spend 50% of their programming time finding and fixing bugs Cambridge University 2013The Costs of So

6、ftware Failures: Time, Money and Your Job. Vincent, Wired 2014Healthy4世衛(wèi)健康標(biāo)準(zhǔn)軀體健康心理健康社會(huì)適應(yīng)良好道德健康5What is software quality?Conformance to specificationQuality that is defined as a matter of products and services whose measurable characteristics satisfy a fixed specification that is, conformance to an i

7、n beforehand defined specification.Meeting customer needsQuality that is identified independent of any measurable characteristics. That is, quality is defined as the products or services capability to meet customer expectations explicit or not. - Hoyer, R. W. and Hoyer, B. B. Y., What is quality?, Q

8、uality Progress, no. 7, pp. 52-62, 2001.6What is software quality?The difficulty in defining quality is to translate future needs of the user into measurable characteristics, so that a product can be designed and turned out to give satisfaction at a price that the user will pay. This is not easy, an

9、d as soon as one feels fairly successful in the endeavor, he finds that the needs of the consumer have changed, competitors have moved in etc.- Out of the crisis : quality, productivity and competitive position, Deming, W. E., Cambridge Univ. Press,1988.7MaCall Software Quality ModelProductRevisionP

10、roductTransitionMcCall, J. A., Richards, P. K., and Walters, G. F., Factors in Software Quality, Natl Tech. Information Service, no. Vol. 1, 2 and 3, 1977.ProductOperationsMaintainabilityTestabilityFlexibilityInteroperabilityPortabilityReusability Correctness ReliabilityUsability EfficiencyIntegrity

11、8MaCall Software Quality Model11 Factors (To specify)Describe the external view of the software, as viewed by the users.23 quality criteria (To build)Describe the internal view of the software, as seen by the developer.Metrics (To control)Defined and used to provide a scale and method for measuremen

12、t.9ISO 9126 Software Quality ModelBased on the McCall and Boehm models.Internal and external quality characteristics of software productsInternal metric: a quantitative scale and measurement method, measuring an attribute or characteristic of a software product, derived from the product itselfExtern

13、al metric: a quantitative scale and measurement method, measuring an attribute or characteristic of a software product, derived from the behavior of the system 10ISO 9126 Software Quality ModelCritical System and Dependable Computing11Critical SystemThe failure of many software-controlled systems ca

14、n result in significant economic losses, physical damage or threats to human life. Safety-critical systemsFailure may results in injury, loss of life or major environmental damageThe control system for chemical manufacturing plant, nuclear power plant Mission-critical systemsFailure may result in th

15、e failure of some goal-directed activityNavigational system for a spacecraftBusiness-critical systemsFailure may result in the failure of the business using that systemCustomer account system in a bank 12Critical SystemThe cost of failure of a critical system are often very highDirect failure costs,

16、 e.g. failure recoveryIndirect cost, e.g. litigation and lost businessThe trustworthiness of development techniques and process is usually more important than the costs of applying these techniques13Critical SystemThe development of critical systemsWell-tried techniques rather than newer techniquesE

17、.g. most systems are still based on a function-oriented design, rather than OOSoftware engineering techniques that are not normally cost-effective may be usedE.g. formal specification and verificationThe costs of verification and validation (V&V) are usually very high and may consume more than 50% o

18、f the total system development costs.14Trustworthy Computing (TWC)“Trustworthiness is assurance that a system deserves to be trusted that it will perform as expected despite environmental disruptions, human and operator error, hostile attacks, and design and implementation errors. Trustworthy system

19、s reinforce that belief that they will continue to produce expected behavior and will not be susceptible to subversion. ”- Schneider FB, Ed. Trust in Cyberspace. Committee on Information Systems Trustworthiness, National Research Council, Washington, DC, 1999. 15Microsoft TWC InitiativeJanuary 2002,

20、 Bill Gates historic emailCraig Mundies white paper and Bills follow up in July“Trustworthy computing is the highest priority for all the work we are doing. Trustworthy computing is computing that is as available, reliable and secure as electricity, water services and telephony.In the past, weve mad

21、e software and services more compelling for users by adding new features and functionality, and by making our platform richly extensible. Weve done a terrific job at that, but all those great features wont matter unless customers trust our software. So now, when we face a choice between adding featu

22、res and resolving security issues, we need to choose security. ”16Dont forget COST17Dependable Computing IEEE-CS TC on Fault-Tolerant Computing, 1970IFIP WG 10.4 Dependable Computing and Fault Tolerance, 1980J. C. Laprie, Dependability: Basic Concepts and Terminology, 1992. 18DependabilityThe abilit

23、y to deliver service that can justifiably be trusted.19Dependable Computing The Threats20FaultA fault is the adjudged or hypothesized cause of an error. A fault is active when it produces an error, otherwise it is dormant. ErrorAn error is that part of the system state that may cause a subsequent fa

24、ilure: a failure occurs when an error reaches the service interface and alters the service. FailureAn event that occurs when the delivered service deviates from correct service. A service fails either because it does not comply with the functional specification, or because this specification did not

25、 adequately describe the system functionFault, Error and Failure21Lifecycle Perspective22Maintenance152050100TestingCodingDesignRequirementThe CostThe DistributionDependable Computing The MeansFault PreventionHow to prevent the occurrence or introduction of faultsFault ToleranceHow to deliver the co

26、rrect service in the presence of faults Fault RemovalHow to reduce the number or severity of faultsFault ForecastingHow to estimate the current number, the future incidence and likely consequence of faults2324Verification and ValidationVerification: Are we building the product right?Software conform

27、s to its specificationValidation: Are we building the right product?Software meets the needs of the customer25V&V TechniquesSoftware inspectionStatic V&VAnalyze and check system representations, such as specification, model, codeSoftware testingDynamic V&VExecute the software with test data and exam

28、ine the outputsThe first mistake that people make is thinking that the testing team is responsible for assuring quality. Brian Marick26The V Process ModelSoftware Inspection2728Software InspectionInvolve people examining the source representation with the aim of discovering anomalies and defectsVery

29、 effective technique for discovering defectsInformal walkthroughPeer reviewFormal inspectionFagan 76, Gilb 94Formal processPlanning Preparation Meeting ReworkRoles and responsibilities29Fagan InspectionIBM, Michael Fagan, 1976A structured process of trying to find defects in development documents su

30、ch as programming code, specifications, designs and others IBM Huston，Aerospace Software，2,000,000 lines of codeAmong all the detected defects, 85% by inspection, 15% by testingIBM North Harbor93% defects detected by inspectionProductivity increased 9%30Fagan Inspection - A brief history1972 Walkthr

31、oughs/reviews common practice in IBM.1976 Paper on inspections in IBM System Journal, by M.Fagan.1979 Value of inspections acknowledged by IBMs largest individual award to M.Fagan.1986 Paper in IEEE Software on inspections, by M.Fagan.2001 100 client organizations trained to date by Michael Fagan.31

32、Fagan Inspection Motivation Used the following working hypotheses of current practice: 50% of development effort was actually used for defect rework. Effort to rework a defect increased in each phase by 10X up to 100X by end of the development cycle and was higher in the field.Recognized that creati

33、ve original work often contains defects and it is our business to do creative original work. People make mistakes!Fagan Inspection The Process32PROCESS OPERATIONOBJECTIVESPlanningMaterial, Inspectors, SchedulesOverviewPresent overviewPreparationLearn material, prepare role, DO NOT focus on finding d

34、efectsInspection meetingFIND DEFECTS!Process improvementLearn from last operations to improve next inspectionReworkRework all defectsFollow upVerify all fixes 33Fagan Inspection RolesThe participants of the inspection process are normally just members of the team that is performing the project. The

35、participants fulfill different roles within the inspection process Author/Designer/Coder: the person who wrote the low-level document Reader: paraphrases the document Tester: reviews the document from a testing standpoint Moderator: responsible for the inspection session, functions as a coach Exampl

36、e Inspection Checks34Fault classInspection CheckData faultsAre all program variables initialized before their values are used?Control faultsIs each loop certain to terminate?Input/Output faultsAre all input variables used?Interface faultsAre the parameters in the right order?Storage Management Fault

37、sIf space explicitly de-allocated after it is no longer required?Exception management faultsHave all possible error conditions been taken into account?35Fagan Inspection Inspection rate500 statements/hour during overview125 source statement/hour during individual preparation90-125 statements/hour ca

38、n be inspected during the meetingInspection is an expensive processInspecting 500 lines costs about 40 man/hours Fault Tolerance ArchitectureFault-Tolerance The property that enables a system to continue operating properly in the event of the failure.Masking redundancy (Neumann, Moore and Shannon,19

39、56)Fault-tolerance /graceful degradation systems designed around the concepts of fault tolerance.Integrating masking with practical techniques of error detection, fault diagnosis, and recovery (A. Avizienis, 1967)N-version programming (A. Avizienis, 1977)Fault-tolerant computer systemsThe Goal: Ensu

40、re that system faults DO NOT result in system failure37Fault tolerance actionsFault detectionThe system must detect that a fault (an incorrect system state) has occurred.Damage assessmentThe parts of the system state affected by the fault must be detected.Fault recoveryThe system must restore its st

41、ate to a known safe state.Fault repairThe system may be modified to prevent recurrence of the fault. As many software faults are transitory, this is often unnecessary.38Approaches to fault toleranceDefensive programmingProgrammers assume that there may be faults in the code of the system and incorpo

42、rate redundant code to check the state after modifications to ensure that it is consistent.Fault-tolerant architecturesHardware and software system architectures that support hardware and software redundancy and a fault tolerance controller that detects problems and supports fault recoveryThese are

43、complementary rather than opposing techniques39Fault DetectionDiscover the error at run-timeDetect an erroneous system state and throw an exception to manage the detected faultTwo types of detectionPreventative Before the state change is committedRetrospectiveAfter the system state has been changedc

44、lass PositiveEvenInteger int val = 0 ;PositiveEvenInteger (int n) throws NumericExceptionif (n 0 | n%2 = 1)throw new NumericException () ;elseval = n ; / PositiveEvenIntegerpublic void assign (int n) throws NumericException if (n 0 | n%2 = 1)throw new NumericException ();elseval = n ; / assignint to

45、Integer () return val ; /to Integerboolean equals (PositiveEvenInteger n) return (val = n.val) ; / equals /PositiveEven40Damage AssessmentAssess what parts of the state space have been affected by the failureGenerally based on validity functions which can be applied to the state elements to assess i

46、f their value is within an allowed rangeSome techniquesChecksums for data transmissionRedundant pointers for integrity of data structuresWatch dog timers for non-terminating processes41Fault RecoveryForward recoveryApply repairs to a corrupted system stateForward recovery is usually application spec

47、ific Domain knowledge is required to compute possible state correctionsBackward recoveryRestore the system state to a known safe stateBackward error recovery is simpler. Details of a safe state are maintained and this replaces the corrupted system state.42class SafeSort static void sort ( int intarr

48、ay, int order ) throws SortError int copy = new int intarray.length;/ copy the input arrayfor (int i = 0; i intarray.length ; i+)copy i = intarray i ;try Sort.bubblesort (intarray, intarray.length, order) ;if (order = Sort.ascending)for (int i = 0; i intarray i+1)throw new SortError () ;elsefor (int

49、 i = 0; i intarray i)throw new SortError () ; / try blockcatch (SortError e ) for (int i = 0; i intarray.length ; i+)intarray i = copy i ;throw new SortError (Array not sorted) ; /catch / sort / SafeSortFault detectionFault detectionFault recoveryRestore the original value of the array43Hardware Fau

50、lt Tolerance ArchitectureDepends on triple-modular redundancy (TMR)There are three replicated identical components which receive the same input and whose outputs are comparedIf one output is different, it is ignored and component failure is assumedBased on most faults resulting from component failur

51、es rather than design faults and a low probability of simultaneous component failure44Hardware reliability with TMR45Output selectionThe output comparator is a (relatively) simple hardware unit.It compares its input signals and, if one is different from the others, it rejects it. Essentially, select

52、ion of the actual output depends on the majority vote.The output comparator is connected to a fault management unit that can either try to repair the faulty unit or take it out of service.Software Fault Tolerant ArchitectureThe success of TMR at providing fault tolerance is based on two fundamental

53、assumptionsThe hardware components do not include common design faultsComponents fail randomly and there is a low probability of simultaneous component failureNeither of these assumptions are true for softwareIt isnt possible simply to replicate the same component as they would have common design fa

54、ultsSimultaneous component failure is therefore virtually inevitableSoftware systems must therefore be diverse Common Cause Fault (CCF) 共因失效Design diversityDifferent versions of the system are designed and implemented in different ways. They therefore ought to have different failure modes.Different

55、approaches to design (e.g object-oriented and function oriented)Implementation in different programming languagesUse of different tools and development environmentsUse of different algorithms in the implementationSoftware analogies to TMRN-version programmingThe same specification is implemented in

56、a number of different versions by different teams. All versions compute simultaneously and the majority output is selected using a voting system.This is the most commonly used approach e.g. in Airbus 320. Recovery blocksA number of explicitly different versions of the same specification are written

57、and executed in sequenceAn acceptance test is used to select the output to be transmitted.N-version programmingOutput comparisonAs in hardware systems, the output comparator is a simple piece of software that uses a voting mechanism to select the output.In real-time systems, there may be a requireme

58、nt that the results from the different versions are all produced within a certain time frame.N-version programmingThe different system versions are designed and implemented by different teams. It is assumed that there is a low probability that they will make the same mistakes. The algorithms used sh

59、ould but may not be different.There is some empirical evidence that teams commonly misinterpret specifications in the same way and chose the same algorithms in their systems.Recovery blocksRecovery blocksForce a different algorithm to be used for each version so they reduce the probability of common

60、 errors.However, the design of the acceptance test is difficult as it must be independent of the computation used.There are problems with this approach for real-time systems because of the sequential operation of the redundant versions.N-Version vs. Recovery BlockN-VersionStatic Forward recoveryReco