軟件外包公司組織形式商務(wù)模板perancemeasurementforsoftwareorganizations

1、 1 DFAS Software Symposium August 25, 1999 Dave Zubrow Software Engineering Institute Sponsored by the US Department of Defense Performance Measurement for Software Organizations 2 Bring Me A Rock! “We need a measurement program. Get one started.” “We dont have time to define our goals. We have prod

2、ucts to deliver.” “We collect a lot of data. We just never seem to use it.” 3 Outline Some questions about performance measurement: What is performance measurement? Who is the audience/consumer of performance information? How are the data for performance measurement produced? What do the results mea

3、n? A process for defining performance measures 4 What is IT Performance Measurement Quantitative characterization of an organizations accomplishment of some aspect of its goals with a focus on the contribution of IT quantitative - need something more discriminating than success/failure, yes/no organ

4、ization - focus is on the organization or enterprise view, not a specific project or program aspect - performance is multidimensional, what to measure is not obvious goals - for measurement to be meaningful, we need a reference point for comparison and judgement contribution of IT - attribution of o

5、rganizational performance to IT performance 5 Performance Management Framework Source: National Academy of Public Administration, “Information Management Performance Measures,” January, 1996. 6 User interfaces ReliabilityPredictability Software Unit Potential Contributions Business Strategy Elements

6、 usability rating user training availability mean time to failure system availability defect density On-time delivery cost variance Metrics High Priority Contributions Relating IT Performance to the Business Strategy 7 Who is the audience? do we all speak the same language? We all want the business

7、to succeed, but revenue market share mean time to failure defect densityinspections Customer satisfaction key process areas 8 Business Managers Attention Business Operating Systems Departments and Work Centers Quality Delivery Cycle Time Waste MarketFinancial Customer Satisfaction Productivity Visio

8、n Flexibility Business Units Adapted from Cross & Lynch “Measure Up: Yardsticks for Continuous Improvement.” 9 QualityDelivery Cycle Time Waste Technical View Business Units Business Operating Systems Departments and Work Centers MarketFinancial Customer Satisfaction Productivity Vision Flexibility

9、10 The Audiences and Their Interests Senior management - for strategic decisions business managers IT managers Improvement team - to implement improvements and know how well they are doing IPTs IT process action teams Customers - to evaluate suppliers and understand their capability 11 How are the d

10、ata produced? QualityDelivery Cycle Time Waste Business Units Business Operating Systems Departments and Work Centers MarketFinancial Customer Satisfaction Productivity Vision Flexibility Data generated by work processes and transactions Information used to assess performance and guide improvement 1

11、2 What do the results mean? Possible Interpretations Accomplished a goal - has the goal been met Progress towards a goal - are trends moving in the right direction according to schedule Impending threat - can signal risk of not meeting future goal Guidance for improvement - what should we look at as

12、 an opportunity for improvement Competitive position - ranking or performance relative to competitors It depends on the goal and strategy 13 A Process for Measuring the Performanceof Information Technology. Follow an IT Results Chain Use a Balanced Scorecard Target Measures at Decision Making Tiers

13、Build a Measurement and Analysis Infrastructure Strengthen IT Processes to Improve Mission Performance Source: “Executive Guide-Measuring performance and demonstrating results of information technology investments,” US General Accounting Office, March 1998. 14 Business Goals define the Need The Proc

14、ess provides the Opportunity Alignment is the Key Business Goals What do I want to achieve? Subgoals To do this, I will need to Mental Model receivesproduces holds consists of entitiesentities attributesattributes Measurement Goals Questions Indicators Measures G1 Q1 I2 Q2Q3 I1I3I4 M1M2M3 G2 What do

15、 I want to know? entities attributes 15 A Balanced Perspective on Performance Customer How do customers see us? Financial How do we look to shareholders? Innovation and Learning Can we continue to improve and create value? Internal Business What must we excel at? A Balanced Perspective Watch out for

16、 masked trade-offs, unintended consequences Can improvement in one area be made without sacrificing another? See Kaplan and Norton, “The Balanced Scorecard - Measures that Drive Performance” Harvard Business Review, Jan/Feb, 1992. 16 A Balanced Scorecard Example Return on Capital Employed Cash Flow

17、Project Profitability Profit Forecast Reliability Sales Backlog Pricing Index Tier II Customers Customer Ranking Survey Customer Satisfaction Index Market Share Business Segment Tier I Customers Key Accounts % Revenue from New Services Rate of Improvement Index Staff Attitude Survey # of Employee Su

18、ggestions Revenue per Employee Hours with Customers on New Work Tender Success Rate Rework Safety Incident Index Project Performance Index Project Closeout Cycle Financial Perspective Customer PerspectiveInternal Business Perspective Innovation and Learning Perspective Source: Kaplan and Norton, ”Pu

19、tting the Balanced Scorecard to Work” Harvard Business Review, Sept-Oct 1993 17 Verify Tasks Action Plans Analysis & Diagnosis Weeks Module Weeks Define Indicators Step 6 Data Elements AvailSource + 0 - 0 + - - QA CM ? Etc. Size Defects Step 7 Identify Data Elements definition checklist _ 4 4 _ 4 4

20、_ 4 4 _ 4 4 _ 4 4 Step 8 Define Data Elements Complete Planning Task Matrix Step 9 Step 5 Formalize Measurement Goals Step 4 Questions, Entities, & Attributes Post Workshop Step 10 Implementation Goal-Driven Process Steps Measurement Workshop Planning Tasks Task 1 Task 2 Task 3 Task n Y Y Y NY Data

21、Elements 1 2 3 4 5 Planning Tasks Task 1 Task 2 Task 3 Task n Y Y Y NY Data Elements 1 2 3 4 5 Step 1 Identify your Business Goals Step 2 Identify what do you want to learn or know about the goals. Identify your Sub-goals Step 3 18 Operational Definitions Project Phases Feasible Study Alternative An

22、alysis Functional Specification Design InitiationDefinitionDesignBuildVerificationImplementation Code & Unit Test Integration Test UATDeployment Start DateStart Date End Date (ship date) ProjectEstimation definition checklist _ 4 4 _ 4 4 _ 4 4 _ 4 4 _ 4 4 definition checklist _ 4 4 _ 4 4 _ 4 4 _ 4 4

23、 _ 4 4 definition checklist _ 4 4 _ 4 4 _ 4 4 _ 4 4 _ 4 4 Effort & Schedule Estimate Key dates - start and end times 19 Characteristics of the Measures Mutually Exclusive Measure different dimensions with each measure Exhaustive Outcomes, Outputs, Inputs, Process Balanced Scorecard Valid The measure

24、s logically relate to their corresponding indicator or use Reliable The same performance would result in the same measurement Interval Scale Need variability to distinguish performance levels 20 Defining Performance Measures Document the why, what, who, when, where, and how INDICATOR TEMPLATE Object

25、ive Questions Visual Display Interpretation Evolution Assumptions X-reference Probing Questions Input(s) Data Elements Responsibility for Reporting Form(s) Algorithm 50 100 150 123456789 10 Weeks Now Planned Actual Trouble Reports Defects Cost of Quality Schedule Predictability Effort Predictability

26、 Cycle Time Maintenance Effort Project Mix Customer Satisfaction Measures INDICATOR TEMPLATE Objective Questions Visual Display Interpretation Evolution Assumptions X-reference Probing Questions Input(s) Data Elements Responsibility for Reporting Form(s) Algorithm 50 100 150 123456789 10 Weeks Now P

27、lanned Actual Trouble Reports INDICATOR TEMPLATE Objective Questions Visual Display Interpretation Evolution Assumptions X-reference Probing Questions Input(s) Data Elements Responsibility for Reporting Form(s) Algorithm 50 100 150 123456789 10 Weeks Now Planned Actual Trouble Reports 21 Criteria fo

28、r Evaluating Performance Measures - 1 Are we measuring the right thing? improvement in performance of mission improvement in performance of goals and objectives value added by IT organization ROI, costs, savings Based on strategy and objectives not whats convenient and “l(fā)ying around” relevant and im

29、portant Source: National Academy of Public Administration, “Information Management Performance Measures,” January, 1996. 22 Criteria for Evaluating Performance Measures - 2 Do we have the right measures? measures of results rather than inputs or outputs linked to specific and critical processes unde

30、rstood by their audience and users effective in prompting action credible and possible to communicate effectively accurate, reliable, valid, verifiable, cost-effective, timely Develop as a Set dont rely on a single indicator will trade-offs in performance be detected? Source: National Academy of Pub

31、lic Administration, “Information Management Performance Measures,” January, 1996. 23 Criteria for Evaluating Performance Measures - 3 Are the measures used in the right way? strategic planning guide prioritization of program initiatives resource allocation decisions day-to-day management communicate

32、 results to stakeholders Source: National Academy of Public Administration, “Information Management Performance Measures,” January, 1996. 24 Example: Process Improvement Goals Internal Processes increase productivity by a factor of 2 over 5 years reduce development time by 40% over 5 years improve q

33、uality by a factor of 5 over 5 year reduce maintenance effort by 40% over 5 years Customer Satisfaction improve predictability to within 10% over 5 years 25 Enterprise Metrics Project Size: Large Small Medium Schedule Predictability 0% 20% 40% 60% 80% 100% Percent Deviation 19961997 1 2341 234 Effor

34、t Predictability 0% 20% 40% 60% 80% 100% Percent Deviation 19961997 1 2341 234 Cycle Time 10 8 6 4 2 0 Calendar Days per Size Unit 19961997 1 2341 234 Quality Defect Density at UAT Number of High Priority Field Defects 10 8 6 4 2 0 1000 800 600 400 200 0 19961997 1 2341 234 Customer Satisfaction Imp

35、lemented Solution Working Relationship Satisfaction index 60% 50% 40% 30% 20% 70% 19961997 1 2341 234 Rework Appraisal Prevention Performance Cost of Quality: COQ Medium ProjectsCOQ Small ProjectsCOQ Large Projects 0 20 40 60 80 12345 Effort Hours per Size Unit 100 0 20 40 60 80 12345 Effort Hours p

36、er Size Unit 100 0 12345 Effort Hours per Size Unit 20 40 60 80 100 Maintenance Effort Non- discretionary Maint.Effort Open Trouble Reports Enhancemen t Effort 100% 80% 60% 40% 20% 0% 19961997 1 2341 234 Trouble Reports Open 1000 800 600 400 200 0 Percent of Tech. Staff-Hours Defect Corrections 26 E

37、xample Output Schedule Predictability The Objective is to understand the effectiveness the user acceptance test (UAT) was to be completed and the actual date when the UAT was completed along with the start date of coding of the project. The Percentage Deviation in schedule for different categories is calculated as follows: Absolute value (Actual Ship date - Planned Ship date) Percent Deviation= - * 100 Planned Ship date - Start date of coding A downward trend predicts improvement in