Experience and Lessons-learned in Applying the Tri-Service Assessment Initiative Process 28 Jan 2003 W. Bail The MITRE Corporation and NAVSEA / PEO IWS Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 28 JAN 2003 2. REPORT TYPE 00-00-2003 to 00-00-2003 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Experience and Lessons-learned in Applying the Tri-Service Assessment 5b. GRANT NUMBER Initiative Process 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Carnegie Mellon University,Software Engineering REPORT NUMBER Institute,Pittsburgh,PA,15213 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES presentation at the Conference on the Acquisition of Software-Intensive Systems, January 28-30, 2003. 14. ABSTRACT 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE Same as 26 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 Overview Description of the experience and lessons-learned as a result of q applying the Tri Service Assessment process to an acquisition organization within NAVSEA (PEO IWS née PEO TSC) IWS = Integrated Warfare Systems Focus on the specific challenges encountered q and the process adaptations made to ensure the assessments were successful These recommendations are pragmatic q Tips to make things better This briefing does not summarize the technical findings q but rather focuses on how they were achieved 28 Jan 2003 2 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 Why assessments? At certain times, for certain programs, assessments are q necessary to Check for latent surprises Accelerate process improvement planning Identify problems and root causes Facilitate budget and resource planning Assess risk for upcoming events (e.g., OPEVAL) ... Use of assessments has proved to be a valuable tool to assist in q acquiring software-intensive systems But assessments must be carefully conducted to achieve q maximum utility 28 Jan 2003 3 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 Why TAI process was selected Key features q Pre-defined, well-used process Network of independent experts available to serving on assessment teams Cross-service perspective (“How did they solve it?”) Overall goal for conducting assessments: q Evaluation and improvement of the software processes used by programs within the PEO Identification of weaknesses and strengths Determination of risk and risk mitigation strategies Establishment of software guidance to raise level of development quality 28 Jan 2003 4 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 Purpose of assessments Prepare for CEC OPEVAL by ensuring no unpleasant surprises q were lurking – assess the five core systems that use CEC as a means for track sharing » CEC - Cooperative Engagement Capability » E2C - NAVAIR program » ACDS - Carrier Self Defense System » Aegis 6 Phase 1 » C2P - Command Control Processor - SPAWAR - PMW 159 Three separate organizations are represented among these q systems (NAVSEA, NAVAIR, and SPAWAR) Four of the five programs were assessed q (Subsequently, an additional six programs have been assessed q within the PEO) 28 Jan 2003 5 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 What we learned... The benefits to performing assessments are substantial q Clear and timely snapshot of current state of program Roll-up of lessons-learned Direct assistance to the programs But, Unless carefully performed, these benefits may be difficult to q achieve There exist several pitfalls that can derail ability to derive value q from assessments Team had to adapt to maximize effectiveness q 28 Jan 2003 6 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 Pitfalls 1 “Slow roll” in scheduling interviews and site visits 2 Site visits causing disruption 3 Ankle biter reputation 4 ”Fire hose" site visits 5 Missing important topics 6 "SW only" assessment 7 Extended assessment schedule 8 High-level and process-only 9 ”Dump and run" syndrome 10 Multiple, unrelated issues 11 Late final reports 12 Unclear presentation of findings 13 Criticism causing defensiveness 14 Recommendations requiring excessive time 28 Jan 2003 7 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 1 – “Slow roll” in scheduling interviews and site visits Being the subject of an assessment is never fun q Like going to the dentist Tendency is to delay it as long as possible q Blaming it on coordinating schedules Tasks that are important, however, seem to always get done q Need to ensure that everyone has same sense of priority q Recommendations: q Secure assurances from PM that assessment is of high-priority Ensure that PM makes this clear to organizations being assessed Assessment team members need to be flexible to accommodate reasonable schedule changes Emphasize that very little staff time is really needed (see #2) Define a schedule and follow it 28 Jan 2003 8 WGBail “Experience and Lessons-learned in Applying Tri-Service Assessment Initiative Process” Presented at the Conference on the Acquisition of Software-Intensive Systems — January 28-30, 2003 2 – Site visits causing disruption In all cases, encountered significant push-back from q organizations being assessed – fear of disruption and delays For a program in trouble, impact could be serious Disrupting the progress of the program by performing the q assessment is counterproductive Often create more problems than those solved Need to ensure a ripple-free process q and to set their minds at ease in advance 28 Jan 2003 9