Available online at www.sciencedirect.com Procedia Computer Science 16 ( 2013 ) 621 – 630 Conference on Systems Engineering Research (CSER’13) Eds.: C.J.J. Paredis, C. Bishop, D. Bodner, Georgia Institute of Technology, Atlanta, GA, March 19-22, 2013. Application of Systems Engineering to USAF Small Business Innovative Research (SBIR) Phillip O’Connell, Maj, USAFa; J. Robert Wirthlin, PhD, Lt Col, USAFb*; James Malas, PhDc; Som Soni, PhDd aAir Force Lifecycle Management Center, 1981 Monahan Way, Wright-Patterson AFB, OH 45433 bAir Force Institute of Technology, 2950 Hobson Way, Wright-Patterson AFB, OH 45433 cAir Force Research Laboratory, 1864 Fourth Street, Wright-Patterson AFB, OH 45433 dAdTech Systems Research, Inc., 1342 N. Fairfield Road, Beavercreek, OH 45432 Abstract The US Air Force Small Business Innovation Research (SBIR) program is a vital element of the Air Force Research Laboratory (AFRL) portfolio. The SBIR program funds early-stage R&D projects at small technology companies that support a US Department of Defense (DoD) need and have the potential for commercialization in private sector and/or military markets. We report on research measuring the application of Systems Engineering (SE) and the degree of Systems Engineering applied in SBIR projects through analysis of subject-matter expert (SME) interviews. SMEs were sampled from AFRL as well as other USAF organizations. The research methodology assessed the current application of SE in SBIR as well the SE expectations government offices have for SBIR projects. The research specifically examined current US DoD and US Air Force Systems Engineering Policy and found that it does not adequately address SBIR projects. SE processes are not well documented within the community. We then identified applicable SE tasks found in Air Force SE policy and suggest a tailored Systems Engineering approach for SBIR projects. We present a SBIR SE checklist recommended for use by SBIR program managers tailored according to the phase and scope of any SBIR project. ©© 22001133 T Thhee A Autuhtohros.r sP.u Pbulibshliesdh ebdy Ebyls eEvliseer vBie.Vr .B.V. SSeelleeccttiioonn a anndd/o/ro pr epeer-erre-vreievwie uwn duenr dreesrp roensspiobinlsitiyb iolfi tGy eoofr gGiae oInrsgtiiatu Iten sotfi tTuetech onfo Tloegcyhnology Keywords:Systems Engineering (SE), Small Business Innovative Research (SBIR), process tailoring, checklist 1. Introduction The US DoD’s SBIR program is a large part of the multibillion dollar federal SBIR program administered by twelve federal agencies across the country [www.sbir.gov/about/about-sbir]. The SBIR program funds early- stage R&D projects at small technology companies that support a US government agency need and have the potential for commercialization in the private sector and/or military markets. SBIR projects are managed by many * Corresponding author. Tel.: +1-937-255-3636 x4650; fax: +1-937-656-4633. E-mail address: [email protected] 1877-0509 © 2013 The Authors. Published by Elsevier B.V. Selection and/or peer-review under responsibility of Georgia Institute of Technology doi: 10.1016/j.procs.2013.01.065 Report Documentation Page Form Approved OMB No. 0704-0188 Public reporting burden for the collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington VA 22202-4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to a penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. 1. REPORT DATE 3. DATES COVERED 2013 2. REPORT TYPE 00-00-2013 to 00-00-2013 4. TITLE AND SUBTITLE 5a. CONTRACT NUMBER Application of Systems Engineering to USAF Small Business Innovative 5b. GRANT NUMBER Research (SBIR) 5c. PROGRAM ELEMENT NUMBER 6. AUTHOR(S) 5d. PROJECT NUMBER 5e. TASK NUMBER 5f. WORK UNIT NUMBER 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION Air Force Lifecycle Management Center,1981 Monahan REPORT NUMBER Way,Wright-Patterson AFB,OH,45433 9. SPONSORING/MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSOR/MONITOR’S ACRONYM(S) 11. SPONSOR/MONITOR’S REPORT NUMBER(S) 12. DISTRIBUTION/AVAILABILITY STATEMENT Approved for public release; distribution unlimited 13. SUPPLEMENTARY NOTES 14. ABSTRACT The US Air Force Small Business Innovation Research (SBIR) program is a vital element of the Air Force Research Laboratory (AFRL) portfolio. The SBIR program funds early-stage R&D projects at small technology companies that support a US Department of Defense (DoD) need and have the potential for commercialization in private sector and/or military markets. We report on research measuring the application of Systems Engineering (SE) and the degree of Systems Engineering applied in SBIR projects through analysis of subject-matter expert (SME) interviews. SMEs were sampled from AFRL as well as other USAF organizations. The research methodology assessed the current application of SE in SBIR as well the SE expectations government offices have for SBIR projects. The research specifically examined current US DoD and US Air Force Systems Engineering Policy and found that it does not adequately address SBIR projects. SE processes are not well documented within the community. We then identified applicable SE tasks found in Air Force SE policy and suggest a tailored Systems Engineering approach for SBIR projects. We present a SBIR SE checklist recommended for use by SBIR program managers tailored according to the phase and scope of any SBIR project. 15. SUBJECT TERMS 16. SECURITY CLASSIFICATION OF: 17. LIMITATION OF 18. NUMBER 19a. NAME OF ABSTRACT OF PAGES RESPONSIBLE PERSON a. REPORT b. ABSTRACT c. THIS PAGE Same as 10 unclassified unclassified unclassified Report (SAR) Standard Form 298 (Rev. 8-98) Prescribed by ANSI Std Z39-18 622 Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 different organizations throughout the DoD. Within the Air Force, SBIR projects are managed by AFRL Technology Directorates, Test Centers and Air Logistics Centers. The Air Force Small Business Innovation Research (SBIR) program is vital element of the Air Force Research Laboratory (AFRL) contracts portfolio operated under the guidance of the Air Force SBIR/Small Business Technology Transfer (STTR) Program Manager within AFRL at Wright Patterson Air Force Base. The DoD and Air Force provide top level SE guidance and policy for the acquisition community. However the current guidance and policy has not yet been tailored specifically for the SBIR community. This research explores this gap and potential solutions in an applied setting at the AFRL. The focus of this paper is to present our assessment of SE applicability to SBIR programs and not address broad SE and acquisition issues that are beyond the scope of S&T programs. The DoD has well defined system engineering processes documented in the department’s 5000 series of instructions on acquisition. The guidance cited in this paper applies to both the AFRL SBIR project manager, since his responsibilities include systems engineering, and the SBIR awardee because both are part of the S&T acquisition community. However a number of challenges exist with applying SE to SBIR projects since they are unique compared to typical acquisition programs. They are managed by many different small businesses that may or may not have an organic SE capability. Additionally they vary significantly in scope, are small in size, short in project length, and are early research projects. They are categorized as Basic Research or Applied Research projects. Topics are generated across the Air Force by Program Executive Officers, Technology Directorates, Air Logistics Centers and Test Centers. A SBIR project is developed in three phases. Phase 1 is a technical feasibility study that allocates up to $150k and 9-12 months to completion. Phase II is concept development and allocates up to $1M and 24 months to completion. There are also Critical Manufacturing SBIR projects that are allocated up to $5M for Phase II. Phase III is the commercialization stage [www.sbir.gov]. As SBIR projects vary considerably in scope, are managed by many different organizations within the government, and work is accomplished by varying small businesses, a problem of consistent SE processes being applied across all projects is presented. 1.2 Research Focus We seek to identify how current SE practices apply to SBIR projects. This includes identifying how and what current DoD and Air Force SE policies apply to SBIR projects during Phase I and II and how to best tailor the guidance to develop a solid SE approach for the technical management of the projects. Thus the subject of this research focuses on implementation of early SE processes for Phase I and II SBIR projects. We suggest without a solid SE approach SBIR projects are at risk to fail, while good SE processes will help ensure projects are better prepared for proceeding to their next phase of development while adequately managing technical risk. 2.0 Background and Literature Review The US Department of Defense publishes the Defense Acquisition Guide (DAG) and the DoD 5000 series of instruction to identify SE processes. The Air Force has also published a series of instructions. Air Force Instruction (AFI) 63-1201 “Life Cycle SE” is for the acquisition community. AFI 63-1201 defines SE responsibilities for program managers and engineers. Additionally, Air Force Material Command, the organization in the Air Force most responsible for system development, created the Air Force SE Assessment Model (AF SEAM) for assessment of Air Force programs. SEAM also contains goals and specific as well as general practices that can be used in assessment. Differences between the DAG, AFI 63-1201 and AF SEAM can be confusing since they vary in terminology. The tables below identify the SE processes identified in each document. Oneillustrates how the DAG has sixteen distinct processes, AF 63-1201 has twelve processes and AF SEAM has ten, and the other shows the breakout of SEAM into goals and practices. Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 623 Table 1: SE Processes (AF SEAM, Sept 2010) Table 2: AF SEAM Tasks (AF SEAM, Sept 2010) AF SEAM Tasks AFI 63-1201 is currently under revision to better align with AF SEAM and the DAG. The revised version of AFI 63-1201 should reduce misalignment by eliminating the current differences with AF SEAM. 2.1 Air Force Systems Engineering Assessment Model (SEAM) The primary purpose of AF SEAM is to promote the application and use of standard SE processes across the AF and to improve the performance of these processes through Continuous Process Improvement [AF SEAM, September 2010]. AF SEAM is not yet mandated across the Air Force however it is used as a reporting tool in some AF communities. AF SEAM identifies “ten AF standard SE process areas” and lists associated goals for each. Specific practices and generic practices are identified for each area. Table 2 identifies 190 total practices in AF SEAM. This suggests a significant SE effort is required for any program to implement AF SEAM. 2.2 AFRL Policy The Air Force Research Labs have two main documents providing guidance for SE. The first is AFRL Instruction 61-104 which addresses SE in the S&T environment. It is in alignment with DoD and Air Force policy but tailors it to the S&T community. It also identifies Eight Systems Engineering Key Questions to guide and assess the SE health of a project. These are mapped back to the DAG SE process areas. The second document is AFRL 61- 204, a Manual for Scientist and Engineers. An attachment identifies “What the Program Manager should know about his or her program.” It says “Use of the key questions during reviews of basic research programs is optional.” Also in an attachment, AFRL translates the 16 SE DAG processes for the AFRL community as shown in Figure 1. It defines each process for AFRL and explains the importance. The AFRI Instruction specifically declares it is tailored for a typical AFRL program and not for SBIR programs. Figure 1: AFRL S&T SE Process (Draft AFRL SE Guidebook, 2012) 2.3 Past Research 624 Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 Several theses and past studies have been accomplished analyzing SE efforts within AFRL. A past research project “A Tailored SE Framework for S & T Projects” captures the SE tasks and rigor applicable for typical AFRL projects (Pitzer, Behm and White, 2009). They developed a tool, the “Systems Engineering Tailoring tool for Science & Technology Projects,” that defined projects by 6 parameters: RDT&E Category, Project Budget, Core Process, Technology Readiness Level, Integration Level, and Requirements Maturity. The tool suggests what SE best practices (mapped from the 16 DAG processes) would apply to that project/program. This tool is notably similar to AF SEAM however it tailors the tasks for a project based on the stated parameters. Additionally four notable AFRL studies are significant for this research: 1. High Energy Laser On a Large Tactical Platform (HELLTP) 2. Deployed Base Energy Alternatives Report 3. Company Grade Officer Initiative Program (CGOIP) 4. AFRL Transformational Activities in Systems Engineering (TASE) Assessment Phase Final Report Findings from 2006 The first two focus on the successful tailoring and streamlining of SE efforts on two larger AFRL projects. The third focused on tailoring and streamlining SE efforts of smaller projects in CGOIP. This study was notable since the projects were being managed by CGOs with limited SE backgrounds – not unlike SBIR project managers. And like the first two studies listed, CGOIP was deemed successful in implementing good SE processes into their projects using a streamlined SE approach. The final study focused on making AFRL research programs more effective and efficient and by improving the transition of technology to the warfighting community through the use of good SE processes. The studies identified common key SE attributes: forming multi-disciplinary teams involving all relevant stakeholders; holding team reviews to monitor project progress; and tailoring their SE approach with a streamlined S&T process consistent with Air Force policy. 2.4 Summary of Literature Review of SE policy within the DoD, Air Force and AFRL has identified defined SE guidance and instruction. AFRL has done a good job tailoring their policy to be in alignment with higher level guidance and continues to be proactive in implementing good SE processes within AFRL. However, the literature does not specifically address the application of Systems Engineering to SBIR. This constitutes a knowledge gap in the understanding of systems engineering. Subsequent discussions with AFRL officials recognized that AF SEAM is a rigorous tool for implementing SE but is truly not tailored for the Science and Technology (S&T) community. In its current configuration, many of the 190 SE tasks may not apply given the attributes of the AFRL project. Further, implementing this model “as is” does not make sense for the SBIR community either due to the more unique, further limited resources and questionable value these tasks will add to a project. A more tailored approach is required. Preliminary research within the SBIR community identified varying levels of SE knowledge and implementation amongst project managers. SE guidance needs to be better defined and tailored for the SBIR community to ensure SE deliverables are sufficient for transitioning the project to the next phase of research. The SETT tool provides a good initial baseline for an AFRL sponsored project, however, SBIR projects do not fit the traditional project mold. Implementing a process or tool that is not tailored to the appropriate level risks creation of non-value added work and can drain valuable resources from a project. Nevertheless, both the SETT Tool and AF SEAM will be good baselines to consider when identifying what SE tasks may apply to SBIR projects. Finally, the various studies underscored the need to develop a tailored SE approach for the SBIR community. They identify many weaknesses in the S&T community and suggest some policy changes for good implementation of SE processes. 3.0 Methodology A reference framework using the material identified in the review of literature, SBIR documentation and applicable SE policy was created. In most cases there was little if any SE documentation available. Therefore, using Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 625 a mixture of qualitative and quantitative methods this research seeks to develop a tailored system engineering approach for SBIR projects as an exploratory work. Inductive theory or grounded theory development is the key methodological component. Interviews were used to gather additional information. This approach also helped identify current data requirements and degree of SE rigor currently associated with SBIR projects. Then interview and literature data was analyzed through structured content analysis using data triangulation methods to identify the relevant SE practices for SBIR. Triangulation methods in data analysis improves the validity and reliability of this type of research [Glafshani, 2003]. Specifically, the interview instrument to collect data from SBIR project managers is semi-structured so that open-ended responses are encouraged and snowball sampling could occur. Interviewees were identified from different AFRL and external AF organizations based upon their position in the organization and familiarity with the SBIR process. This group included several SBIR program managers. Once all interviews were complete and data had been collected the results were analyzed to help suggest the appropriate SE tasks for SBIR projects. Figure 2 represents the flow of the methodology used in this research. Figure 2: Methodology 3.1 Research Objectives and Hypothesis One objective of this research project is to define the SE rigor that should be best applied to SBIR projects. To define the rigor and design a tailored approach requires identifying what degree of SE is applicable in the SBIR environment and how those SE processes vary amongst projects with respect to project maturity, size and other factors. A working hypothesis is that different organizations implement varying levels of SE into their SBIR projects based upon some of their defined needs. Furthermore, organizational SE policies and the SE knowledge base vary amongst SBIR project managers. Therefore, these projects carry some risk to meet DoD and AF standards required by policy and instruction in the implementation of SE and SE processes. 3.2 The Interview Instrument and Sample Selection The interview was designed to investigate current AFRL policy. One question for each SE process outlined in the DAG was created corresponding with the specific tasks called out in AFRL policy. Additional questions were asked regarding the effectiveness of the Eight SE Key Questions outlined in AFRL policy. Table 3: Total Interview Participants Test Air Logistics Technology Directorates Centers Centers Other Materials & Space Human Manufacturing Propulsion Vehicles Effectiveness Munitions Arnold Robins ASC 4 7 3 2 2 2 3 1 Table 3 indicates the total numbers of personnel interviewed and the different organizations they represent. The ideal participants for the interview were identified as SBIR Program Managers (PM), engineers (EN) and AFRL organizational chief engineers (CE) because they are the most familiar with the daily technical management of a SBIR project. The goal was to interview several PMs, ENs, and CEs from each organization. SBIR project managers were a natural choice as they perform day-to-day management of the SBIR project. These project managers are engineers or program managers with varying levels of experience. Chief Engineers of the directorates 626 Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 oversee the management of these projects within AFRL. Participants for the interview were identified through purposeful and snowball sampling. 4.0 Analysis and Results Interview results are analyzed in terms of the tasks identified from the DAG. Then the findings surrounding the 8 SE Key Questions are discussed and finally, the results relative to AF SEAM are given. Figure 3 below illustrates the percentage of positive responses for the tasks (ranging from two to five for each) identified from AFRL Policy captured in the interview for each SE process area. For areas with more than 50% task use, green is shown; yellow is used showing 30% to 50% and red for areas with less than 30% task use in each SE process area. Roughly speaking, participants identified that only a total of 55% of the SE tasks listed in the survey were applicable to SBIR projects. % of SE Tasks Accomplished by Technology Directorates 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 627 % of SE Tasks % of SE Tasks Accomplished by Test Accomplished by ALC's Centers 100% 100.00% 80% 80.00% 60% 60.00% 40.00% 40% 20.00% 20% 0.00% 0% Figure 3: Results from Test Centers, Technology Directorates, and Air Logistic Centers (Horizontal axis is the same on all charts – abbreviated on last two for space) Test Center interviewees complete 42% of SE Tasks. No interviewees identified the SE tasks of Architectural Design, Decision Analysis and Interface Management as applicable. Since the interview instrument was developed using AFRL policy, we believe some of the tasks were couched in AFRL language and participants did not necessarily identify with them. Technology Directorate interviewees complete 62% of SE Tasks. This is notable for the Technology Directorates because so many of the SE tasks identified in AFRL policy are not accomplished during Phase I & II of SBIR projects. Additionally several areas of weakness are seen in areas where few tasks are completed in a particular DAG categorized SE area. This may illustrate how SBIR projects are unique when compared to typical S & T projects within AFRL. Air Logistics Center interviewees complete 35% of SE Tasks. As before, since the interview instrument was developed using AFRL policy, we believe some of the tasks were couched in AFRL language and participants did not necessarily identify with them. Preliminary analysis of these results also suggests that Air Logistics Centers and Test Centers have a different focus or purpose in their SBIR projects. 4.1 Value of the AFRL Eight Systems Engineering Key Questions Only those interviewees from AFRL organizations were asked about the value and usefulness of the Eight SE Key Questions identified in AFRLI 61-104. Table 4 shows the qualitative value interviewees associated with the questions. Table 4: Value of the 8 AFRL SE questions from interviewee responses Didn't Some- know they not a little what moderately very extremely Total existed useful useful useful useful useful useful useful Participants 8 SE Key Questions 4 1 1 2 0 2 3 2 15 Almost seventy percent of participants who knew of the SE questions thought they were useful to some degree. No definitive conclusions about the value of these questions are drawn due to the small sample size, but there is concern that these questions were completely unknown to several interviewees. Table 5 shows how each participant subjectively identified the most and least useful of the 8 SE questions. Some interviewees gave multiple answers on the most useful questions while others did not identify any “Least Useful” questions. 628 Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 Table 5: AFRL SE Key Questions Mapped to Interview Results Most Useful Least Useful 1. Who is your customer? 5 2 2. What are the customer’s requirements? 5 3. How will you demonstrate you have met the requirements? 5 1 4. What are the technology options? 1 2 5. Which is the best approach? 1 6. What are the risks to developing the selected technology? 2 1 7. How will you structure your program to meet requirements and mitigate risk? 2 8. What is your business-based transition plan that meets customer approval? 4 Participants felt that the first three questions were the most important and the least important question was question 8. This is likely because not all SBIR projects transition into larger projects. These responses warrant further investigation as conclusions cannot definitively be draw based upon the interview design. 4.2 AF SEAM Comparison Comparing the interview results with AF SEAM the applicable SE tasks for the SBIR community were identified. However, as discussed earlier, AF SEAM and AFRL policy do not directly align since AFRL policy maps back to the DAG. Only about 50% of the tasks from AF SEAM were found to be applicable for the SBIR community. Many of the tasks identified in AF SEAM are not applicable until the later phases of a program. Further, AF SEAM requires a large manpower effort to complete, due to the sheer number of its SE tasks (190). Therefore it would not be useful to implement AF SEAM within the SBIR community, without tailoring. 5.0 Conclusions and Recommendations Based upon the analysis and results, a SBIR SE Checklist (see Table 6) was developed as a guide for project managers and engineers to ensure SE is adequately addressed in SBIR projects. This checklist also aligns with the 10 AF SEAM SE processes areas. Those using this checklist should begin to accomplish some of these tasks in Phase 1 with the emphasis of demonstrating the project is feasible, identifying stakeholders and defining requirements. For instance, it is critically important that AFRL project managers clarify AF requirements or basic research objectives. Small businesses do not usually possess organic resources to determine DoD requirements. AFRL project managers, though, are in a position to get access to capability gaps or engage AF Programs of Record or contact other key contractors for more information on AF requirements. By the end of Phase 2 all of the listed tasks should have been tailored to the specifics of each SBIR project and accomplished. Projects that enter Phase 2.5 should emphasize further defining and documenting information and demonstrating the technology with the hopes to aid in the future transition of the project to Phase III. The following symbols are used to interpret the checklist: Represents general AF SE process tasks tailored for SBIR - Represents specific SE tasks captured in analysis Table 6: SBIR SE Checklist Determine requirements to include stakeholder needs, expectations, constraints, and interface requirements. - Translate all stakeholder needs to technical requirements. - Requirements made quantifiable, have unique definitions, and specified thresholds and objectives. - Work with stakeholders to refine requirements. - Performance parameters and constraints allocated and derived technical requirements defined. - Maintain the traceability of all requirements from needs. Requirements - Document changes and record rationale of changes. Identify project milestones to include cost, schedule and technical milestones. - Define the scope of the tech effort required to achieve program technical goals. Project Planning - Define exit criteria and products/deliverables which can be tracked with progress measured. Phillip O’Connell et al. / Procedia Computer Science 16 ( 2013 ) 621 – 630 629 Risk Management Develop a risk management plan and identify, analyze, identify handling options, mitigate and track risk. Establish selection criteria, identify & evaluate alternatives and select solution. - Criteria selected for decision & methods to be used in conducting the analysis. Decision Analysis - Identify analysis methods and conduct analysis of alternatives. Establish the design and integration baseline. - Incorporate the lower-level system elements into a higher-level system element in the physical architecture. Design - Indentify constraints & interfaces that will result in derived technical requirements. Establish and maintain the project environment, integrated product teams (IPT), measurements approach Technical and monitor technical reviews, work products, project data, corrective actions and technical milestones. Management & - Measure technical progress, technology maturity and the effectiveness of plans and requirements. Control - Demonstrate and confirm completion of required accomplishments and project exit criteria. Establish the technical baseline, track and document changes. - Maintain record of all configurations to include hardware, software and test set up and document changes. Configuration - Define internal and external interfaces. Management - Project data managed through the Defense Technical Information Center. Establish and maintain the overall verification strategy and plan to include verification and validation criteria and an integrated testing approach when applicable. Verify and Validate that the project has meets the required parameters. - Confirm project meets design specifications. Verification & - Test the system elements against their defined requirements. Validation - Test the performance of the technology against the original program goals. Indentify future transition, fielding, & sustainment requirements as needed to proceed to the next phase of the project. - Needs of follow-on phases considered early in the program and included in all of the technical Transition, Fielding, management processes. & Sustainment - Yield the fundamental capability of the program. Identify and maintain documentation relevant to the future production of the project. Manufacturing - Develop supporting documentation for the system. Current policy does not fully define SE for the SBIR community and SE is being implemented at various levels amongst the different organizations that manage SBIR projects. Leadership and project managers must ensure adequate levels of SE are being incorporated into their projects to improve their chance of success, limit cost and schedule overruns and meet performance goals. Good implementation of SE processes can be measured by the smooth successful transition of these projects to the next phase of development. Completion of all SE tasks in this checklist can be used as a way to measure implementation of SE processes for each SBIR project. Failure to follow established SE processes in any one area can have significant negative consequences to the project. Specifically, the research identified: The SBIR community does not believe all SE tasks in AFRL policy apply to their projects The SBIR community does not consistently implement SE tasks. There is a wide spectrum of interpretation that is partially due to the sponsoring organization type and the phase of the SBIR project. Those results identified weak areas within the current policy. DoD and Air Force SE processes must be tailored for the scope of the SBIR project. Applicable SBIR SE tasks identified by this research are shown in Table 6 above. 5.1 Significance of Research This research revealed a SE policy gap in the SBIR community and defined applicable SE tasks. This represents a huge risk as millions of dollars are spent within the DoD each year on SBIR projects. Failure to implement good SE principles can and will lead to cost overruns, schedule slips and performance short falls. Findings from this research should be used to tailor a SE approach for SBIR projects to ensure SE practices are being implemented in a best practice manner.