AFRL-ML-WP-TR-2001-4010 PROBABILITY OF DETECTION (POD) ANALYSIS FOR THE ADVANCED RETIREMENT FOR CAUSE (RFC)/ENGINE STRUCTURAL INTEGRITY PROGRAM (ENSIP) NONDESTRUCTIVE EVALUATION (NDE) SYSTEM DEVELOPMENT VOLUME 1 – POD ANALYSIS ALAN P. BERENS UNIVERSITY OF DAYTON RESEARCH INSTITUTE 300 COLLEGE PARK DAYTON, OH 45469-0120 JANUARY 2000 FINAL REPORT FOR PERIOD 29 SEPTEMBER 1995 – 31 DECEMBER 1999 Approved for public release; distribution unlimited. MATERIALS AND MANUFACTURING DIRECTORATE AIR FORCE RESEARCH LABORATORY AIR FORCE MATERIEL COMMAND WRIGHT-PATTERSON AIR FORCE BASE, OH 45433-7750 Report Documentation Page Report Date Report Type Dates Covered (from... to) 00012000 N/A - Title and Subtitle Contract Number Probability of Detection (POD) Analysis for the Advanced Retirement for Cause (RFC)/Engine Structural Grant Number Integrity Program (ENSIP) Nondestructive Evaluation (NDE) System-Volume 1: POD Analysis Program Element Number Author(s) Project Number Berens, Alan P. Task Number Work Unit Number Performing Organization Name(s) and Address(es) Performing Organization Report Number University of Dayton Research Institute 300 College Park Dayton, OH 45469-0120 Sponsoring/Monitoring Agency Name(s) and Sponsor/Monitor’s Acronym(s) Address(es) Materials and Manufacturing Directorate Air Force Research Laboratory Air Force Materiel Command Sponsor/Monitor’s Report Number(s) Wright-Patterson AFB, OH 45433-7750 Distribution/Availability Statement Approved for public release, distribution unlimited Supplementary Notes The original document contains color images. Abstract Subject Terms Report Classification Classification of this page unclassified unclassified Classification of Abstract Limitation of Abstract unclassified UU Number of Pages 99 Form Approved REPORT DOCUMENTATION PAGE OMB No. 074-0188 Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Washington Headquarters Services, Directorate for Information Operations and Reports, 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-4302, and to the Office of Management and Budget, Paperwork Reduction Project (0704-0188), Washington, DC 20503 1. AGENCY USE ONLY (Leave blank) 2. REPORT DATE 3. REPORT TYPE AND DATES COVERED JANUARY 2000 Final, 09/29/1995 – 12/31/1999 4. TITLE AND SUBTITLE 5. FUNDING NUMBERS PROBABILITY OF DETECTION (POD) ANALYSIS FOR THE ADVANCED C: F33615-95-C-5242 RETIREMENT FOR CAUSE (RFC)/ENGINE STRUCTURAL INTEGRITY PE: 63112F PROGRAM (ENSIP) NONDESTRUCTIVE EVALUATION (NDE) SYSTEM PN: 3153 DEVELOPMENT TN: 00 WU: 19 VOLUME 1 – POD ANALYSIS 6. AUTHOR(S) ALAN P. BERENS 7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES) 8. PERFORMING ORGANIZATION REPORT NUMBER UNIVERSITY OF DAYTON RESEARCH INSTITUTE UDR-TR-2000-00007 300 COLLEGE PARK DAYTON, OH 45469-0120 9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES) 10. SPONSORING / MONITORING AGENCY REPORT NUMBER MATERIALS AND MANUFACTURING DIRECTORATE AIR FORCE RESEARCH LABORATORY AFRL-ML-WP-TR-2001-4010 AIR FORCE MATERIEL COMMAND WRIGHT-PATTERSON AIR FORCE BASE, OH 45433-7750 POC: Charles Buynak, AFRL/MLLP, (937) 255-9807 11. SUPPLEMENTARY NOTES This is Volume 1 of 3 Volumes. 12a. DISTRIBUTION / AVAILABILITY STATEMENT 12b. DISTRIBUTION CODE Approved for public release; distribution unlimited. 13. ABSTRACT (Maximum 200 Words) In 1995, the US Air Force initiated a multifaceted program to upgrade the Eddy Current Inspection System (ECIS) system. Planned changes to the ECIS system included updating a) the eddy current instrument, b) the station computer and its operating system, and c) the robotics controller. These system changes were to be “drop-in” to avoid the costs of repeating a complete capability demonstration program for ongoing engine inspections. In addition, a new approach to calibration was demonstrated and a new Probability of Detection (POD) computer program was written for the analysis of data from capability demonstrations. This report presents the results of the study whose objectives were updating the POD analysis program and evaluating the data collected to demonstrate the drop-in compatibility of the upgraded and original ECIS systems. An additional task was later added to analyze data from an experiment designed to validate two proposed methods for inferring the POD of a geometry/material combination using inspection results from other geometry/material combinations. While the primary focus of this study was POD analysis for the RFC/ENSIP application with its highly automated ECIS, the POD analysis methods are applicable to a broad range of nondestructive evaluation (NDE) systems. 14. SUBJECT TERMS 15. NUMBER OF PAGES retirement for cause, engine structural integrity, probability of detection, eddy current 102 16. PRICE CODE 17. SECURITY CLASSIFICATION 18. SECURITY CLASSIFICATION 19. SECURITY CLASSIFICATION 20. LIMITATION OF OF REPORT OF THIS PAGE OF ABSTRACT ABSTRACT Unclassified Unclassified Unclassified SAR NSN 7540-01-280-5500 Standard Form 298 (Rev. 2-89) Prescribed by ANSI Std. Z39-18 298-102 i Table of Contents Section Page List of Figures ...............................................................................................................................v List of Tables .............................................................................................................................vii Acronyms, Abbreviations, and Symbols...................................................................................viii Foreword ...............................................................................................................................x 1 Introduction..................................................................................................................1 1.1 Overview of POD Demonstrations.......................................................................1 1.2 Update POD Program...........................................................................................3 1.3 Evaluation ECIS Modifications............................................................................3 1.4 Inferring POD(a) Through Material Correlation..................................................4 2 Capability Demonstrations...........................................................................................5 2.1 Demonstration Plan...............................................................................................5 2.1.1 Controlled and Uncontrolled Factors........................................................5 2.1.1.1 Inherent Variability....................................................................6 2.1.1.2 Flaw Variation............................................................................6 2.1.1.3 Hardware Variability..................................................................7 2.1.1.4 Human Factors...........................................................................8 2.1.2 Sample Size Requirements........................................................................9 2.1.2.1 Sample Size Requirements for â versus a Analysis.................10 2.1.2.2 Sample Size Requirements for Pass/Fail Analysis..................11 2.1.2.3 Unflawed Inspection Sites.......................................................12 2.1.3 Specimen Flaw Size Requirements.........................................................12 2.2 â Versus a Analysis............................................................................................17 2.2.1 â versus a Model Formulation................................................................18 2.2.2 Variability of â About Mean Response..................................................20 2.2.2.1 Inherent Variability in RFC/ENSIP ECIS Demonstration Data.................................................................20 2.2.2.2 Sensitivity of a to Parameter Variations...............................23 90 2.2.3 Transformation of Signal Response and Crack Size...............................25 2.3 Pass/Fail Analysis...............................................................................................34 2.3.1 Pass/Fail Model Formulation..................................................................34 2.3.2 Confidence Bounds on a ......................................................................36 90 2.3.2.1 Confidence Bounds for the Cumulative Lognormal Model......37 2.3.2.2 Confidence Bounds for the Log Odds Model..........................39 2.3.3 Goodness of Fit Test of Pass/Fail Models..............................................40 3 POD Program Update.................................................................................................41 3.1 User Interface......................................................................................................41 3.2 Analysis Additions..............................................................................................44 3.2.1 Transformations......................................................................................45 3.2.2 Tests of Assumptions for â versus a Analysis........................................46 iii 3.2.2.1 Linearity...................................................................................46 3.2.2.2 Homogeneity of Variance........................................................47 3.2.2.3 Normality.................................................................................48 3.2.3 Alternate Values of POD and Confidence Level....................................48 3.3 Output of POD....................................................................................................48 3.3.1 Results Sheet...........................................................................................49 3.3.2 Residuals Sheet.......................................................................................50 3.3.3 Threshold Data Sheet..............................................................................51 3.3.4 POD Data Sheet.......................................................................................51 3.3.5 Ahat vs a Sheet........................................................................................51 3.3.6 Fit Plot Sheet...........................................................................................52 3.3.7 Residual Plot Sheet..................................................................................52 3.3.8 Threshold Plot Sheet...............................................................................52 3.3.9 POD Sheet................................................................................................52 4 RFC/ENSIP ECIS Evaluations...................................................................................56 4.1 Eddy Current Instrument Validation...................................................................56 4.2 Station Computer Validation..............................................................................62 4.3 Robotic Controller Validation.............................................................................64 4.4 Integrated System Validation..............................................................................69 4.5 Scanner Validation..............................................................................................75 4.6 Calibration Method Validation...........................................................................80 5 Summary and Conclusions.........................................................................................84 6 References..................................................................................................................86 iv List of Figures Figure Page Figure 1 Example Plot of â versus a Data................................................................................10 Figure 2 Example â versus a Data for Small Crack Depths.....................................................14 Figure 3 Correlation of Length with Depth for IN 100 Flat Plate Specimens .........................15 Figure 4 75-Percent Limits on Crack Depths for IN 100 Flat Plate Specimens......................16 Figure 5 Example POD(a) Calculation from â versus a Data..................................................19 Figure 6 Example â versus a Inspection Data with Three Probes............................................21 Figure 7 Percent Error in a versus Error in s for Selected B ...............................................24 90 0 Figure 8 Example Non-linear ln â versus ln a Response for Titanium Bolt Holes..................28 Figure 9 Example Non-linear ln â versus ln a Response for Nickel Flat Plates......................29 Figure 10 Linear Fit to ln â versus l/a for Nickel Flat Plates.....................................................30 Figure 11 Fit from l/a Transformation on ln â versus l/a for Nickel Flat Plates........................31 Figure 12 a versus Decision Threshold Comparing Results from ln â versus l/a to 90 Original ln â versus ln a Analysis..............................................................................31 Figure 13 Linear Fit to ln â versus l/a for Titanium Bolt Holes................................................32 Figure 14 Fit from l/a Transformation on ln â versus ln a for Nickel Flat Plates......................33 Figure 15 a versus Decision Threshold Comparing Results from ln â versus l/a to 90 Original ln â versus ln a Analysis..............................................................................33 Figure 16 Comparison of POD(a) Confidence Bounds from Pass/Fail Analysis......................38 Figure 17 Partial Data Worksheet for an â versus a Analysis Using POD................................43 Figure 18 Example Info Sheet for an â versus a Analysis Using POD......................................43 Figure 19 Example â versus a Plot Exhibiting Smaller Residual Variance for Cracks of the Same Size as Compared to Variance of All Residuals....................................47 Figure 20 Example POD Results Sheet for an â versus a Analysis...........................................50 Figure 21 Example POD Results Sheet for Pass/Fail Analysis .................................................51 Figure 22 Example an Ahat versus a Sheet................................................................................53 Figure 23 Example Fit Plot Sheet for an â versus a Data...........................................................53 Figure 24 Example Fit Plot Sheet for Pass/Fail Data..................................................................54 Figure 25 Example Residual Plot Sheet.....................................................................................54 Figure 26 Example Threshold Plot Sheet...................................................................................55 Figure 27 Example POD Sheet...................................................................................................55 Figure 28 Comparison of Average â for Ti-6246 Bolt Hole Specimens, 2 MHz......................57 Figure 29 Comparison of Average â for Ti-6246 Bolt Hole Specimens, 6 MHz......................57 Figure 30 Comparison of Average â for Waspaloy Flat Plate Specimens, 2 MHz....................58 Figure 31 Comparison of Average â for Waspaloy Flat Plate Specimens, 6 MHz....................58 Figure 32 a Threshold Comparisons for Eddy Current Instruments – Ti-6246, 90 0.155” Bolt Holes, 2 MHz.........................................................................................60 Figure 33 a Threshold Comparisons for Eddy Current Instruments – Ti-6246, 90 0.155” Bolt Holes, 6 MHz.........................................................................................60 Figure 34 a Threshold Comparisons for Eddy Current Instruments – Waspaloy Flat 90 Plates, 2 MHz.............................................................................................................61 Figure 35 a Threshold Comparisons for Eddy Current Instruments – Waspaloy Flat 90 Plates, 6 MHz.............................................................................................................61 Figure 36 Comparison of â Values from the dt-rel Scan Plan...................................................64 v Figure 37 Comparison of â Values from Controllers – Waspaloy Flat Plates...........................66 Figure 38 Comparison of â Values from Controllers – IN 718 Bolt Holes...............................66 Figure 39 Comparison of â Values from Controllers – Ti-6246 Broach Slots, Mid..................67 Figure 40 Comparison of â Values from Controllers – Ti-6246 Elongated Scallops................67 Figure 41 Comparison of â Values from American Robotics Inspections.................................68 Figure 42 Comparison of Integrated System â Values – Waspaloy Flat Plates.........................70 Figure 43 Integrated System a Threshold Comparisons – Waspaloy Flat Plates....................70 90 Figure 44 Comparison of Integrated System â Values – IN 718 Bolt Holes.............................71 Figure 45 Integrated System a Threshold Comparisons – IN 718 Bolt Holes........................71 90 Figure 46 Comparison of Integrated System â Values – Ti-6246 Small Bolt Holes.................72 Figure 47 Integrated System a Threshold Comparisons – Ti-6246 Small Bolt Holes............72 90 Figure 48 Comparison of Integrated System â Values – Ti-6246 Elongated Scallops..............73 Figure 49 Integrated System a Threshold Comparisons – Ti-6246 Elongated Scallops.........73 90 Figure 50 Comparison of Integrated System â Values – Ti-6246 Broach Slot, Mid.................74 Figure 51 Integrated System a Threshold Comparisons – Ti-6246 Broach Slot, Mid............74 90 Figure 52 Comparison of Scanner â Values – IN 718 Bolt Holes.............................................76 Figure 53 Scanner a Threshold Comparisons – IN 718 Bolt Holes........................................76 90 Figure 54 Comparison of Scanner â Values – Waspaloy Flat Plates, 2 MHz............................77 Figure 55 Scanner a Threshold Comparisons – Waspaloy Flat Plates, 2 MHz.......................77 90 Figure 56 Comparison of Scanner â Values – Waspaloy Flat Plates, 6 MHz............................78 Figure 57 Comparison of Scanner â Values – Ti-6246 Broach Slot Edge Cracks....................79 Figure 58 Scanner a Threshold Comparisons – Ti-6246 Broach Slot Edge Cracks...............79 90 Figure 59 Calibration Comparison with ECIS Master Block – 2 MHz, Probe #1.....................81 Figure 60 Calibration Comparison with ECIS Master Block – 2 MHz, Probe #2.....................81 Figure 61 Calibration Comparison with ECIS Master Block – 6 MHz, Probe #1.....................82 Figure 62 Calibration Comparison with ECIS Master Block – 6 MHz, Probe #2.....................82 Figure 63 ECIS Master Block Probe Variability –2 MHz.........................................................83 vi List of Tables Table Page Table 1 Standard Deviation of a Values from Repeat Inspections under 90 Identical Conditions for Geometry by Material Combinations .................................21 Table 2 Median Standard Deviations of ln â by Source of Variability for Material by Geometry Combinations............................................................................................22 Table 3 Statistical Summary of s Values Obtained in ECIS Evaluations..............................23 Table 4 Specimen Test Matrix for PC Station Computer Validation.....................................62 Table 5 Summary Statistics from PC Computer Validation...................................................63 Table 6 Comparison of a Values from American Robotics and Original 90 Controller Units..........................................................................................................65 vii Acronyms, Abbreviations, and Symbols AFRL Air Force Research Laboratory AHAT Computer program for POD analysis from quantified signal responses ECIS Eddy Current Inspection System EDM Electrical Discharge Machined ENSIP Engine Structural Integrity Program NDE Nondestructive Evaluation P/F Computer program for POD analysis from hit/miss inspection responses POD Probability of Detection POD(a) Probability of Detection of flaws of size a RFC Retirement For Cause a Crack size â or ahat Response of NDE system to flaw A* Anderson - Darling test statistic for normality a "90 percent detectable crack size, POD(a90) = 0.9" 90 a Best estimate of 90 percent detectable crack size (about 50 percent confidence) 90/50 a Estimate of 90 percent detectable crack size with 95 percent confidence 90/95 â Response signal threshold for crack detection dec ahat or â Response of NDE system to flaw a Size of crack i i â Response of NDE system to flaw i i a Reliably detected crack size NDE a Crack size for which POD(ap) = 100 p p a Estimate of p percent detectable crack size with q percent confidence p/q B , B Slope and intercept of linear relation between log â and log a 0 1 C Differential random effect on â due to calibration blocks c 2 Equal variance test statistic - c 2 distribution c Differential random effect due to crack i i CP Differential random effect on â due to the interaction of calibration blocks and probes d Difference between predicted and actual â or partial derivative operator d* Sum of random effects due to c and p(i) i i j DX Error in estimate of parameter X F Lack of fit test statistic - F distribution F( ) Standard cumulative normal distribution, i.e., zero mean and unit standard deviation f(a) Functional relation between a and â g(â) Monotonic transformation of â G M Specimen comprised of Geomtry i and Material j i j h(a) Monotonic transformation of a i Index parameter j Index parameter k Number of inspections of a flaw L(qq) Likelihood of qq as a function of observed inspection results viii