California HVAC Contractor & Technician Behavior Study, Phase II CALMAC Study ID: SCE0375.01 April 30, 2015 F I N A L REPORT Presented To: Presented By: Jesse Emge Ellen Steiner & Donna Whitsett Project Manager, DSM Program EMI Consulting Measurement and Evaluation 83 Columbia St. Suite 400 Southern California Edison Seattle, WA 98104 1515 Walnut Grove Ave. Rosemead, CA 91770 TABLE OF CONTENTS TABLE OF CONTENTS 1. Introduction ................................................................................................................................................................ 1 1.1 Study Background and Overview ..................................................................................................... 1 1.2 Research Questions ............................................................................................................................ 2 2. Methodology ........................................................................................................................................................... 3 2.1 Implementer Interview Methodology .............................................................................................. 4 2.2 Market Actor Interview Methodology ............................................................................................. 4 2.3 Contractor Interview Methodology ................................................................................................. 5 Sample Frame Development & Completes - Participating Contractors ............................ 6 Sample Frame Development & Completes - Nonparticipating Contractors .................... 6 2.4 Technician Survey Methodology ..................................................................................................... 8 Technician Survey Sample Frame Development .................................................................... 8 Survey Completes and Dispositions ............................................................................................ 9 Assigning Survey Respondents to Survey Modules ................................................................ 11 2.5 Secondary Research Review to Operationalize MTIs ............................................................... 12 2.6 Research Limitations .......................................................................................................................... 12 3. Implementer Interview Results .......................................................................................................................... 14 3.1 Opportunities to Improve Training/Testing .................................................................................. 14 3.2 Communicating Industry Standards to Contractors and Technicians ................................... 15 3.3 Ensuring Standards are Enacted in the Field .............................................................................. 16 3.4 Best Practices and Contractor Business Models ........................................................................ 17 3.5 Barriers to QI/QM Program Success .............................................................................................. 18 3.6 Key Takeaways from Implementer Interviews ............................................................................ 19 4. Market Actor Interview Results ....................................................................................................................... 20 4.1 Contractor Business Models .......................................................................................................... 20 Successful Business Models ....................................................................................................... 20 Challenges and Trade-offs .......................................................................................................... 22 Participation Factors Considered ............................................................................................... 23 4.2 Technician Field Practices ............................................................................................................... 23 Challenges ....................................................................................................................................... 24 Respondent Recommendations for Addressing Implementation Challenges ............... 24 4.3 Sales Skills and Technical Training ............................................................................................... 25 Best Practices for Selling QI/QM ................................................................................................ 25 Technician Training and Testing ................................................................................................ 26 Training Methods and Topics ................................................................................................................. 26 Trainers and Timing .................................................................................................................................. 26 Types of Testing ......................................................................................................................................... 27 Perceptions of HVAC Industry Standards ................................................................................ 27 4.4 Key Takeaways from Market Actor Interviews ........................................................................... 28 5. Contractor Interview Results ............................................................................................................................ 29 5.1 Awareness and Use of Standards ................................................................................................. 29 5.2 Communicating Industry Standards to Technicians ................................................................. 29 5.3 Training and Certifications .............................................................................................................. 30 i CA HVAC C&T Behavior Study, Phase II 5.4 Sales and Sales Training ................................................................................................................... 31 5.5 Training Needs .................................................................................................................................... 31 5.6 Business Models ................................................................................................................................ 32 5.7 Benefits of Program Participation .................................................................................................. 32 5.8 Key Takeaways from Contractor Interviews ............................................................................... 33 6. Technician Survey Results ................................................................................................................................ 34 6.1 Respondent Characteristics ............................................................................................................ 34 Firm Size ............................................................................................................................................ 34 The Importance of Firm Size ................................................................................................................... 36 Years of Experience ....................................................................................................................... 36 Work Performed by Sector ........................................................................................................... 36 Service Territories .......................................................................................................................... 38 6.2 Training and Certifications .............................................................................................................. 38 Types of Training Received ......................................................................................................... 39 Effectiveness of Training Received ........................................................................................... 40 Certifications and Licenses Held by Technicians .................................................................. 42 Additional Training Topics ........................................................................................................... 44 6.3 Technician Field Experiences ......................................................................................................... 46 Specific Tasks Performed During Installation Jobs ............................................................... 46 Specific Tasks Performed During Maintenance Jobs ........................................................... 49 Technicians’ Top Priorities Onsite ............................................................................................ 50 Barriers to Making Repairs Onsite ............................................................................................. 53 6.4 Awareness and Use of Standards ................................................................................................. 54 Codes and Standards - Unprompted ........................................................................................ 54 Technician Definitions of Quality Installation and Quality Maintenance ........................ 56 Awareness of Installation Standard (ACCA 5) – Prompted ................................................ 59 Awareness of Maintenance Standard (ACCA 4, ACCA/ASHRAE 180) – Prompted ...... 61 6.5 Communication of Standards ......................................................................................................... 64 Sources of Awareness of Industry Standards ......................................................................... 64 Ensuring Standards are Followed in the Field ........................................................................ 65 6.6 Sales and Sales Training .................................................................................................................. 66 Selling Standards-based Installation Services ....................................................................... 66 Selling Standards-based Maintenance Services ................................................................... 68 Sales Training .................................................................................................................................. 70 6.7 Key Takeaways from Technician Surveys .................................................................................... 71 7. Operationalization of Market Transformation Indicators ......................................................................... 75 7.1 Methodological Considerations ..................................................................................................... 75 Proximate versus Ultimate Indicators ....................................................................................... 75 Long Term Comparability and Cost-effectiveness ................................................................ 76 7.2 MTI HVAC-1a and HVAC-1b ............................................................................................................. 76 Considerations for Operationalizing HVAC-1a and HVAC-1b ............................................. 77 Recommendations for Operationalizing HVAC-1a and HVAC-1b ...................................... 79 7.3 MTI HVAC-2 and HVAC-3 ................................................................................................................ 79 Considerations for Operationalizing HVAC-2 and HVAC-3 ................................................ 80 Recommendations for Operationalizing HVAC-2 and HVAC-3 .......................................... 81 7.4 MTI HVAC-4 ........................................................................................................................................ 83 ii TABLE OF CONTENTS Considerations for Operationalizing HVAC-4 ......................................................................... 83 Recommendations for Operationalizing HVAC-4 .................................................................. 84 8. Summary and Recommendations ................................................................................................................... 86 8.1 Summary of Key Findings ................................................................................................................ 86 Awareness, Use, and Communication of Industry Standards ............................................ 86 Terminology: Standards and Program Names ....................................................................... 87 Quality Contractor Best Practices .............................................................................................. 87 Training ............................................................................................................................................. 88 HVAC Market Transformation Indicators (MTIs) .................................................................... 90 Differences by Size and Sector ................................................................................................ 90 8.2 Recommendations .............................................................................................................................. 91 LIST OF TABLES Table 2-1: Implementer and Program Staff Interview Participants ............................................................ 4 Table 2-2: Market Actor Interview Participants .............................................................................................. 5 Table 2-3: Participant and Nonparticipant Contractor In-Depth Interview Sample Targets and Completes ................................................................................................................................................................ 6 Table 2-4: Stratification and Completes for Participating Contractor Interviews .................................. 6 Table 2-5: Stratification and Completes for Nonparticipating Contractor Interviews .......................... 8 Table 2-6: Completed Technician Surveys by Sector and IOU ................................................................. 9 Table 2-7: Technician Survey Call Dispositions ............................................................................................ 10 Table 2-8: Online Technician Survey Dispositions ...................................................................................... 10 Table 2-9: Mean Percentage of Jobs in Each Sector, by Survey Module Assignment ....................... 11 Table 6-1: Number of Employees in Respondents’ Firms .......................................................................... 35 Table 6-2: Statistical Summary of Firm Size by Assigned Survey Module ............................................ 35 Table 6-3: Statistical Summary of Technicians’ Years of HVAC Experience by Assigned Survey Module .................................................................................................................................................................... 36 Table 6-4: Number and Percentage of Technicians Performing Work by Sector ............................... 37 Table 6-5: Type of Work Performed by Technicians by Assigned Survey Module ............................ 37 Table 6-6: Electric and Gas Utility Service Territories in Which Technicians Performed Work ....... 38 Table 6-7: Types of Training Received by Technicians ............................................................................. 40 Table 6-8: Perceived Effectiveness of Training Categories in Which Technicians Participated ...... 41 Table 6-9: Technician Ratings of Training Effectiveness (Effective or Very Effective) by Assigned Survey Module ...................................................................................................................................................... 42 Table 6-10: Certifications and Licenses Held by Technicians .................................................................. 43 Table 6-11: Number of Certifications Held ..................................................................................................... 44 Table 6-12: Topics for Which Technicians Wish They Had Additional Training .................................. 45 Table 6-13: Percent of Technicians Reporting No Training Desired, by Experience Level .............. 46 Table 6-14: Number of Additional Training Topics Mentioned by Technicians ................................... 46 Table 6-15: Percent of Technicians Performing Specific Installation Tasks .......................................... 48 Table 6-16: Percent of Technicians Performing Specific Maintenance Tasks ..................................... 50 Table 6-17: Technicians’ Rankings of Top Priority Onsite ......................................................................... 52 Table 6-18: Barriers to Making a Necessary Repair Requiring an Additional 30 Minutes Onsite ... 54 Table 6-19: Codes or Standards Followed on a Typical Job .................................................................... 56 Table 6-20: Technicians’ Unprompted Definitions of the Term “Quality Installation” ....................... 57 Table 6-21: Technicians’ Unprompted Definitions of the Term “Quality Maintenance” .................... 59 Table 6-22: Awareness of ACCA Standard 5 by Technicians Who Do Installation Work ................ 60 iii CA HVAC C&T Behavior Study, Phase II Table 6-23: Portion of Installation Standard Followed on Typical Installation Job by Survey Module .................................................................................................................................................................... 60 Table 6-24: Do You Associate This Installation Standard with Any Utility-Sponsored Programs? .. 61 Table 6-25: Utility-Sponsored Program(s) with which Technicians Associate ACCA Standard 5 ... 61 Table 6-26: Awareness of Maintenance Standards by Technician Respondents Who Conduct Maintenance .......................................................................................................................................................... 62 Table 6-27: Portion of Installation Standard Followed on Typical Maintenance Job by Survey Module .................................................................................................................................................................... 63 Table 6-28: Do You Associate This Maintenance Standard with Any Utility-Sponsored Programs? .................................................................................................................................................................................. 63 Table 6-29: Utility-Sponsored Program(s) with which Technicians Associate ACCA 4, ACCA/ASHRAE 180 ............................................................................................................................................. 64 Table 6-30: How Technicians First Became Aware of Standards ........................................................... 65 Table 6-31: What Does Supervisor do to Ensure Proper Procedures are Followed in Field? .......... 66 Table 6-32: Percent of Technicians Responsible for Selling New HVAC Equipment ........................ 67 Table 6-33: Primary Barrier to Selling Installation Services Based on Standards ............................... 68 Table 6-34: Percent of Technicians Responsible for Selling Maintenance Agreements .................. 69 Table 6-35: Primary Barrier to Selling Maintenance Services .................................................................. 70 Table 6-36: Did Your Company Provide Any Training to You Regarding How To Sell or Recommend New Equipment Or Maintenance Contracts To Customers? .......................................... 70 Table 6-37: Effectiveness of Employer-Provided Sales Training .............................................................. 71 Table 6-38: Do You Think Sales / Additional Sales Training Would Be Helpful? ................................. 71 Table 7-1. Recommended Secondary Indicators for Quality Installation (SIQI) for MTIs HVAC-2 and HVAC-3 ................................................................................................................................................................... 83 Table 7-2. Recommended Secondary Indicators for QM (SIQM) for MTI HVAC-4 .............................. 85 LIST OF FIGURES Figure 2-1. Research Design ............................................................................................................................... 3 Figure 2-2: Location of Nonparticipating Contractors by Zip Code, Classified by IOU Territory ....... 7 Figure 6-1: Technicians' Ratings of Priorities Onsite During a Typical Job ............................................ 51 Figure 6-2: Technicians Primary, Secondary, and Tertiary Priorities Onsite (n = 216) ....................... 53 APPENDICES Appendix A: HVAC Contractor Interview Guide ......................................................................................... A-1 Appendix B: Technician Telephone Survey Instrument ........................................................................... B-1 Appendix C: Implementer & Program Staff Interview Guide ................................................................... C-1 Appendix D: Market Actor Interview Guide ................................................................................................. D-1 Appendix E: Comments and Revisions .......................................................................................................... E-1 Appendix F: Recommendations Table ........................................................................................................... F-1 iv E x e c u t i v e S u m m a r y For the California HVAC Contractor and Technician Behavior Study, Phase II, the research team explored implementation challenges faced by technicians, qualities of contractors’ business models that prevent or contribute to ideal field practices, and training/certification needs in order to fully transform the market. The study included multiple phases of data collection: (1) in-depth interviews with contrac- tors, (2) in-depth interviews with market actors, (3) in-depth interviews with implementer staff, and (4) a telephone survey with techni- cians. In addition, we sought to operationalize and determine the feasibility of measuring the five market transformation indicators (MTIs) indicated by the California Public Utilities Commission (CPUC). Key findings are presented below, grouped by the five research topics/ questions included in the study. Awareness and Use of Industry Standards While a sizable minority of technicians report they are aware However when prompted... of industry standards, in most cases, the standards do not appear to be “top-of-mind.” When asked in an unprompted 59+41 41% of technicians asked about installa- question what codes or standards they use, only 9% of tech- tion work indicated they were aware of nicians indicated that they use Quality Installation, Quality ACCA Standard 5 (prompted). Maintenance, or ACCA/ASHRAE standards. City and state codes appear to be much more salient, with 60% of techni- 66+34 34% of technicians asked about residen- cians mentioning city/municipal codes, and 51% mentioning tial maintenance work indicated they were state code, state building code, or Title 24. aware of ACCA Standard 4 (prompted). 91+9 51+49 Only 9% of technicians mentioned a QI/ 49% of technicians asked about commer- QM standard (unprompted) as one of the cial maintenance work indicated they were standards they follow on a typical job. aware of ACCA/ASHRAE 180 (prompted). Terminology: Standards and Program Names 97+3 Technicians generally do not associate the terms “quality Out of all technicians performing installation work, only 3% of tech- installation” and “quality maintenance” with the standards or with utility programs. When asked to define “quality main- nicians were both aware of ACCA tenance” or “quality installation,” technicians generally think 5 and correctly associated it with a of completing a job “the right way” or having a clean or neat specific utility program (unaided). system. Only 1% of respondents associated ACCA Standard 5 with “quality installation” and 1% of respondents associated utility programs with “quality installation.” None associated 98+2 Out of all technicians performing “quality maintenance” with the standards and less than 1% maintenance, only 2% were both associated it with utility programs. aware of either ACCA 4 or ACCA/ ? ASHRAE 180 and correctly associ- Specific ated it with a specific utility program QI/QM utility (unaided). Standards programs Quality Contractor Best Practices Quality contractors emphasize Quality contractors understand Quality contractors are focused customer service. internalize the value continuous improve- and on ment. of QI/QM. Firms already focused on superior cus- tomer service are able to incorporate Quality contractors are those who Quality contractors who success- QI/QM standards more easily because understand and buy in to the value fully implement QI/QM are those who the value proposition aligns with their proposition of QI/QM and have the undertake and emphasize continuous emphasis on service. ability to demonstrate the value of QI/ training, invest in their staff, are open QM to customers. to feedback, and perform QA/QC regularly. ES-1 E x e c u t i v e S u m m a r y Training A combination of classroom and in-field training is viewed as Most technicians play a role in sales and are interested in most effective by contractors and market actors. While on the sales training. job training was rated as most effective by technicians (with 11+89 89% of technicians who perform installa- 90% rating it as “very effective”), this type of training is likely extremely variable, as it is dependent on the knowledge, skills, tion work are responsible for selling new and efficacy of the trainer. HVAC equipment to customers. 15+85 85% of technicians who perform main- There appears to be a training need for technicians on the tenance work are responsible for selling topics of airflow, building science, and HVAC fundamentals. HVAC maintenance services to customers. However, nearly one third (32%) of technicians we surveyed indicated there were no topics on which they could use ad- 27+73 73% of technicians responsible for sales ditional training. are interested in sales training. Market Transformation Indicators A number of key terms will need to be clearly defined before Additionally, proximate (or leading) indicators may help mea- the MTIs can be operationalized or consistently measured. For sure any market transformation that takes place in the near example, in order to accurately estimate MTI HVAC-1a (market term. For example, proximate changes in awareness and under- share of climate appropriate HVAC equipment) and MTI HVAC- standing of the standards, which are necessary for implement- 1b (market share of climate appropriate energy efficient HVAC ing standards-based work, could be tracked in addition to the equipment), the term “climate appropriate” must be clearly longer-term market transformation indicators. defined and a list of qualifying unit types developed. Abbreviated Recommendations* * For full text recommendations, please refer to the Chapter 8 in the report. 1. Include guidelines for how to enact 4. Provide sales training to technicians. 7. Consider pairing Quality Mainte- the standards when training technicians This is important because they are nance contracts with financing of new and contractors on the specifics of often the first point of contact with the HVAC equipment purchases. Pairing the ACCA/ASHRAE standards. These customer. Sales training should teach a maintenance contract at the point of guidelines will provide a necessary contractors, technicians, and sales purchase, and including that contract in foundation for education and training of staff how to speak to customers about the financing, will help overcome cost contractors and technicians. sales in a language that customers will barriers and encourage proper mainte- understand. nance. 2. Conduct case studies with technicians to better understand how 5. Craft QI and QM training so 8. Educate customers about the the standards are currently enacted in training/certification requirements contractors and technicians are well- the broader marketplace. These case for trade allies to participate in the QI versed in the value proposition. Buy-in studies could involve either shadowing and QM programs. This will allow them from contractors and from technicians is technicians or conducting covert field to differentiate QI/QM services from necessary because if they do not believe observations. Understanding the standard practice. In turn, customers may in the value, it will be difficult at best to standard “baseline” practice will help begin to demand contractors/technicians convince customers. inform where to focus training efforts. with these credentials. 3. Develop a proactive branding 6. Provide tools such as case studies 9. Design and teach ways to build QA/ strategy. To do this, we recommend and data that contractors and techni- QC into contractors’ internal processes. that the IOUs develop several potential cians can use to demonstrate energy This will help contractors sustain quality branding strategies and test these with savings and reliability to customers. practices even after the programs end. contractors, technicians, and customers Communicating the value proposi- before adopting a strategy. We recom- tion will be much easier if those selling mend using program names that can be standards-based services can show differentiated from “good” installation or customers examples of how much other maintenance. similar customers have saved. ES-2 Chapter 1 INTRODUCTION 1. INTRODUCTION This chapter first summarizes the impetus for this research and provides an overview of the study. This is followed by a summary of the research questions. 1.1 Study Background and Overview Understanding how to improve the quality of technicians’ HVAC installation and maintenance practices in California is of the utmost importance. In June 2008, the California Energy Commission (CEC) estimated that improper installation and maintenance of cooling systems can result in a 20% to 30% increase in summertime peak electricity needed by such systems. As such, quality installation and quality maintenance have particular strategic importance as demonstrated in the California Long Term Energy Efficiency Strategic Plan (CLTEESP), which identifies one of the four HVAC goals as: "Quality Installation and maintenance becomes the industry and market norm. The marketplace understands and values the performance benefits of quality installation and maintenance."1 This HVAC Contractor & Technician Behavior Phase II Study builds upon the work the EMI Consulting Team completed in 2012 for the California HVAC Contractor & Technician Behavior Phase I Study, which focused on contractors and explored business models, maintenance sales practices, and understanding of HVAC maintenance and industry standards.2 The Phase II study also builds on the CQM Rapid Feedback Process Evaluation, which identified procedural issues with contractor and technician vetting, training, and quality control.3 Additionally, this study was conducted simultaneously with the HVAC Customer Decision-Making study, which focused on understanding how residential and commercial customers in California make purchasing decisions related to HVAC installation and maintenance.4 This Phase II Study, coupled with the HVAC Customer Decision-Making study and aforementioned past studies yields additional insights to moving California closer to the CLTEESP HVAC Big Bold Energy Efficiency Strategy: “Heating, Ventilation and Air Conditioning (HVAC) will be transformed to ensure that its energy performance is optimal for California‘s climate.”5 This research characterizes the opportunities and challenges that HVAC contractors and technicians face in implementing quality installation (QI) and quality maintenance (QM) in line with industry standards that were not addressed in prior studies. For the purposes of this study, the term "quality Installation" and its initials "QI" refer to ACCA 1 California Public Utilities Commission. (2011). The California Energy Efficiency Strategic Plan. Available at: http://www.cpuc.ca.gov/PUC/energy/Energy+Efficiency/eesp/. 2 EMI Consulting. (2012). California HVAC Contractor and Technician Behavior Study. CALMAC Study ID SCE0323.01. Available at http://www.calmac.org/publications/ca_hvac_behavior_study_finalreport_2012sept14_final.pdf 3 EMI Consulting. (2014). Southern California Edison HVAC Quality Maintenance Program Rapid Feedback Process Evaluation. CALMAC Study ID SCE0344.01. Available at http://www.calmac.org/publications/EMI_SCE_Rapid_Feedback_CQM_Report_Final_2_3_14.pdf 4 EMI Consulting. (Forthcoming). California HVAC Quality Installation/Quality Maintenance Customer Decision-Making Study. 5 California Public Utilities Commission. (2011). The California Energy Efficiency Strategic Plan. Available at: http://www.cpuc.ca.gov/PUC/energy/Energy+Efficiency/eesp/. 1 CA HVAC C&T Behavior Study, Phase II Standard 5. "Quality maintenance" and its initials "QM" refer to ACCA Standard 4 (Residential), and ASHRAE/ACCA Standard 180-2008 (Commercial). The CLTEESP emphasized these industry standards because they are comprehensive in their specifications, whereas the HVAC quality control requirements set forth by Title 24 are optional.6 Thus, the investor owned utilities’ (IOUs’) QI and QM programs are based on these standards. 1.2 Research Questions For this project, the research team explored implementation challenges faced by technicians, qualities of contractors' business models that prevent or contribute to ideal field practices, and training/certification needs to fully transform the market. In addition, we sought to operationalize and determine the feasibility of measuring the five market transformation indicators (MTIs) indicated by the California IOUs and the California Public Utilities Commission (CPUC). The key research questions this study addressed are grouped by topic below:7 • Awareness and Use of Industry Standards o What is the level of technicians' awareness and use of industry standards? o How are industry standards communicated to technicians and how do contractors and implementers follow up with technicians to ensure that standards are enacted in the field? • Terminology: Standards and Program Names o Are “quality installation” and “quality maintenance” terms that technicians can readily differentiate from “good” maintenance, as it is regularly understood? o What is the level of fluency with which the IOU program names can be linked with standards-based practices? • Quality Contractor Best Practices o What does a quality contractor look like? o What are the best practices for organizing a QM and QI HVAC business? o What are the best practices for ensuring that technicians are delivering QM and QI in the field? • Training o What is the role and efficacy of various types of technician training and certification? o What are the key training needs that exist with respect to technician practices, such as test instrument selection and calibration, protocols, and tasks performed in the field? o What are the best practices in sales training processes/approaches specific to HVAC services? • HVAC Market Transformation Indicators (MTIs) o How can the five HVAC MTIs be operationalized for future data collection? 6 California Public Utilities Commission. (2011). The California Energy Efficiency Strategic Plan, p. 55. 7 Originally, there was one additional research question, but SCE and the project team agreed to remove this question because other ongoing studies were addressing this question in detail, and also to decrease respondent burden inherent with such a large number of research questions. The deleted question was: "What is the rate of Title 24 Code compliance as demonstrated by pulling permits, having Home Energy Rating System (HERS) inspections, and making final the permit?" 2