Preparing for Your Next ALM Exam or Audit Review, documentation and model testing to prepare for an examination of model risk by: Michael R. Guglielmo, Managing Director Darling Consulting Group Ongoing validation of the asset/liability management (ALM) model and modeling process is an ever-growing challenge as the complexity of bank balance sheets continues to expand and the systems used to measure and manage risk become more sophisticated. Given the significance accurate risk measurement plays in support of an effective ALCO (asset/liability management committee) decision-making process, and the related regulatory, accounting and compliance (Sarbanes-Oxley Act of 2002—SOX) implications, managing model risk has never been more important. This article addresses the recent trends in balance-sheet management, the evolution of the tools used to measure and manage risk, regulatory insights and what processes and controls institutions are implementing to ensure that model risk is adequately quantified, understood and managed. Darling Consulting Group 978.463.0400 Page 1 What Is Model Risk? In the context of ALM, model risk is simply the risk of not accurately estimating the interest- rate-risk sensitivity of an institution (earnings and value). In our many years of providing independent model validations, we have found the most common sources of model risk are (1) wrong assumptions, (2) bad or insufficient source data, (3) inadequate model structure and setup, (4) lack of adequate processes and controls. With the advent of more complex products being offered by banks today and the multifaceted elements being incorporated into the ALM modeling systems to reflect these products, model risk is an ever-growing concern for many in the industry. Increased Balance-Sheet Complexity and Optionality Within the past five years, there has been a proliferation of new and more complex financial products and instruments that banks have been incorporating into their balance sheets. All of these products have some form of risk associated with them. Some are very straightforward and easy to model; others can challenge the most seasoned veterans of ALM modeling. In addition to Wall Street developing more complex instruments for banks to invest, there has been a notable growth in more sophisticated loan products with embedded options that make it increasingly more difficult to model and validate using traditional methods, tools and resources. Examples of such products include hybrid adjustable-rate mortgages (ARMs, for which there is an initial fixed/lockout period before theoretically becoming an adjustable-rate loan); variable- payment loans (the loan’s payment characteristics change over time); and home equity loans and lines with multifaceted fixed lockout periods and prepayment penalty structures that can greatly affect repayment activities. Due to the lack of historical data on the behavioral aspects of these products, they are often a contributing factor to an ALM model’s variance despite many back- testing processes. The commercial lending arena also has its share of more complex products brought about by increased competition and a desire to build whatever will “win the deal.” These products now include multifaceted prepayment penalty structures and characteristics that can change over time Darling Consulting Group 978.463.0400 Page 2 (for example, cash flow/repayment schedules, varying repricing elements such as index, spread, caps and floors). Competition and accounting pronouncements such as Statement of Financial Accounting Standards No. 133 (FAS-133, “Accounting for Derivative Instruments and Hedging Activities”) have had Wall Street and others such as the Federal Home Loan Bank system developing more complex funding structures that banks can use to manage earnings and interest-rate risk. It seems like there is a new product released every month with some creative name or mnemonic associated with it (knockouts, flippers, etc.). These structures can include a variety of embedded options, some of which can even be related to a notional amount higher than the actual amount of the funding (up to two times) without being subject to FAS-133 accounting issues. Many of these products can be very beneficial to banks and their interest-rate-risk-management process, but additional modeling capabilities and efforts are required. Even traditional funding sources have grown in complexity as banks compete for what may be a limited supply of core deposits. Advances in technology, online competition, liquidity challenges and creative pricing schemes designed to attract and retain core deposits have generated new products and bred different behaviors in depositors. Ads for products are full of fine print, penalties and options that make financial modeling that much more challenging. All told, these changes to the industry and the products on the balance sheet are greatly affecting the ALM modeler’s job to build accurate representations of the institution’s risk profile and to validate the results. Identifying and managing model risk is quickly becoming as important as the model results as ALCOs struggle to remain confident in their risk-measurement processes. Increased Sophistication of ALM Systems Many companies that develop ALM modeling software have been busy enhancing the tools used by banks for measuring and managing risk. The systems used by larger and more complex institutions have, by and large, been keeping pace with the need for more robust modeling capabilities, with deterministic models giving way to more sophisticated stochastic modeling processes. These increased capabilities, however, come at a significant cost in terms of dollars, Darling Consulting Group 978.463.0400 Page 3 resources and time. In addition to increased software fees and hardware costs (with multiserver configurations often needed to handle the millions of calculations being performed at the transaction level), additional staffing, perpetual training and more robust processes and controls all are required. In many instances, third-party tools or analytics are also required (at an additional cost) in order to take full advantage of the features. Elements such as multipath interest-rate generators, automated investment cash flow processors, multicomponent prepayment and other behavioral models are becoming more prevalent, putting additional pressure on model managers to know, understand and validate all of these moving parts. Systems used by community banks are also rising to the occasion: Instrument-level processing capabilities are now available in most of the systems used for in-house ALM modeling or by outsourcing providers (although many have yet to adopt these capabilities). In addition to eliminating the potential skewing of results when option-based characteristics (that is, embedded caps and floors) are averaged together, this modeling method can automate the processing of more complex structures (for example, callable/putable securities, deposits and borrowings). In addition, robust third-party tools and analytics can be integrated, whereby more complex elements can be modeled (collateralized mortgage obligations [CMO], loan-level prepayments) in a fairly automated way. But the costs associated with these capabilities are not insignificant in terms of dollars, resources and time—all scarce commodities for many community banks today. Regulators’ View of the Risk/Reward Trade-Off Examiners and compliance experts recognize that the ALM modeling process is growing in complexity and that the tools required are a function of the level of optionality embedded in a given balance sheet. There is not always a clear appreciation for the risk/return trade-offs among accuracy, effectiveness and cost, however. The bottom line is that examiners (and compliance experts) want to be confident that the information used by an institution is reliable for managing the type and quantity of risk embedded in the balance sheet. In general, the focus has been on the following:  Appropriateness of the measurement tools (given the nature and complexity of the balance sheet) for both short- and -longer-term risk Darling Consulting Group 978.463.0400 Page 4  Accuracy of results (through independent model validation and back-testing)  Validity of the key modeling assumptions (optionality, prepayment, deposit sensitivity, growth, scenarios)  Ongoing reliability (sufficiency and adherence to processes and controls) Much of the focus in recent exams and audits has been on evaluating the key assumptions that go into the ALM model, as well as evaluating the processes and controls in place to minimize the potential for making bad decisions based upon bad information. “Model governance” is the term most recently used by regulators. It is believed (and rightly so) that with properly designed procedures that are implemented and followed, models stand the best chance of achieving their intended purpose. In terms of supervisory review, the suggested framework is broken down into three parts: model oversight, model controls and model validation. In terms of model oversight, examiners will evaluate the governance policies established by management and the board to determine if they are adequate relative to the use of the model and if the procedures used to comply with the policies are followed. In evaluating model controls, examiners will review model documentation, particularly limitations and operating procedures. In addition to a review of the data reconciliation procedures, they are looking for participation by the various business lines in terms of developing assumptions and verifying results. As part of this evaluation, they will also review the security and change control procedures. The level of expectation generally varies upon the complexity of the modeling process. Some level of security control is warranted; limiting use to authorized users only and backup procedures that ensure an institution can recover quickly should a technological disruption occur. More formal change control procedures can be expected for larger institutions using more complex systems. In this regard, procedures that ensure that all changes are justified and properly approved are instituted. Whether your ALM modeling process is in-house or outsourced, it is important that you understand the process that goes into building the model, developing the assumptions and verifying the accuracy of the results. Not all model processes (whether in-house or outsourced) Darling Consulting Group 978.463.0400 Page 5 produce reliable results, and examiners and auditors need to be convinced that your bank is doing what is necessary to ensure that your process is one that is reliable. As an outsource provider of ALM modeling and balance-sheet advisory, DCG participates and is subject to hundreds of exams and audits on behalf of our clients each year. In addition, we evaluate and validate many bank processes each year—and each year, expectations increase and the standards are raised. Effective ALM modeling requires ongoing fine-tuning as balance sheets grow in complexity and advances in technology provide more information and tools to support the process. You can do a number of things to be prepared for your next exam or audit. All it takes is some reading, ‘riting and ’rithmetic. Reading: Review Your Past Exam Preparing for the next ALM exam or audit starts with knowing what is expected of you and your institution. A good starting point is to review the findings from your previous exam or ALM audit. This is one of the first places we look when we perform independent reviews, as it provides valuable insight into an institution’s risk-management process—and where on the priority list this process resides. If any recommendations were made, or your institution agreed to make specific improvements, be sure that the recommendations were considered and that the agreed-upon changes were implemented. An effective ALM process requires ongoing care and feeding to meet the challenges posed by the ever-increasing complexity and volatility of the banking business and, most likely, your balance sheet. A process that does not evolve over time can lead to risks being overlooked or, even more important, bad and costly decisions. Review your ALM-related internal processes and controls documentation to be certain that it reflects your current ALM process. This procedural documentation serves three primary purposes. First, it establishes guidelines and expectations for those responsible for building the model. Second, it provides a “corporate memory,” allowing employees new to the modeling process to more quickly participate. Third, this documentation provides a means to articulate the underlying assumptions used in the modeling process, particularly those that do not change regularly. This is useful not only to the model manager but also to those who are auditing the process and to those who are using the model output for decision-making purposes (ALCO). Too Darling Consulting Group 978.463.0400 Page 6 often, we find this documentation to be outdated or incomplete, putting the institution at risk should the model manager leave without providing guidance to a successor. We’ve encountered numerous instances where model results were significantly off due to a poorly documented model that was “inherited,” and the original processes couldn’t be maintained. At a minimum, this procedural documentation should include the following:  A detailed list of the data sources and instructions as to how data is obtained and modified prior to use  Detailed instructions for data import into the model and techniques unique to the institution’s model and modeling process  Reconciliation and other internal control procedures  Explanation of how options are evaluated  Sources for behavioral assumptions (prepayments, core deposits) used in the model  Key deposit pricing assumptions  Internal validation and back-testing procedures  Model and system maintenance and backup/restoration procedures Have your internal auditors obtain and review the most recent copy of your ALM system vendor’s internal processes and controls. If you outsource the ALM modeling function, as many of DCG’s clients do, have them obtain and review the most recent copy of your provider’s processes and controls along with documentation on the software they are using. While these vendors are not required to have a Statement of Auditing Standards No. 70 (SAS-70, “Service Organizations”) audit performed, they should be able to provide you with documentation that explains the processes and controls applied to their process (model preparation, data and results validation, security and software development and testing, if applicable). Be certain that you understand the processes and tools used by the vendor(s) to ensure you receive accurate and reliable results. In the case of outsourcing, examiners and auditors want to make sure the process is accurate, independent and sufficient for your balance sheet’s complexity and that you have not abdicated your responsibility to internally verify and validate the results. In addition, they are looking for evidence that you are actively involved in developing and managing assumptions. The bottom line is that the model and process should not be a black box upon which ALCO Darling Consulting Group 978.463.0400 Page 7 relies for decision making. Unfortunately, with all of the additional tools being developed and implemented today, this is becoming more and more challenging. Read your policies! Confirm that they still reflect your institution’s current operating philosophies. They will also provide a good refresher for you so that when questions arise, you are prepared to respond. Also, if you have not read the regulatory guidelines to ALM model validation, the OCC’s (Office of the Comptroller of the Currency’s) Bulletin 2000-161 is a good place to start. This nine-page bulletin will provide you with a valuable perspective on the regulators’ expectations, including the accurate use of data, assumption validation and reporting. In addition, the Federal Deposit Insurance Corporation (FDIC) published an article on model governance in late 2005 that expands upon elements contained in the Joint Policy on Interest Rate Risk (SR96-13).2 Writing: Document, Document, Document A well-documented ALM process can have a notable impact on your examination or audit results. The best processes can appear deficient if they are poorly communicated to those outside ALCO. In addition, an otherwise adequate process can be rendered inadequate if written communication within ALCO is insufficient. As described earlier, your ALM modeling process and controls need to be documented and reviewed each year. This documentation should include descriptions of the sources of data used, the process of acquiring the data, verifying its accuracy, inputting the data into the model and validating the results. In addition, a description of key assumptions, including their sources and some form of periodic review and validation, is also an important element. Finally, backup and restoration procedures should be included in this documentation. While not critical to the survival of your institution, loss of a model can be costly in terms of time—not to mention the potential opportunity forgone because proper analysis couldn’t be performed when needed. 1 http://ithandbook.ffiec.gov/media/resources/3676/occ-bl2000-16_risk_model_validation.pdf 2 www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/article01_model_governance.html Darling Consulting Group 978.463.0400 Page 8 Outsourcing doesn’t excuse you from having to provide this documentation to your examiner or auditor; such documentation should be obtained and reviewed annually. Other than through the interview process, the effectiveness of ALCO’s decision-making process can only be gleaned from your ALCO meeting minutes and any documentation you have for potential and implemented strategies. Be certain that your minutes describe ALCO’s discussions regarding your institution’s current position, including liquidity, interest-rate risk and capital adequacy and their relation to your policy’s guidelines or limits. In addition, confirm that discussions of both potential strategies and the status of those previously implemented are well documented. When decisions are made, record who is responsible for their execution and for providing a status report at the next meeting. These straightforward elements can go a long way to providing all the stakeholders, including ALCO, the board, auditors and your examiner, with confidence in ALCO’s decision-making effectiveness. It is easy for those outside of the bank’s ALCO process to be Monday-morning quarterbacks and question certain decisions that have been made over the course of a year. If the documentation establishes the context in which decisions were made (that is, the risks present in the balance sheet at the time and the risk/rewards of taking action vs. inaction), your ALCO’s success as decision makers will be more effectively conveyed. Your polices should be updated to reflect any changes to your institution’s operating philosophies, risk-measurement/monitoring processes and any new regulatory guidelines or accounting pronouncements. In addition, if there are any new structured products (on or off balance sheet) the bank may use to manage risk, include language that describes the product, potential uses, benefits and risks. We often tell clients to treat their policy statements like internal “playbooks” that, once read by a stakeholder, clearly convey management’s understanding of the risk/reward trade-offs of its activities. Many of our clients begin their exams by requesting their examiners read their policies first. In this way, they inform and in some cases educate their examiner and reduce the questions and challenges that would otherwise occur. Darling Consulting Group 978.463.0400 Page 9 Arithmetic: Verify Results to Manage Model Risk Accuracy, reliability and timeliness are the cornerstones of an effective ALM process. Without these elements, an ALCO’s effectiveness as a decision-making body is not possible. While technology has strengthened our ability to measure and monitor risk, the industry continues to introduce more complexity and optionality into bank balance sheets. As a result, more assumptions are required for our models, increasing the potential for model risk. Potential model risk is a legitimate concern of the regulators (and of practitioners as well), and banks need to have processes in place to verify results and substantiate the assumptions that are used as part of the ALM modeling process. Verifying results should begin with an ongoing back-testing process that periodically (at least quarterly) tracks a model’s forecasted results to current interest income and expense levels. The effort expended and the level of detail analyzed depends upon an institution’s balance-sheet complexity and the extent of the historical variances. When notable variances exist, a more detailed analysis of the underlying assumptions may be required to provide explanations for the differences. At a minimum, we encourage our clients to use a 90-day look-back whereby variances in interest income and expense are compared for each of the major portfolios. Adding traditional rate/volume/mix style analysis can provide some insight into the potential sources of model risk and help a model manager to prioritize model enhancement efforts. Given the effect that faulty assumptions can have on projected earnings and risk, the old cliché “close enough for government work” no longer resides in the world of ALM. In terms of validating assumptions, experience and gut instinct now need to be empirically supported with historical evidence. This has encouraged a whole new generation of ALM support services, since many institutions simply can’t obtain and use the necessary information alone or in a cost- effective manner. DCG has focused over the past few years on developing additional tools and services to capture, track and validate key assumptions. ALM-specific data warehousing tools that can continuously validate inputs (that is, data, prepayments, replacement assumptions, deposit sensitivity), more robust investment analytics capabilities and core deposit study services are all contributing to more informed clients and better development of assumptions. Darling Consulting Group 978.463.0400 Page 10