Fannie Mae

Number of people on the test team: 150+

Number of developers supported by the test team: 400+

Best Practice Description : Two years ago, a major Fortune 500 financial institution was required by the SEC to restate its Financial Statements. This effort has been described as one of the largest financial restatement efforts ever undertaken in US history, taking several years and hundreds of millions of dollars. Hundreds of staff and consultants, external regulators, internal and external auditors were among the key stakeholders in this monumental task.

One of the significant challenges involved in this effort was timely and accurate Systems Testing. Given the focus on accuracy, the testing needed to deliver 100% business requirements coverage of all historical data and corresponding results. Of the 30+ major financial systems associated with this effort, the majority applied highly-complex accounting and business rules to several million varied and complex data records which represented the investment holdings of this institution.

To aid in this effort, the System Test team worked towards these high-level goals:

• Maximizing Flexibility to Rapidly Absorb Change
• Minimizing Test Cycle Time
• Maximizing Test Coverage
• Maximizing Resource Utilization

Best Practice 1 : Decompose the application under test

The projects that comprised this effort were broken down into phases with each phase containing multiple builds. Business requirements and code changed with each delivery, which typically causes significant test preparation and re-planning. The test team determined that automated test case creation would be critical to ensuring flexibility and time-savings in test planning

Best Practice 2 : Standardize requirements into code readable format Best

Best Practice 3 : Automate test case creation from requirements

Test case automation was accomplished using two methods – scripting cases from requirements and generating test cases from predefined application validation points. The test team worked upfront with the Requirements teams to standardize both the structure and content of the business requirements. As a result, the test team was able to write automated scripts that parsed each requirement and generated a suite of test cases that was then imported into Quality Center (QC). In one instance, an accounting policy change necessitated ~200 rules changes in one requirements version. The test team was able to absorb those changes and automatically re-generate revised test cases in less than 10 minutes! Manually, a change of this magnitude might have taken a team of four resources more than three days – over 12 person days.

For the purposes of testing, the application was carefully decomposed into intermediate logical validation points – output files, interfaces, database tables, reports, transaction queues, etc. This decomposition allowed for independent testing of components once the application arrived into the test environment. As a result, defects were pinpointed to the specific processing stage where they occurred. This made for quick resolution by developers as compared to testing approaches that focus on end-of-processing results – giving little guidance on where the defect occurred. Additionally, this strategy prepared the test team to receive incremental individual component builds into the test environment as components were completed (testing early and often) versus waiting for all components to be delivered -- greatly reducing the overall test timeline. A generic test case creation framework was built to automatically convert each physical point and data element of the application into atomic test cases. We estimated that by automating test case preparation for constantly changing requirements and rapidly changing code we reduced test preparation time by more than 75%.

Best Practice 4 : Generate useful metrics automatically from QC

An important ancillary benefit of the granularity in test case definition was the accuracy and consistency of application health metrics within, and across, projects. Health and progress metrics are generally used to convey progress relative to plan, as well as gauge overall quality of the application under test. The usefulness and reliability of these metrics -- generated from granular test cases – far exceeds that of larger, more comprehensive test cases.

Best Practice 5 : Profile production data, evaluate business requirements, and establish test data

Once the test cases were stored in QC, expected results were determined for each validation point. Initially, a representative “control” data set was generated that considered input source data combinations by profiling the production data to determine ranges of attribute values, data relationships, etc. Once the control set was defined, testers further analyzed the requirements to determine what additional coverage was necessary – requirements/scenarios not covered by production data, boundary conditions, negative tests, etc. As a result, we achieved 100% business requirements coverage with a comprehensive test data set. From this data set, expected results were manually calculated for each test case to ensure results accuracy, per the requirements. Since there was a requirement to create expected results for varying volumes of production data, the team leveraged the concept of a parallel Test Oracle (TO) to programmatically generate the results, per the business requirements.

Best Practice 6 : Generate, evaluate, and update test results programmatically

A “Test Oracle” is a piece(s) of software that automatically predicts expected results based on a set of business requirements. It provides test teams expected results for any possible input value. The TO can be used with many different data sets - one month, one year, or many years worth of data to verify both the performance of the application and its accuracy.

As compared to manual tests, far more powerful automated tests are built using Test Oracles, which can be used for expected results generation for test/control data as well as large volume production data from which one cannot easily, if at all, manually determine expected results. The TO was created with the same internal breakdown as the previously-generated test cases and provided corresponding results for each validation point. These results were then validated utilizing the control data as input and the manually calculated results as the answer key. Validating the TO against the manual expected results ensured the accuracy of the TO implementation and indicated the TO's readiness and reliability for use as a tool in test.

Having the robust capability to generate expected results for many fields and validation points on full volume data is made effective only by implementing an Automated Execution Framework (AEF) to allow quick, detailed evaluation, persistence, and reporting of pass/fail results. The results produced by the application under test and the TO results were deposited into the generic AEF which then executed automatic comparisons for each test case, uploaded the results, and updated the disposition of each test (pass/fail) in QC. The seamless integration with QC allowed for accurate and real-time metrics to be reported throughout the test cycle. Automating the test execution of both regression and functional testing for one project allowed over 6000 test cases to be executed and evaluated by a team of four in less than one day. This level of coverage and test granularity simply could not have been achieved manually.

Having automated test preparation and execution, the testing workload could be optimally distributed and executed in parallel amongst N resources. The execution time saved by automation allowed the test team to focus solely on the analysis of test case failures – where the actual and expected results differed. The test team's ability to only analyze and concentrate on failures resulted in earlier discovery of defects in test. Additionally, since the test cases were atomic and granular (allowing focus on business requirements in conjunction with technical analysis of the application in test), and the test team could direct effort to focus on the failed test cases, the usefulness of the defect reports increased dramatically. High-quality defect reports, in turn, promoted faster, more accurate resolution from the Development teams.

Organizationally, we determined that a Center of Excellence (COE) approach would be the most effective way to implement the best practices highlighted in this paper. At the core of the COE is a set of four horizontal teams which supply test products and services as follows: test data strategy, test case automation, Test Oracle development, and test architecture (test design, methodology, etc.). These teams are involved upfront, as requirements are written and work through each lifecycle phase in conjunction with a set of test execution teams – one per application under test. The COE promotes standardization, consistency, and the following additional benefits across the test organization:

• Consolidates and front-loads highly-skilled resources to initialize each project; resulting in the best test strategy possible and providing challenge to top talent
• Allows for deploying less expensive, more junior resource for the less complex task of test execution
• Creates a pool of resources who can be redeployed/redistributed to meet critical or time-to-market demands
• Allows for more flexibility for outsourcing/off shoring test execution functions which require minimal institutional knowledge

In conclusion, the System Test organization at this major Fortune 500 financial institution was able to quickly and completely revamp and implement its test methodologies and organizational structure to be poised to aggressively tackle the challenges presented by the company's mission-critical, Financial Restatement undertaking. The methodologies and processes reviewed in this paper have been utilized and refined over the last two years on tens of projects with great success and overwhelming positive impact to quality, as well as monumental time and resource savings.