Office of the Inspector General
Jo Anne B. Barnhart
Commissioner

Inspector General

Performance Measure Review: Reliability of the Data Used to Measure Anti-Fraud Performance (A-02-01-11013)

We recently completed our first 3-year cycle of reviewing the Social Security Administration's performance measures. In conducting this work, we used the services of an outside contractor, PricewaterhouseCoopers, LLP, (PwC) to assist us in our efforts.

For this report, we used PwC to conduct the review of four of the Agency’s performance indicators related to anti-fraud activities. The objective of the review was to assess the reliability of the data used to measure the Agency’s anti-fraud efforts. These anti-fraud indicators measure the results of the work conducted by the Office of Investigations within the OIG. To ensure an independent review of these indicators, we relied on our contractor to assess the reliability of the data behind these measures.

The attached final report presents the results of the contractor’s work and its recommendations for improvement. We are generally in agreement with all of the contractor’s recommendations focusing on OIG operations. My office will track the individual recommendations and we will periodically provide our status in implementing these recommendations. We plan to work closely with the Agency on those recommendations where coordination would be appropriate. If you wish to discuss the final report, please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

PERFORMANCE MEASURE REVIEW:

This report is one of five separate stand-alone reports, corresponding to the following Social Security Administration (SSA) process and performance measures (PM):

• Number of investigations conducted (i.e., closed) (PM #5)

• Old-Age, Survivors and Disability Insurance (OASDI) dollar amounts reported from investigative activities (PM #6)

• Supplemental Security Income (SSI) dollar amounts reported from investigative activities (PM #7)

• Number of criminal convictions (PM # 8)

This report reflects our understanding and evaluation of the process related to PMs #5 through #8. To achieve its strategic goal "To make SSA program management the best-in-business, with zero tolerance for fraud and abuse," SSA has developed several strategic objectives. One of these objectives is "To aggressively deter, identify, and resolve fraud." SSA’s FY 2001 Annual Performance Plan (APP) contains four performance indicators developed to meet this objective as follows:

• Number of investigations conducted (i.e., closed) - Prior to FY 2000, this goal was based on cases opened. The FY 2000 goal was revised upward from 7,200 to 7,600 investigations conducted (i.e., closed) in SSA’s FY 2000 Revised Final Performance Plan; this reflects the benefits of increased resources the Office of the Inspector General (OIG) devoted to investigative activities.

• OASDI dollar amounts reported from investigative activities - The FY 2000 goal was revised upward from $9 million to $40 million in SSA’s FY 2000 Revised Final Performance Plan; this reflects an anticipated return on investment from investigative activities as described above.

• SSI dollar amounts reported from investigative activities - The FY 2000 goal was revised upward from $55 million to $80 million in SSA’s FY 2000 Revised Final Performance Plan to reflect an anticipated return on investment from investigative activities as described above.

• Number of criminal convictions - The FY 2000 goal was 1,800 convictions.

We performed our testing from September 21, 2000 through February 15, 2001. Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland and the OIG office in Philadelphia, Pennsylvania. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.

SSA has been engaged in an aggressive program to deter, detect, investigate and prosecute fraud. To carry out this effort, SSA and the OIG have cooperated in developing a comprehensive anti-fraud plan. The SSA OIG, Office of Investigations (OI) is responsible for investigating allegations of fraud, waste and abuse. The four indicators evaluated focus on the OIG’s output efforts to achieve improvements in deterring, identifying and resolving fraud.

The second and third performance measures, "OASDI dollar amounts reported from investigate activities" and "SSI dollar amounts reported from investigative activities," represent amounts reported by the OI from fines/penalties, assessments, savings, recoveries and restitution/judgments related to investigative activities. The components of investigative activities, or "monetary achievement" are defined as follows:

• Fines/penalties are court-ordered, and include any special assessment fees, imposed upon conviction in a criminal case or judgment in a civil case, which require a specified sum of money be paid to the court.
• Program savings is a calculation of the avoidance of actual dollar loss by actions that result in the termination of improper payments of program funds, which only can relate to SSA cases.
• Restitution can only be recorded in a criminal case, in which the court-ordered repayment that resulted from pretrial diversions and convictions. The amount of restitution may be categorized as SSA or non-SSA program amounts. In addition, the amount of the restitution claimed by OI should match the amount of restitution documented on the Judgment and Commitment Order (J&C) or Pretrial Diversion agreement.
• A judgment is a judicially ordered payment resulting from a civil action, either through Department of Justice civil proceedings or the Civil Monetary Penalty Program, which can be categorized as SSA or non-SSA program related. The distinguishing characteristic of a judgment is its nexus to a civil action, as opposed to a criminal restitution.
• A scheduled recovery is a total sum of non-court ordered repayment of funds to which an individual was not entitled, or the total of funds to be returned because the individual was not entitled, which can be categorized as SSA or non-SSA program related. In those SSA program cases in which the court does not order restitution because of a defendant’s inability to pay, OI will provide a copy of the J&C to SSA, accompanied by a memorandum requesting SSA to recover the program losses. SSA’s initiation of withholding future payments will constitute a scheduled recovery.

The objective is to report the dollar values of the OI efforts/activities to identify and resolve fraud, which is directly related to the strategic goal "To aggressively deter, identify and resolve fraud."

The last performance measure, "Number of Criminal Convictions," represents the number of criminal convictions as a result of OI activities. This performance measure is presented as a workload count of all cases concluding in a criminal conviction. In addition to purposes served by formally charging a person with the commission of a crime, the criminal prosecution process has an impact, which may deter others from committing violations, and therefore is directly related to the strategic goal "To aggressively deter, identify and resolve fraud."

Information concerning potential wrongdoing involving SSA programs, employees, or operations which are received by an SSA OIG component are accounted for in ACIS. The system encompasses the initial receipt of an allegation, all steps taken throughout the investigative process, and the final outcome of all investigations. The system resides in a database (ADABAS) application, at the Center for Information Technology (CIT) of the National Institutes of Health (NIH).

The ACIS process begins with the receipt of an allegation. The Allegation Management Division (AMD) staff records these allegations in the allegation management module of ACIS. They are then electronically forwarded to either an OI field division (FD), a SSA field component, another agency, or are closed. Allegations may be received from Congress or from the public via mail, e-mail, fax, Internet, or telephone (i.e., 800 Hotline). FDs may also receive allegations directly, and are also required to enter the allegation information into ACIS. If an investigation is warranted, all information and supporting documentation will be forwarded to the appropriate individuals within the FD.

When an investigation is opened, ACIS generates an OI-1, ACIS Case Opening Report. To complete and document an investigation, the Special Agent (SA) is required to complete a series of forms. Once the case is taken to court, if the court orders a restitution or judgment, the agent completes the OI-68, Report of Court Ordered Restitution/Judgment. The OI-67, Monetary Achievement Worksheet, may also accompany an OI-68. The OI-67 is the form used to document any civil monies recovered during the investigation, as well as to assist in calculating the monetary achievement documented on the OI-68. Once all aspects of the investigation are completed the SA completes an OI-9, ACIS Criminal and Administrative Disposition Form. This form includes the judicial and criminal disposition data, as well as criminal and administrative monetary achievements, and is used by the administrative personnel, SAs, Assistant Special Agents-in-Charge (ASAC) or Special Agents-in-Charge (SAC) to input the monetary achievements into ACIS.

The OI-68 is mailed to OI Manpower and Administration Division (MAD) in Woodlawn, Maryland. It is then reviewed against the data, which is entered into ACIS for accuracy and distributed to SSA’s Debt Management Section for notification of the court-ordered restitution. In addition, the OI-68 and the payments are sent to the Mid-Atlantic Program Service Center (MATPSC) to be processed. The MATPSC cross-references the checks to the OI-68 and posts them to the correct Social Security numbers (SSN).

During the period of September 21, 2000 to February 15, 2001, we evaluated the current processes, systems and controls, which support the FY 2000 SSA performance measurement process. In addition, we determined the accuracy of the underlying performance measure data. Our evaluation of the information provided by SSA management as of February 15, 2001, allowed us to determine that the reported FY 2000 results of the four performance measures tested (as itemized below) were reasonably stated based on the methodology used by SSA.

However, we noted certain deficiencies in SSA’s methodology used to analyze the performance measures and certain limitations with ACIS. Although we do not consider any of the deficiencies noted during our evaluation to be significant, consideration should be given to the following recommendations, as we believe there are opportunities for improvement, thereby increasing the value of the performance measures as management tools.

1. ACIS has some data integrity deficiencies.
2. The savings calculation is not adequately supported.
3. SSA OIG lacks sufficient documentation for several key control areas related to ACIS. These areas are change control policies and procedures, user access recertification procedures and systems development documentation.
4. SSA OIG lacks appropriate supervisory review in the change control process.
5. OASDI and SSI dollar amounts from investigations may be over or understated.
   

These items were noted as a result of our testing the underlying performance measure data, as well as the Electronic Data Processing (EDP) and manual controls of the systems generating the performance measure data, and are discussed in detail below.

To ensure that the amounts reported in the SSA Government Performance and Results Act of 1993 (GPRA) section of the Performance and Accountability Report were reasonably accurate and reliable, we obtained a random sample of 45 OASDI, 45 SSI and 20 criminal conviction cases from the ACIS system and requested supporting documentation for each of the cases. In addition, we judgmentally selected 10 cases from the closed-case cabinet at the Philadelphia sub-office, traced them to ACIS and ensured the cases were properly documented.

The results of our evaluation of the cases are as follows:

• Overstatement of OASDI Monies from Investigative Activities

We noted that 5 of the 45 OASDI cases evaluated had double-counting errors totaling $235,911 for OASDI amounts reported from investigative activities. This resulted from dollar amounts in both the scheduled recovery and restitution fields in ACIS. Per the Special Agent Handbook (SAH), these two fields are mutually exclusive, and therefore a case should not have the same dollars entered in both fields. These two fields are then combined with savings, judgments, settlements, fines, assessments, and penalties to arrive at the OASDI monies from investigative activities.

• Policy Errors

In our evaluation of the 120 ACIS cases selected, we found 2 cases in which the procedures used to calculate savings were not consistent with those procedures outlined in the SAH. The first was a fugitive felon case, for which the amount of savings was calculated over 3 years. Per the SAH, Chapter 3, Section 3-75-C8 "Program savings for fugitive felon cases will be calculated on a 2-year projection." In the second case OI found an individual embezzling benefit checks from an entitled beneficiary. A savings amount was calculated for this case, even though in a similar case, savings were not calculated.

• Data Anomalies

Our evaluation of the cases also noted that 36 out of the 120 ACIS cases selected contained data anomalies. The data anomalies represent instances where ACIS data did not match the data reported in the supporting documentation (OI-68, OI-9, OI-4, or OI-1). For example, in several cases the fraud loss entered into ACIS was higher than the fraud loss reported in the supporting documentation. In some cases, the recovery or assessment included in the supporting documentation was not reported in ACIS. These errors could have been caused by either 1) a data entry error made by administrative personnel, SACs, ASACs or SAs, or 2) by documentation errors within the case file, made by SAs.

OI has in place a supervisory review of case files at the field division level. However, the supervisory review, Form OI-20, includes a review of the status of the case, not a review of the forms within the case file to ensure that the forms are completed appropriately. The SAH, Chapter 3, Section 3-60 B, states the following: "While the review of documents and evidence in the case file is important, the case review process should not be viewed as merely a review of documents. Rather it is an evaluation of the investigative progress and potential of the SA’s cases." In addition, the SAH does not include procedures to ensure that the data entered in ACIS by the OI field divisions is correct.

Although the OI MAD has a Quality Review process, this process is limited to a comparison of Form OI-68 "Report of Court-Ordered Restitution/Judgment" to the restitution report produced at MAD, and the restitution amounts reported in ACIS. In addition, since not all cases require an OI-68, not all cases are reviewed. The lack of supervisory review of the forms and limited review by OI Headquarters has led to errors going undetected, thus diminishing the accuracy of the number reported by OI as part of the GPRA section of SSA’s Performance and Accountability Report.

ACIS is an antiquated system that lacks sufficient capabilities to continue to meet OIG’s needs. The OIG stated that the system has been "frozen" and that it is in the process of evaluating commercial off the shelf software packages as an alternative system. These data integrity opportunities for improvement should be considered when selecting and implementing the new system.

3. SSA OIG lacks sufficient documentation for some control areas related to ACIS.

There is a lack of documentation for some ACIS processes in SSA OIG operations, as follows:

• ACIS Change Control: Process in which any modification requests to ACIS, and subsequent modifications of ACIS are formally requested, documented, tested, and implemented.

• ACIS User Access Recertification: Review of users’ access rights to ACIS, to ensure that their access matches their job responsibilities.

• ACIS System Design and Development: System design and development documentation includes addressing the entire life cycle of a system, (i.e., initiation, definition, system design, programming and training, evaluation and acceptance, and installation and operations). As such it would include, for example, the functional requirements, data requirements, system/sub-system requirements, and both user and technical manuals, etc.
  

Without formally documented processes, management cannot be assured that SSA OIG personnel understand all of the requirements for successful change control, user access recertifications, and systems design and development processes.

When processes are not documented, such as formal processes for change control, user access recertification, and ACIS system design and development, there is no accountability that the procedures were followed. When there is no accountability for the procedures associated with these processes, it becomes difficult to resolve any problems or issues. This lack of assurance may negatively impact SSA OI operations.

4. SSA OIG lacks appropriate supervisory review in the change control process.

Office of Management and Budget (OMB) Circular A-130, Appendix III, Security of Federal Automated Information Resources, states that agencies are required to establish controls to assure adequate security for all information processed, transmitted, or stored in Federal automated information systems. This appendix stresses the importance of management controls affecting individual users of information technology. The appendix states: "Technical and operational controls support management controls. To be effective, all must interrelate."

In addition, Federal Information Processing Standards (FIPS) Publication 73 entitled Guidelines for Security of Computer Applications, Section 6.3.4 highlights the increased risk associated with programmers having access to program code once an application is operational. This increases the possibility for unauthorized changes to existing code that could benefit the programmer without being detected.

5. OASDI and SSI dollar amounts from investigations may be over or understated

The program category field within ACIS is used to identify which program (OASDI or SSI) the fraud was committed against. The possibility exists that an investigation may involve fraud against both OASDI and SSI concurrently. While agents have the ability to enter in more than one program category per case into ACIS, the associated Monetary Statistics report only pulls the information from the first program category in each case. Therefore, in the case of concurrent fraud, agents are instructed to enter the dollar amounts into the program category where the greater amount of fraud occurred. This results in an overstatement of dollar amounts in the program category that the case was entered into, and an understatement of dollar amounts in the other program category.

Throughout our evaluation of the four performance measures, we noted the strong commitment of SSA's OIG staff to correctly implement GPRA. Our evaluation found that the FY 2000 results of the four performance measures tested were reasonably stated. However, our evaluation noted that: 1) ACIS has some data integrity deficiencies; 2) the savings calculation is not adequately supported; 3) SSA OIG lacks sufficient documentation for several key ACIS control areas; 4) SSA OIG lacks appropriate supervisory review in the change control process; and 5) OASDI and SSI dollar amounts from investigations may be over or understated. We recommend that the OIG take the following corrective actions:

To ensure that the OI offices across the country are following the procedures outlined in the SAH, that data entry and documentation errors do not go undetected, and to correct internal control issues found during our case testing we recommend that OI:

APPENDIX A – Scope and Methodology

APPENDIX B – Acronyms

APPENDIX C – Performance Measure Summary Sheets

APPENDIX D – Performance Measure Process Maps

APPENDIX E – Performance Measure Taxonomy

The Social Security Administration’s (SSA) Office of the Inspector General (OIG) contracted PricewaterhouseCoopers, LLP (PwC) to evaluate 11 SSA performance indicators identified in its Fiscal Year (FY) 2001 Annual Performance Plan (APP). This report reflects our understanding and evaluation of the process related to PMs #5 through #8. We performed our testing from September 21, 2000 through February 15, 2001. Since FY 2001 performance results were not yet available as of the date of our evaluation, we performed tests of the performance data and related internal controls surrounding the maintenance and reporting of the results for FY 2000. Specifically, we performed the following:

1. Obtained an understanding and reviewed the current Allegation and Case Investigative System (ACIS) data flows and processes;
2. Identified and tested critical controls (both electronic data processing (EDP) and manual) of ACIS;
3. Tested the accuracy of the underlying FY 2000 data for each of the specified performance measures;
4. Recalculated each specific FY 2000 measure to ascertain its mathematical accuracy;
5. Determined whether performance measures were meaningful and in compliance with the Government Performance and Results Act of 1993 (GPRA);
6. Evaluated the impact of any relevant findings from prior and current audits with respect to SSA's ability to meet performance measure objectives; and
7. Identified findings relative to the above procedures and provided recommendations for improvement.

Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland and the OIG office in Philadelphia, Pennsylvania. The procedures that we performed were in accordance with the American Institute of Certified Public Accountants’ Statement on Standards for Consulting Services, and are consistent with appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did not conduct an audit, the objective of which would be the expression of an opinion on the reliability or accuracy of the reported results of the performance measures evaluated. Accordingly, we do not express such an opinion.

We obtained an understanding of the underlying processes and operating procedures surrounding ACIS and the generation of performance measures through interviews and meetings with the appropriate SSA OIG personnel and by reviewing the following documentation:

• Policies and procedures manual for procedures surrounding the processing, accumulating, and reporting of the data for the four performance measures;
• PwC system walk-through descriptions;
• PwC system flowcharts;
• SSA OIG-provided system descriptions, including the ACIS User Manual and the Special Agent Handbook;
• FY 2000 ACIS data and corresponding data definitions;
• The National Institutes of Health Statement on Auditing Standards (SAS) 70 Reports for FY 1999 and FY 2000, describing the general controls surrounding the ACIS application;
• Internal report on Investigation Related Recoveries; and
• Internal and external reports on the four performance measures (including OIG, General Accounting Office, etc.)

Based on the understanding we obtained during the planning part of this engagement, a review of related prior-year audit work at SSA, and our understanding of the Federal Information Systems Controls Audit Manual (FISCAM) methodology, we developed and performed tests of internal controls (both general and application) related to ACIS, for the following areas:

• Access Control (including Separation of Duties);
• Data Input;
• Data Rejection;
• Data Processing (including backup and recovery); and
• Data Output.

To verify, validate, and test the accuracy of the FY 2000 data we performed the following:

1. Obtained a copy of the ACIS data as of September 30, 2000 from the OIG Office of Executive Operations;
2. Created and executed audit control language (ACL) programs to extract random samples for Old-Age, Survivors and Disability Insurance, Supplemental Security Income and Conviction Cases; and
3. Created an ACL program to display the monetary figures for the cases selected.

1. Verified that a completed OI-1 was properly signed and completed;
2. Verified that the OI-68 was completed and was supported by documentation in the file;
3. Verified that a completed OI-9 was properly signed by the Field Division SAC;
4. Traced the monetary achievement documentation on the OI-9 and OI-68 to the amount of monetary achievement in ACIS to ensure the monetary amount reported in ACIS was in fact correct; and
5. Ensured that the OI-4 agreed with information on OI-9, OI-68, and ACIS. OI-4 obtained from sub-offices that choose to include them in the documentation sent.

• Judgmentally selected a sample of 10 case files from the closed cases at the Philadelphia field division (FD), and with the assistance of FD personnel we traced and agreed the OI-9 to the information in ACIS. In addition, we ensured that the information included on the OI-9 was valid by performing the following:

Ensured the following documents were included in the file:

1. Allegation report (i.e., SSA-8551, Fraud Referral Form).
2. OI-1 Case Opening Report, properly signed and completed by the FD Special Agent-in-Charge (SAC).
3. Overpayment recipient Numident, and Master Beneficiary Record (MBR)/Supplemental Security Record (SSR).
4. OI-4 Report of Investigation, reviewed and signed by the FD Assistant Special Agent-in-Charge (ASAC).
5. Supervisory File review sheet.
6. OI-9 ACIS input form, properly signed and completed by the FD SAC. Ensured that the OI-9 agreed with the information on the OI-4.
7. OI-31 Case Closing Checklist reviewed and signed by the FD ASAC.

1. OI-16A Statement of Overpayment Recipient signed by the subject and the SAC.
2. Copy of check provided by the subject.
3. Check receipt from SSA personnel.

Based on our understanding of SSA and Performance Measures (PM) #5 through #8, we obtained the FY 2000 ACIS data and performed Computer Assisted Audit Techniques (CAATs) using ACL to accumulate counts for four measures, and then compared those results to the figures reported for GPRA. We then reconciled any differences through data analysis and subsequent discussions with SSA OIG personnel.

As part of this engagement, we evaluated the appropriateness of each of the performance measures with respect to GPRA compliance and SSA’s APP. We determined whether the specific indicators and goals corresponded to the strategic goals identified in SSA’s APP, determined whether each of these indicators accurately measure performance, and determined their compliance with GPRA requirements.

ACIS Allegation and Case Investigative System
ADABAS A Database
AMD Allegation Management Division
AMS Allegation Management System
APP Annual Performance Plan
ASAC Assistant Special Agent-in-Charge
CAATs Computer Assisted Audit Techniques
CIT Center for Information Technology
DI Disability Insurance
DMS Debt Management System
EDP Electronic Data Processing
FD Field Division
FIPS Federal Information Processing Standards
FISCAM Federal Information System Controls Audit Manual
FY Fiscal Year
GPRA Government Performance and Results Act
IT Information Technology
J&C Judgment and Commitment Order
MAD Manpower and Administration Division
MATPSC Mid-Atlantic Program Service Center
MBR Master Beneficiary Record
NIH National Institutes of Health
OASDI Old-Age and Survivors and Disability Insurance
OEO Office of Executive Operations
OI Office of Investigations
OIG Office of the Inspector General
OMB Office of Management and Budget
PM Performance Measure
PwC PricewaterhouseCoopers LLP
SA Special Agent
SAC Special Agent-in-Charge
SAH Special Agent Handbook
SAS Statements on Auditing Standards
SLA Service Level Agreement
SSA Social Security Administration
SSI Supplemental Security Income
SSN Social Security number
SSR Supplemental Security Record