Separating Signal from Noise: a Framework for Monitoring Compliance Program Performance

By Waqas Shahid, Randall H. Cook, Rosanne Giambalvo

September 27, 2017

As a compliance professional, you learn something about organizational noise. On any given day, you may hear from a program manager that you are single-handedly shutting down the business, from the Chief Financial Officer that you are breaking the budget, and from the General Counsel that you are enabling excessive risk. In this real compliance world of cross-cutting interests, tight timelines, and hard choices, you need a clear picture of your company’s compliance posture and performance. But how do you make sure that your picture is accurate? Is the crisis du jour or a squeaky wheel stakeholder obscuring the riskiest and most critical parts of your compliance program? Can you objectively measure your program’s performance and improvement over time to justify the budget allocated to your program? Are you sure you are effectively allocating your limited resources to tackle your highest-risk items instead of spending time on a low-risk, low-activity area? And can you communicate the effectiveness of your program to others inside and outside your organization? This piece describes a deliberate process for collecting and utilizing data and metrics to assess and optimize your company’s compliance programs.

THE CASE FOR COMPLIANCE ANALYTICS

A data analytics-based oversight framework is now a necessary component of every competent compliance program. Indeed, US enforcement authorities increasingly expect to see the metrics and data that companies use to evaluate compliance performance and make compliance decisions, and have repeatedly emphasized the need for such risk-tailored compliance programs at industry conferences and in compliance consent agreements. Equally important (and typically more immediate) corporate operational and financial decisions rely heavily on quantitative metrics.

A data and metric-based monitoring effort can, however, be more than a shield against government enforcement and internal budget constraints. It can also be used as a sword to relentlessly attack risks and inefficiencies, identify hidden opportunities, and nip problems in the bud. For example, data showing that export license applications or entertainment authorization requests bottleneck at certain times of the year could provide insights about business and workflow cycles, and could present an opportunity to pre-allocate additional resources during those periods to support optimal business performance or tweak workflow cycles to avoid those time periods altogether. Similarly, tracking “Returned Without Action” rates for export license applications companywide and by preparer over time could help you identify when additional or refresher training on license preparation is warranted.

These factors highlight the need to obtain hard data about your company’s compliance programs through a systematic monitoring effort. Indeed, the collection and analysis of hard data about your company’s business and compliance activities is a critical part of identifying compliance patterns over time, pinpointing potential problems, and forming effective solutions. But taking an ad-hoc approach to gathering and analyzing data can sometimes make it hard to separate signal from noise and lead to “analysis paralysis.” You need to take a measured, systematic approach. Below, we present a robust framework for effectively measuring and communicating the effectiveness of your compliance programs to internal and external stakeholders.

A FRAMEWORK FOR SUCCESS

In a nutshell, our compliance measurement framework consists of two three-step sub-cycles that recur with different frequencies: an annual Planning Cycle aimed at developing relevant, useful metrics and key performance indicators (KPIs), and figuring out how to collect the needed data; and a monthly Execution Cycle that focuses on recurring collection and analysis of relevant data to drive continuous improvement of your compliance program.

A. The Monitoring Planning Cycle (yearly)

  1. Understand Your Business
  2. Articulate What You Want to Measure
  3. Build/Update Your Data Map

B. The Monitoring Execution Cycle (monthly)

  1. Collect, Sanitize, and Transform
  2. Chart, Analyze, and Present
  3. Take Action

Chart displaying the planning cycle and execution cycle

 

A. THE PLANNING CYCLE

A.1. Understand How Your Business Operates, Its Risks, and Its Objectives

Understanding your business activities and objectives is an important first step of this framework. At a fundamental level, you cannot determine whether your company is conducting business in a compliant manner and whether your compliance programs are efficient if you do not understand your company’s business.

The specific, relevant facets you need to understand will vary by issue area and organization. For an export compliance program, for example, understanding the business means knowing the export touchpoints within your company through an understanding of, among other things, key company products and services (what they do, how they are classified, who buys them, etc.), how critical business processes work, transactional volumes, key customers and locations, key vendors and suppliers, critical enterprise systems (including engineering and IT systems), and existing compliance controls (policies, procedures, systems) you may already have in place. For comparison, for an anticorruption program, you should focus on understanding how your business facilitates and incentivizes sales and transactions, particularly when public officials are involved. Who and where are your sales representatives? How do they operate? How do people get paid? How does your company account for transfers of non-monetary valuables such as services and products? Of course, you should also aim to have a deep understanding of the systems and procedures used for payments and accounting for transfers of value.

Although you may think you have a good idea of how your company functions and how relevant regulations may apply to those activities, the fact is that it is almost impossible for one person to know all the details within an organization of any appreciable size. Conducting a systematic and thorough compliance self-assessment can help you fully identify, document, and understand the key compliance touchpoints for your company, and identify specific risk areas for improvement and compliance program focus.

But don’t stop there. Compliance professionals often mistakenly believe that the success of their compliance program is somehow insulated from the strategic objectives and success of their company’s business activities. It is not. Although compliance with the law is the necessary cornerstone of any credible compliance program, it is not enough. You need to make sure that your compliance programs achieve compliance with the law at the speed of business, promoting the achievement of business objectives rather than hindering them. Otherwise, you run the risk of business personnel finding creative ways to work around compliance systems and safeguards, or applying organizational pressure to scale back compliance.

Accordingly, prior to deciding what and how to measure, make sure you understand your company’s strategic and tactical objectives and how your compliance program may help or hinder those objectives. Doing so will allow you to focus on key compliance processes and craft KPIs related to those processes, resulting in more effective engagement of internal stakeholders and demonstration of compliance program value. For example, if the company is focused on globalizing its manufacturing supply chain, in the export control context, make sure you closely monitor the volume, destination, and speed of technology exports; track remaining values on export authorizations; and track your export authorization pipeline and lead times. Do whatever is in your power to ensure the export process is as smooth as possible. Concurrently, to address the increased sanctions and corruption risks, you should enhance monitoring of third-party screening and hit resolution, and of the number and character of international financial transactions utilized to support the new activities.

Of course, in addition to knowing where you are going, you should also consider where you have been. If your company has faced particular compliance issues in the past, it is a good idea to continue monitoring those specific areas. Even if you think you have fixed the problem, it helps to have data to provide evidence that the fix is effective. For example, if you created an automated workflow to make your gifts and gratuities approval process more efficient and accountable, you should track process performance before and after the automation transition so you can ensure that the company gets the anticipated benefit out of the enhancement and demonstrate the value of the investment. Similarly, in the export control context, if your company previously submitted disclosures regarding failure to timely update your authorizations following a party’s name change, you will want to monitor and track current performance in this area until you are sure that the problem is effectively remedied.

A.2. Articulate What You Want to Measure

Once you have a functional understanding of how your business operates, its risks, and its objectives, you can start identifying relevant metrics and KPIs.[1] You cannot measure everything (not enough time, bandwidth, or money), so it is important to be methodical in selecting the metrics and KPIs you want to focus on. Broadly speaking, you should aim to select metrics that give you a good idea of what is going on with respect to (1) transactional activity (e.g., export volume, third-party payments, commissions); (2) the health and status of your compliance program; and (3) the impact of compliance activity on key business objectives. KPIs should be selected based on their usefulness and criticality as proxies for measuring the success of your compliance program based on your understanding of company objectives, risks, and known compliance weak spots or areas of concern.

The table below provides an example set of metrics and KPIs for an export control compliance program. These are only intended as examples, and the relevant metrics and KPIs for your program will differ depending on the nature of your business, compliance risks, and the results of your self-assessment. For example, if you discover during your self-assessment that you have no or very immature processes for dealing with required notifications to the State Department for export authorizations, you may want to craft some metrics specifically to monitor compliance in that area.

CATEGORY EXAMPLE KPIS AND METRICS
EXPORT ACTIVITY/VOLUME
  • How many hardware units were exported last month? Break down by:
    • Product category;
    • Program;
    • Export classification;
    • Authorization type used (license, agreement, exemption/exception, “no license required,” etc.);
    • Exporting business unit/shipping location;
    • Destination country;
    • Receiving customer/partner, etc.
  • How many separate export shipments were made last month?
  • How many files (technology/technical data) were exported electronically last month? Break down along different dimensions, as with hardware shipments.
Compliance Program Status
LICENSING
  • How many export licenses (including agreements) does the company currently hold? Break down by:
    • Product category;
    • License type;
    • Product category;
    • Program;
    • Destination country;
    • Receiving customer/partner, etc.
  • How many license requests are currently in the pipeline? Break down by:
    • Type;
    • Status;
    • Requesting BU/department;
    • Destination, etc.
  • What is the value remaining on export licenses (export/import/hardware manufactured abroad)?
  • What is the forecast for license-required exports forecast v. available export authorization value?
  • What are the DSP-83 cycle times and other export authorization-related process times?
  • What was the average cycle time last month for preparing and submitting license requests? Break down by license type, requesting business unit, license specialist, etc.
  • What was the on-time submission percentage for required license notifications to the government? Break down by license type, BU that owns license, compliance professional, etc.
JURISDICTION & CLASSIFICATION
  • How many new part numbers/catalog items were created last month? Break down by jurisdiction/classification.
  • How many new technology/technical data jurisdiction/classifications were made last month?
  • What was the average cycle time last month for conducting a jurisdiction/classification determination? Break down by product, ultimate classification, and reviewer.
  • What is the current classification status (classified, not classified) breakdown of all company part numbers? All technology/technical data to be exported?
  • What is the status breakdown of all third-party classification requests?
  • What is the burndown status of part numbers affected by ECR change(s)?
AUDITS & INVESTIGATIONS
  • What is the completion status for prior audit observations/corrective actions?
  • What is the completion status and results for specific area follow-ups/reviews required because of the last audit?
  • How many new compliance issues were reported in the last month? Break down by reporting source, regulatory issue area, compliance program issue/area, implicated policy/procedure, implicated program/product category, severity/importance, etc.
  • What is the completion status of corrective actions committed as part of disclosures/investigations?
  • What is the year-to-date breakdown of confirmed violations? Break down by regulatory issue area, compliance program issue/area, root cause(s), etc.
EXPORT REVIEW PROCESSES
  • What was the average export review cycle time last month? Break down by jurisdiction/classification time, end user/end use review time, export authorization review time, and export reviewer.
  • What was the average export review cycle time last month for technical data/technology shipments? Break down by above dimensions.
  • What percentage of export requests were cleared within X days?
  • What was the average review cycle time last month for site visits by foreign nationals?
  • What was the average review cycle time last month for overseas trips by company personnel?
  • What percentage of export reviews were automated vs. manual?
PERSONNEL (INCLUDING TRAINING)
  • What is the headcount for compliance professionals? How many empowered officials? Break down by qualification/role, department, average tenure, etc.
  • What is the training rate for the organization? Break down by level of compliance training completed, total population trained by business unit, function, etc.
  • What percentage of discovered violations resulted from lack of training or employee misconduct?
BUSINESS OBJECTIVE KPIS
  • What was the average export review cycle time for key strategic products?
  • Identify other metrics/KPIs as dictated by your company’s specific business objectives

In addition to the above, you should also consider metrics for any special projects that the compliance program may be undertaking (e.g., export classification effort for a new system/product/bill of materials, burndown of export authorization review requests). For KPIs, also consider establishing various performance targets that will allow you to quickly build scorecards late. Doing so enables you to set a baseline standard and quickly and visually communicate performance against those standards. For example, for a KPI measuring the average export review cycle time, you may want to consider establishing the following performance level targets: “Green – Excellent (< 24 hours),” “Yellow – Acceptable (< 3 days),” and “Red – Remediation Required (> 3 days).”

A.3. Build a Map

The next step (and the last in the Planning Cycle) is to build a data map that allows you to methodically correlate metrics to data sources. Of course, it would be easiest to bake in your metrics within your enterprise systems from the very beginning. But in reality, you rarely get that chance. Instead, most corporate compliance personnel must deal with an array of legacy compliance, enterprise resource planning, engineering, quality management, technology exchange, and other systems that all utilize and host data that will be relevant to metrics of interest. These systems are rarely owned by the compliance function or well-integrated. Even when you think you know all the relevant enterprise systems, it can be daunting to wade through databases, schemas, tables, columns, lookup values, and other technical implementation details every time you want a new metric to figure out what is relevant and what you should ignore.

Accordingly, before you start collecting and crunching data from enterprise systems to feed your metrics machine, take a step back and build a map of your compliance technology ecosystem to guide your data collection and analysis efforts. Like the self-assessment exercise discussed earlier, this step consists of a systematic inventory. But this time, instead of business processes, you are mapping technology systems relevant to export controls.

Once you have an inventory of the different systems that may hold potentially relevant data (e.g., licensing & classification data, payment records, shipment data, training records), identify which systems (or combination of systems) are most likely to hold the data relevant to the KPIs and metrics you previously identified. For each of these key systems, probe deeper to understand the context in which the data is created and maintained, and where to find the specific pieces of information you will need for your analysis.

For example:

  • What types of facts or events do the data reflect?
  • How are records in this system created, updated, maintained, and purged?
  • What do specific tables and fields within a database schema represent?
  • How do you distinguish complete records from incomplete or abandoned records?
  • Does the system provide the data you want in a readily digestible means (e.g., a report/dashboard form)? How can data/records be exported from the system?
  • Who has access to this data/records system? How is access managed?
  • Who uses the system? What is the process for creating data?

One thing you might discover through this process is that your technology systems do not contain the requisite data you need for certain metrics or KPIs you want to track. If this is the case, you should consider whether there is a need to plug a critical information gap in your company’s documented procedures and systems of record.

Through this mapping exercise, you should develop a good documented idea of where your data resides, what data/records are relevant, what information to ignore, and how to extract the information you need. Although this exercise may appear cumbersome, it is a necessary step to ensure that the data you collect is complete and an accurate reflection of reality. You should only need to conduct a full inventory and mapping exercise once and update, as needed, when new systems are deployed or existing ones are updated.

B. THE EXECUTION CYCLE

Now that you have completed a Planning Cycle and created a roadmap for what data to collect, you are ready to start running your data analysis machine. We describe the three steps of the Execution Cycle below:

B.1. Collect, Sanitize, and Transform Raw Data

The first step in feeding the analytics machine is to create a process for the regular, periodic collection of relevant data from the various systems identified in the previous step into a central data store (i.e., a database or even an Excel workbook). Centrally collecting this data has several benefits, including allowing you to:

  • Interconnect, cross-reference, and manipulate data from multiple systems without worrying about how such manipulations will affect live enterprise systems. Doing so allows for richer insights into your compliance program.
  • Evaluate trend data over time and conduct advanced data analysis.
  • Utilize a single analysis tool (Excel or other commercial data analytics tools) for your analysis.
  • Record the basis for your metrics and KPIs, allowing traceability.

In addition to collecting the data, you should also make sure you have a process for sanitizing and transforming the data. Bad, duplicate, or incomplete data can disrupt your analysis efforts, or worse, result in misleading metrics and KPIs. Accordingly, before you start slicing and dicing your data, it is important to ensure that your data is “clean.” Sanitizing the data involves identifying and removing/flagging duplicate, incomplete, and irrelevant data records. Of course, this requires a high level of familiarity with the systems from which the data originated and is another reason why the “Build a Map” step above is critical.

After you’ve cleaned your data, you may also need to do a bit of transformation. Basically, this step involves creating/calculating data points you need for your metrics but which may not be readily available. This can be as simple as creating a new calculated field (e.g., to track the hours to complete a task if what you have are the start and end times for a task) or a complex transformation such as connecting sets of records from different systems into a new set of records (e.g., combining data from shipping and export classifications systems for a fuller picture).

Although the collection, sanitization, and transformation steps may seem daunting, many commercially available tools can help you efficiently achieve these tasks through scripting. Once set up, such scripting will help you quickly and efficiently collect “clean” data sets for analysis.

One final thing to keep in mind is that not all data you will want is in a technology system. For example, to collect data regarding company culture, level of awareness, or effectiveness of communication campaigns, you may need to conduct personal interviews and survey company personnel. Collecting such data is vital to arriving at a complete picture regarding the health of your company’s compliance programs, and you should consider making surveys and sensing sessions a regular and recurring part of your compliance programs.

However, be careful when collecting such data to avoid inadvertently introducing bias into the equation. Consider the difference between the following two questions:

  • This data shows that reports to the compliance hotline have dropped in the last year. Do you think employees are afraid to report violations?
  • This data shows that reports to the compliance hotline have dropped in the last year. Why do you think this may have occurred?
  • The first question requires merely a “yes” or “no” response from the interviewee and suggests a reason for the decline of hotline reports. The second question is open-ended, which may elicit more diverse, detailed information or reasons the interviewer had not considered.

B.2. Analyze and Present

  • BLUF AND KISS. “Bottom Line Up Front” and “Keep It Simple, Sam.” One of your primary goals in monitoring your compliance program is to effectively communicate its performance to critical stakeholders. Accordingly, always state your conclusions up front and do not let your audience wrestle with raw data on their own, struggling to answer the question, “Yes, but what does this all mean?” Along these same lines, the most compelling ideas are often the ones presented most simply and clearly. Make sure your presentations are clean, crisp, and to the point. Save the razzle-dazzle (e.g., advanced statistical evaluation methods or newly released Excel features) for when absolutely necessary and generally after you have made your fundamental points.
  • KEEP IT CLEAN AND CONSISTENT. Keep charts and graphs clean, visually appealing, and consistent. Draw attention to what is important and away from what is not through thoughtful use of colors (do not use crimson red unless you absolutely want to call attention to something; do not use very similar colors in legends, etc.), organizing charts along dimensions that make sense for the metric (e.g., from highest to lowest; alphabetically by BU), and obfuscating unnecessary detail — for example, in a pie chart, do you really need to separately represent 15 different data points, each representing less than 1 percent of the pie? Lump these smaller slices together into a slice labeled “Others” with some detail in the legend. Also, make sure you keep your presentation language consistent. That is, use colors, chart types, and other visual elements consistently from one presentation to the next so the audience can quickly digest the presented information.
  • SNAPSHOTS ARE GOOD; TRENDS ARE BETTER. Metrics that represent snapshots in time rarely give the full picture. For most metrics/KPIs, you should ensure that you are tracking, evaluating, and presenting trends over time rather than just the latest measurement. Doing so will allow you to see current performance in context and give you and your audience a better idea of whether your program is improving, stagnating, or in need of course correction.
  • CHART THE SAME DATA ALONG MULTIPLE DIMENSIONS. The devil is in the details. Good-looking averages can often mask worrisome performance in specific areas or by specific business units or persons. Accordingly, when looking at a KPI, it is useful to break down the KPI along multiple dimensions to ferret out details. For example, in addition to determining the company’s overall license application cycle time, you should also consider breaking down this metric by product family involved, business units involved, or even licensing specialists involved. Doing so will allow you to better understand if there are any significant deviations from the average that warrant further attention. Of course, if you find nothing interesting in the details, you do not need to include such a breakdown in your presentations.
  • BEWARE AND FOCUS ON THE OUTLIERS. In preparing your charts and metrics, make sure you also look at the underlying data to identify any outliers. An outlier is any data point that is well outside of the mean. For example, if most business units approve exports within three days, but one unit’s average approval time is 20 days, you should investigate why this is the case. It could be that the outlier merely shows unaccounted-for complexity or unique circumstances within that business unit (e.g., it sells particularly complex technology that requires a close analysis before each export can be approved). However, it also could be that the outlier business unit has been improperly siloed from other business units and has not received the same training as other units. The outlier could also indicate some other distinctive attribute for that specific business unit. The important thing is (i) to understand why the outlier occurred and (ii) to either confirm that it does not indicate a compliance gap or, if it does, to remedy the issue.

To make the best use of your monitoring efforts, it is important to ensure that you run your metrics on a recurring, scheduled basis. We have found that doing so monthly is usually sufficient, although you can adjust that cycle time depending on your resources and needs. Even when the metrics are not used for critical stakeholder presentations, it is still important to have them available for your own planning and compliance program prioritization, and for potential future use (e.g., in trends analysis).

B.3. Take Action

The final step in this framework is to act. Your monitoring efforts will often lead to the identification of problem areas or efficiency opportunities. When you find such golden nuggets of opportunity, make sure you have an action plan to address the issue. Once implemented, make sure the action taken has the desired effect by closely monitoring relevant metrics in subsequent iterations of the Execution Cycle.

In the larger picture, you should leverage the metrics to enrich your understanding of program priorities, risks, and opportunities. In the scarce economics of compliance resourcing, it is vital to use your data-based insights to prioritize your existing work backlog, identify efficiency opportunities, allocate and prepare your budget, and plan your compliance program’s next steps.

Compliance programs are inevitably under the scrutiny reserved for company overhead. The constant trend is to be asked to do more with less. In this lean operating environment, a systematic data monitoring program is a critical enabler. Done properly, it will give you needed insight into your program’s performance, risks, opportunities, and priorities. Moreover, it provides the quantitative basis to both demonstrate value to business stakeholders and to justify continued investments in your program. You can use your methodically analyzed data and reports to communicate the direct relationship between compliance program performance and your company’s most important assets, relationships, and reputation. Effectively used data can shift the discussion around your compliance budget from the downward ratchet of overhead costs to a more rewarding analysis of risk, value, and investment.