How to Identify Quality Benchmark Data

Benchmark reports are increasingly common in the human services sector, but how do you know if this data is valuable to your organization? This blog post provides an overview of benchmarking and tips for evaluating the quality of benchmark data available to you.

What does a bench have to do with it?

Not much – unless you’re a craftsperson in antiquity. The term benchmark comes from the marks made on wooden workbenches to indicate shoe size (for cobblers), the length of board needed (for carpenters), etc. Today, a benchmark is a summary calculation – typically an average or median – which attempts to tell us what is normal for a metric.

Benchmarking within the human services sector usually comes in 3 flavors:

• Outcomes benchmarks are complex measures of change in behavior, thinking, or status. Examples include permanency in child welfare systems (such as these dashboards from the Minnesota Department of Human Services) or abstinence rate/reduction in use measures in substance use disorder treatment services. They speak to the success of a program, service, or intervention.
  
• Administration and management benchmarks are measures of organizational health and stability and target human resource management, financial management, governance, and similar activities. Related metrics speak to the foundational elements of an organization on which quality service delivery depends, and include staff retention rate, months cash on hand, or average board attendance rate.
  
• Fundraising/development benchmarks are measures of marketing and development success and efficiency. Examples include the open rate of an email seeking donations or giving rates by demographic categories. These benchmarks are the most commonly available (often at a price) as they are standard in marketing software solutions and easy to calculate and share.

What makes a good benchmark?

A benchmark is composed of 3 basic elements: the metric, the benchmark value, and the comparison group. You will need a strong definition of each to determine the value of a benchmark to your organization and should expect any quality benchmark report to outline them.

The metric

The metric should be clearly defined with a title, description, purpose, time period, and the underlying formula. Let’s use a common Human Resources measure, staff turnover, as an example:

Staff turnover is the rate of employees who leave an organization and are replaced by new employees in the stated fiscal year. This measure demonstrates an organization’s ability to maintain a stable and qualified workforce and is calculated as: (Total number of employees at the start of the fiscal year minus the total number of employees who left) divided by the total number of employees at the start of the fiscal year.

A well-defined metric is a strong indicator that all participating organizations calculated their figure consistently.

The benchmark value

This is the summary calculation and is typically an average, median, or—occasionally —a range of the sample. Here is a description of each:

• The average or mean is the “typical” value in the data. The average is only appropriate for data with a normal distribution (the bell curve shape) and is sensitive to outliers in the data.
  
• The median is the middle value in the data. It is less susceptible to outliers and appropriate for data with a non-normal distribution. Not sure what a normal or non-normal distribution is? No worries! The average and median will be extremely close if the data has a normal distribution, so—in any case—the median is always a safe bet.
  
• A full set of descriptive statistics provide a nuanced but complex view of the data’s distribution; it is the most illustrative method of describing benchmark data but can easily overwhelm most readers. These calculations can include: minimum value, maximum value, average, median, standard deviation, quartiles/percentiles, and the interquartile range.

The comparison group

The comparison group is comprised of the organizations participating in the benchmarking project. Understanding the comparison group is more than knowing the number of organizations which submitted data to the project; their overall demographic profile is the best way to assess the benchmark’s value to your organization.

What is the value of a staff turnover benchmark to a substance use treatment organization if the benchmark was generated using data from 100 arts nonprofits? A benchmark is only as valuable as its comparison integrity: the extent to which the comparison group matches the organization utilizing the benchmark to contextualize its performance. And this is what makes quality benchmarking so difficult.

A well-defined comparison group allows you to gauge the relevance of the benchmark to your organization. To have any value, a benchmarking project must collect a variety of demographic data about the participating organizations. Typical variables include:

• Revenue/budget amount: a general measure of organization size; other measures can include workforce size or average clients served per year
  
• Location: city, state, region, and/or community types such as urban, suburban, and rural
  
• Services provided: anywhere from general service categories (child welfare, behavioral health, etc.) to specific program models; this is exceptionally difficult because there are few taxonomies (systems of categorization) for human services

Why is a good benchmark so hard to find?

Generating high-quality benchmark data is a complex, time-consuming task, and, for many benchmarking projects, there is no guarantee of success. A benchmarking project requires the following.

The participants

As stressed above, coding the demographic characteristics of participating organizations is paramount. Other demographic variables typically include organization size (determined by revenue or employee count), location (region, state, and/or city), types of clients served (age, needs, or similar), and – principally – the types of services provided by the organization.

Unfortunately, the only ubiquitous, standardized method of coding human services organizations by services provided comes from the National Taxonomy of Exempt Entities (NTEE) code assigned to all tax-exempt organizations by the Internal Revenue Service (IRS); see a complete list here. However, because this code is determined solely by the IRS, it may not accurately reflect the organization’s actual services.

Well-defined, desirable metrics

The entity leading the benchmarking project must identify commonly-accepted metrics for which participating organizations see a valuable ROI. There must be clear definitions and calculation steps for each metric (yes, the dreaded data dictionary), and participating organizations must agree on their value.

Two-way reporting system

Participating organizations must have a way to provide their data and receive a benchmark report in return. Ideally, this benchmark report is dynamic and displays the data submitted by the organization side-by-side with the benchmark figure.

Quality assurance system

The system to collect and store data must have quality assurance checks to both prevent bad data from entering and identify bad data if it circumvents initial safeguards.

Longitudinal data

The best benchmarking projects will repeat their procedure on a regular schedule and collect data from the same group of participating organizations, or a fluid group of participating organizations with similar demographic characteristics.

Why utilize benchmark data?

There are 4 great reasons to seek out external benchmark data relevant to your organization.

1. Share your success with clients, grant-makers, regulators, and your Board of Trustees

These folks love data, and you love contextualizing your performance. Enriching your reports with external benchmark data can further highlight your successes, pad seemingly-poor performance (“Our turnover rate has risen but is still well under industry benchmarks”), and demonstrate your love of data and participation in the human services world beyond your walls.

2. Feed your performance and quality improvement system

A rich, enmeshed performance and quality improvement (PQI) system is the cornerstone of the Council on Accreditation’s standards and process. From the PQI standard itself:

COA’s Performance and Quality Improvement (PQI) standards provide the framework for implementation of a sustainable, organization-wide PQI system that increases the organization’s capacity to make data-informed decisions that support achievement of performance targets, program goals, positive client outcomes, and staff and client satisfaction. Building and sustaining a comprehensive, mission-driven PQI system is dependent upon the active engagement of staff from all departments of the organization, persons served, and other stakeholders throughout the improvement cycle.

Your organization is likely benchmarking performance already – that is, anytime you look to past performance to evaluate current performance. But finding benchmarks derived from peer organizations further contextualizes performance and pulls your organization out of the limited, sometimes-myopic world of its own data.

3. Get folks talking!

If you use dashboards, you’re probably seeing the same data every day. The line rises; the line falls. The number goes red; the number goes green. Over time, familiarity with the trends in your data can be desensitizing. You understand what is normal and what is not – but all of this happens within the insular universe of your organization.

External benchmark data – and particularly longitudinal benchmark data – gives more context to performance and can reinvigorate data-informed conversations which have faded over time.

4. Set goals and acknowledge high performance

Your performance and quality improvement system craves goals as a driver of performance. External benchmark data can be a powerful tool for pushing staff/programs, but this data can also further celebrate staff/programs performing above the benchmark.

What has been your experience with benchmark data?

Tell us your favorite sources of benchmark data or how you’ve integrated it into your organization’s operations in the comments below!