ROI

By Patti P. Phillips, Ph.D.

Chief learning officers often must evaluate their key learning programs, collecting several types of data—reaction, learning, application, impact, intangibles, and maybe even return on investment. What if the evaluation produces disappointing results? Suppose application and impact were less than desired, and the ROI calculation negative. This prospect causes some learning executives to steer clear of this level of accountability altogether.

For some CLOs, negative results are the ultimate fear. Immediately, they begin to think, “Will this reflect unfavorably on me? On the program? On the function? Will budgets disappear? Will support diminish?” These are all appropriate questions, but most of these fears are unfounded. In fact, negative results reveal the potential to improve programs. Here are 11 ways to address negative results and use them to facilitate positive transformations:

Recognize the Power of a Negative Study

When the study results are negative, there is always an abundance of data indicating what went wrong. Was it an adverse reaction? Was there a lack of learning? Was there a failure to implement or apply what was learned? Did major barriers prevent success? Or was there a misalignment in the beginning? These are legitimate questions about lack of success, and the answers are always obtained in a comprehensive evaluation study.

Look for Red Flags

Indications of problems often appear in the first stages of initiation—after reaction and learning data have been collected. Many signals can provide insight into the program’s success or lack of success, such as participants perceiving that the program as not relevant to their job. Perhaps they would not recommend it to others or do not intend to use it on the job. These responses can indicate a lack of utilization, which usually translates into negative results. Connecting this information requires analyzing data beyond overall satisfaction with the program, the instructor and the learning environment. While important, these types of ratings may not reveal the value of the content and its potential use. Also, if an evaluation study is conducted on a program as it is being implemented, low ratings for reaction and learning may signal the need for adjustments before any additional evaluation is conducted.

Lower Outcome Expectations

When there is a signal that the study may be negative, or it appears that there could be a danger of less-than-desired success, the expectations of the outcome should be lowered. The “under-promise and over-deliver” approach is best applied here. Containing your enthusiasm for the results early in the process is important. This is not to suggest that a gloom-and-doom approach throughout the study is appropriate, but that expectations should be managed and kept on the low side.

Look for Data Everywhere

Evaluators are challenged to uncover all the data connected to the program—both positive and negative. To that end, it is critical to look everywhere for data that shows value (or the lack of it). This thorough approach will ensure that nothing is left undiscovered—the fear harbored by many individuals when facing negative results.

Never Alter the Standards

When the results are less than desired, it is tempting to lower the standards—to change the assumptions about collecting, processing, analyzing and reporting the data. This is not a time to change the standards. Changing the standards to make the data more positive renders the study virtually worthless. Without standards, there is no credibility.

Remain Objective Throughout

Ideally, the evaluator should be completely objective or independent of the program. This objectivity provides an arms-length evaluation of its success. It is important not only to enter the project from an objective standpoint but also to remain objective throughout the process. Never become an advocate for or against it. This helps alleviate the concern that the results may be biased.

Prepare the Team for the Bad News

As red flags arise and expectations are lowered, it appears that a less-than-desired outcome will be realized. It is best to prepare the team for this bad news early in the process. Part of the preparation is to make sure that they don’t reveal or discuss the outcome of the program with others. Even when early results are positive, it is best to keep the data confidential until all are collected. Also, when it appears that the results are going to be negative, an early meeting will help develop a strategy to deal with the outcome. This preparation may address how the data will be communicated, the actions needed to improve the program, and, of course, explanations as to what caused the lack of success.

Consider Different Scenarios

Standards connected with the ROI methodology are conservative for a reason: The conservative approach adds credibility. Consequently, there is a buy-in of the data and the results. However, sometimes it may be helpful to examine what the result might be if the conservative standards were not used. Other scenarios may actually show positive results. In this case, the standards are not changed, but the presentation shows how different the data would be if other assumptions were made. This approach allows the audience to see how conservative the standards are. For example, on the cost side, including all costs sometimes drive the project to a negative ROI. If other assumptions could be made about the costs, the value could be changed, and a different ROI calculation might be made. On the benefit side, a lack of data from a particular group sometimes drives a study into negative territory because of the “no data, no improvement” standard. However, another assumption could be made about the missing data to calculate an alternative ROI. It is important for these other scenarios to be offered to educate the audience about the value of what is obtained and to underscore the conservative approach. It should be clear that the standards are not changed and that the comparisons with other studies would be based on the standards in the original calculation.

Find Out What Went Wrong

With disappointing results, the first question usually asked is, “What went wrong?” It is important to uncover the reasons for the lack of success. As the process unfolds, there is often an abundance of data to indicate what went wrong. The follow-up evaluation will contain specific questions about impediments and inhibitors. In addition, asking for suggestions for improvements often underscores how things could be changed to make a difference. Even when collecting enablers and enhancers, there may be clues as to what could be changed to make it much better. In most situations, there is little doubt as to what went wrong and what can be changed. In worst-case scenarios, if the program cannot be modified or enhanced to add value, it may mean that it should be discontinued.

Adjust the Story Line

When communicating data, negative results indicate that the storyline needs to change. Instead of saying, “Let’s celebrate—we’ve got great results for this program,” the story reads, “Now we have data that show how to make this program more successful.” The audience must understand that the lack of success may have existed previously, but no data were available to know what needed to be changed. Now, the data exist. In an odd sort of way, this becomes a positive spin on less-than-positive data.

Drive Improvement

Evaluation data are virtually useless unless used to improve processes. In a negative study, there are usually many items that could be changed to make it more successful. It is important that a commitment is secured to make needed adjustments so that the program will be successful in the future. Until those actions are approved and implemented, the work is not complete. In worst-case scenarios, if the program cannot be changed to add value, it should be terminated, and the important lessons should be communicated to others. This last step underscores that the comprehensive evaluation is used for process improvement and not for performance evaluation of the staff.

Negative study results do not have to be bad news. Negative results contain data that can be used not only to explain what happened but also to adapt and improve in the future. It is important to consider the potential of a negative study and adjust expectations and strategies throughout the process to keep the negative results from being a surprise. In the worst-case situation, negative data will surprise the key sponsor at the time of presentation.

Join our upcoming HRDQ-U webinar titled “What Caused It?: Connecting Programs to Results” on July 22, 2020 at 2pm ET/11am PT.

By Patti P. Phillips, Ph.D., and Jack J. Phillips, Ph.D.

Have you ever asked top executives or a chief financial officer about the value they would like to see from talent development? How many discussions have you had about the value of learning with the C-Suite?

We have had many of those conversations routinely over the past 25 years and we know clearly what they need. Their responses have been documented quite well, dating back to a major study that we conducted with ATD nearly a decade ago. That study, involving Fortune 500 CEOs, indicated that 96 percent of executives wanted to see a business connection to learning. Yet, at that time, only 8 percent of them had that type of data. This is their No. 1 desired data category. Further, 74 percent of the executives wanted to see the ROI from learning investments, but only 4 percent said they have it now. This is their No. 2 measure. The No. 1 measure provided to executives from L&D was reaction data, but only 28 percent wanted to see this category of data.

This study, first published in our book with ATD, Measuring for Success, was a wake-up call for many CLOs and others involved in talent development.¹ Collectively, they said “we must do better.” The good news is that was 10 years ago. We are well on the way.

More recent data from the Business Intelligence Council of Chief Learning Officer Magazine showed that improvements are happening. When asked about how the learning organization shows its contribution to the broader enterprise, 36 percent said they use business data for the request, and 22 percent say they use ROI. When asked if they plan to implement ROI, 49.6 percent said they planned to implement ROI in the future. All totaled, 71.2 percent of respondents said they were either using ROI or planning to implement it. We think that is a little ambitious, although it came from 335 CLOs.

Fast forward to 2017, and we noticed a major benchmarking report from Training Magazine. This report examined the organizations that were “Hall of Famers” in their awards system. These are the organizations that are consistently at the top of their 125 best learning organizations lists. These “Hall of Famers” are very important for benchmarking because others want to know what makes them so successful. The opening statement in the report states, “Ultimately, the success of any program is based on whether it improves business results.” —Training Top 10 Hall of Fame, May 2017.

These top learning organizations advise that you must connect learning to the business to capture executive attention. This benchmarking report is generated every year. In the following year, 2018, this report contained three best practice case studies: onboarding, an ROI calculation on a follow-up basis, and an ROI forecast. You can see that we are making progress to meet the request from top executives.

What can you do if you are not showing the business value of learning? You can take five very important steps:

1. Be proactive. Don’t wait for the request to show business value. Start delivering business value on a major program now. Take charge and drive the evaluation initiative. Keep ROI on your agenda, not your executive’s agenda.
2. Be selective on which programs you evaluate at the business impact and ROI levels. Use ROI for programs that are very expensive, strategic, important to organizations, and yes, those that attract executive attention. That will usually be about 10-20 percent of the programs each year at the impact level and approximately 5-10 percent at the ROI level.
3. Change the thinking of the complete learning cycle. Start with why for your programs; connect it to the business measure at the beginning. Then make sure you have the right solution. Next, expect success with very specific objectives through to impact and share them with the team. With this approach, you are designing for the results you need. With the business data clearly defined in the beginning, you will have the desired results at the end.
4. Share the joy. Make sure that the entire team is involved in designing, developing, and implementing learning and development to deliver impact. Designers, developers, facilitators, participants, and managers of participants are critical to achieving impact success. Each stakeholder has a role, not just the evaluator. This approach makes a world of difference.
5. Think about all the benefits. While business data will convince executives to continue to fund your programs, connecting to the business will help you build partnerships with business leaders, obtain needed support to make programs more effective, and secure the commitment you need to be successful.

Collectively, the team can make a difference.

Reference

1. Phillips, Jack J., and Patti P. Phillips. Measuring for Success. Alexandria, VA: ASTD Press. (2009) Paperback

By Patti P. Phillips, Ph.D. and Jack J. Phillips, Ph.D.

There are five reasons why you should consider measuring the impact and ROI of your programs. However, there are situations where maybe you shouldn’t measure the ROI—but we will come back to that. First, let’s review what is involved when measuring ROI.

Essentially, an ROI evaluation requires evaluating the success of a learning program on five levels of outcomes. The first level is the reaction to the program in terms of relevance, importance, and intent to use. The second level is learning the knowledge and skills required to make the program successful. The third level, application, is tracking how the content has been used and how much success participants have had with its use, along with any barriers and enablers.

The fourth level is impact—the connection of the program to key business measures. At this outcome level, steps must be taken to isolate the effects of the program on those measures and then convert the business measure to money and calculate the ROI. The fifth level, ROI, is the comparison of the monetary benefits of the program to the costs of the program.

Now that you know what’s involved in an ROI study, let’s discuss why you should measure the ROI of your programs:

1. 1. To make the programs better. This has always been our preferred reason for pursuing the impact and ROI analysis. In the past, most of our efforts have been aimed at evaluating level 1—(reaction) and level 2 (learning). However, executives who provide the support and funds would prefer to see the impact of the program. Of course, impact won’t develop unless there is application. The challenge is to improve level 3 and 4 (application and impact) results. An ROI evaluation will indicate if a program is successful or not. If it’s not, the evaluation will show what should be done to make the program better. If it is working, the analysis will give insight for how to make the program even more successful.

1. 1. To satisfy key stakeholders. Programs have many stakeholders, but no one is more important than those who sponsor, support, or provide the budget for programs. These individuals want to see the business connections. Even in governments or nonprofits, alignment to business measures is still something stakeholders want to see. Dozens of studies have shown this is needed, and it is not debatable. Just ask your top executives.

1. 1. To please your team. When we first used this methodology many years ago, we were involved in managing learning and development teams. We wanted to know that our work was connected to the business. There is a sense of satisfaction in knowing that you are making a difference and the difference is not just having people attend programs and provide good ratings. It is the realization that the participants are using what they have learned on the job and are having an impact in their work and in the organization. That’s a great feeling.

1. 1. To maintain or enhance your budget. This is the No. 1 reason we see ROI implemented now. Globally, economies are in a state of uncertainty. When that happens, organizations must be lean and agile to sustain what is next to come. To do so, budgets are trimmed—eliminating anything perceived as a cost or otherwise not absolutely necessary. Unfortunately, this almost always affects the L&D budget. One of the best ways to convince an executive that your program is not a cost (that should be cut), but rather an investment, is to determine the return on investment, using the same calculation that a chief financial officer would use.

1. To change the perception of the L&D function. Sometimes learning and talent development programs are considered a “necessary evil” or compulsory. Many executives agree with this comment: “We know we have to fund training for compliance and to teach people how to do their jobs, but beyond that, we don’t like to provide courses unless there’s extra money.” This is not a positive statement. We have hundreds of studies that show that the highest ROIs come from soft skills programs, such as team building and leadership development, and it is now the time to invest heavily in these areas. But it’s difficult to invest more unless executives see the value. Having consistency within your L&D function routinely—showing the value and the connection to the business for major programs—is the best way to change the perception from a necessary process to a business driver. This will lead to more support, better partnerships, and yes, a seat at the table.

So there you have it, five reasons to measure ROI. But, as we stated in the beginning, maybe you shouldn’t measure your program. This process is not intended for every program. Only programs that are important to the organization, expensive, strategic, or those that attract the interest of top executives should be evaluated at this level.

Our benchmarking studies show that this should not be more than 5 to 10 percent of programs each year. Consider the advantages of and reasons to evaluate a program, and then decide—is it worth it? For more information, email Info@roiinstitute.net.