In-Depth

10 Ways to Debunk ROI Claims

Vendors have lots of ways to manipulate ROI data to their advantage, making it seem as if every product is the answer to your IT prayers. Don't believe the hype.

• By Doug Barney
• 12/01/2004

IT pros are bombarded daily with ROI studies claiming that this product or that can pay for itself in months by slashing costs and making workers far more productive. Microsoft has dozens of ROI studies and reports on its Web site, and Redmond reps never come calling without packing a handy ROI calculator and a batch of glowing case studies.

If all the claims were true, we'd all have peanuts for payroll and take six-week vacations. The truth is, those rosy ROI scenarios are typically full of holes. The trick is identifying which benefits you can really get and what they are worth. "What somebody [else] got doesn't tell you what you'll get," argues Ian Campbell, president of Nucleus Research, a Wellesley, Mass.-based research firm focused exclusively on ROI.

Many IT pros are already wary of ROI claims. "I have never seen a project that resulted in a significant ROI," says Michael Ferguson, Director of Technical Services for Frontier Futures Inc. "I'm not saying it doesn't exist, but usually there's a huge transition which absorbs a lot of the expected benefits. Long-term I have yet to see a reduction of staffing or increase in production that could be tied to [a specific project]."

Others agree: "I view [ROI claims] like advertising. Many [studies] have good ideas but you just can't take the check to the bank," adds Brian Cady, systems administrator for steel manufacturer Austin Mohawk and Company Inc.

1. Be Realistic
Many ROI pitches assume that a new tool will reduce headcount or allow for worker reassignment. But it's never this simple. Headcount savings calculations raise ticklish questions: Can IT be impartial if a new product puts IT jobs at stake? What are the internal organizational realities? Can unnecessary workers be effectively reassigned?

If your organization doesn't have the stomach or the culture to follow through on layoffs, then headcount shouldn't be part of the ROI conclusions.

Productivity is just as tricky. Humans aren't machines. If a product or service makes an employee potentially twice as productive, will that person actually get twice as much done, or will the free time be wasted playing Solitaire or looking for Angelina Jolie JPEGs? Nucleus Research suggests a correction factor of 50 percent for white-collar workers, so only half the newly available time will be spent doing something productive.

Vendors love to tout indirect benefits, and many productivity gains fall into this category. While indirect benefits can have economic value, you should be far more focused on the direct side. For instance, can a new server OS reduce third-party tech support or increase uptime? If so, by exactly how much?

2. Control the ROI Analysis
According to Alinean, which sells ROI software, more than 80 percent of buyers look to vendors to quantify the value of technology under consideration. Fortunately, buyers rarely rely on the vendor's analysis. Fewer than 5 percent believe the vendors do a good job quantifying the value, an Alinean white paper says.

Microsoft has more than its share of ROI calculators, including one from Forrester Research that many Microsoft reps use to sell the controversial Software Assurance (SA) program. This model focuses on the benefits of SA, and calculates an economic advantage and payback for improved support, new product features you're expected to upgrade to, better training and so on. Once you answer a series of questions and plug in some numbers, there's a good chance the template will return an impressive ROI.

It's a great theoretical exercise, but letting the rep control your ROI analysis is like letting Martha Stewart pick her own jury: there's a clear conflict of interest. In the case of SA, the tool doesn't really account for Microsoft's failure to deliver promised upgrades, product problems and other negatives.

Feel free to work through the Microsoft-backed ROI models. But you, not your rep, should calculate the benefits of any Microsoft package or licensing program. Your own analysis should drive the decision.

Keep vendors' feet to the fire with lots of questions: How did the vendor measure the actual ROI? What happens if ROI fails to materialize? Is it willing to offer an ROI service level agreement (SLA)?

Pre-built calculators, even in the hands of IT, can also be dangerous. ROI calculators often use average data from customers, and don't usually accommodate inputs specific to your shop.

Like ship dates, ROI in reality has a range that depends on any number of circumstances.

Always consider vendor ROI claims to be the best-case scenario.

Tips and Tricks Gather all costs. Look for hidden costs, such as the cost of integration. Only include benefits of features you absolutely know you will use. If your ROI equations assume you’ll implement a feature, then do so and do it well. Know how fast you can get something done. Speed of actual implementation has a direct and sometimes devastating impact on actual ROI. Reality Check: Does your company need the promised benefits? If you can produce more, is there a market? Define a methodology and use it enterprise-wide. Because ROI has a range, you should consider best- and worst-case scenarios. — Doug Barney

3. Beware of ROI Gamesmanship
One of the problems with letting a vendor or sales rep handle the calculations is that small changes in assumptions can dramatically alter payback. While you can, with enough understanding, use certain accounting tricks appropriately, watch out when vendors try them.

One way to change ROI is to lower costs, something vendors are typically loathe to do. Instead, vendors may suggest changing the timing of the costs, spreading out the bills so the initial ROI looks better. They may also suggest delaying some investments; for instance, spending on user training later in the project.

Some may pitch a scaled-down project that focuses on those users that benefit the most, such as aiming a collaboration tool at software developers who are spread across the globe rather than a wider rollout that includes workgroups that already share information efficiently. Another trick is to overestimate indirect or soft benefits. ROI can be dramatically oversold if you don't insist on a correction factor that shows how the numbers change if you don't realize all potential productivity boosts. While all of these techniques can be legitimate, make sure that the techniques employed match your needs, and not just the vendor's spreadsheets.

4. Question Assumptions
The biggest problem with vendor-driven ROI analyses is that the vendor makes the assumptions. While these may be educated guesses, they're still guesses and tend to be overly optimistic.

One basic assumption is that IT will implement new features on time and that end users will use them. But end users, and even IT pros, don't always fully exploit new features.

" I can still recall working with a user on setting up macros in Excel only to find out later he was printing each sheet, manually performing calculations and then placing the answers back in the spreadsheet," notes Michael A. Ferguson, director of technical services for Frontier Futures Inc. "Look at Word. I have yet to see a normal user do anything more complicated than create a mailing list. Where is the real ROI? Even though the product has significantly improved, is it really saving users any time?"

The only real savings Ferguson can point to come from upgrades that offer more stability, and thus fewer help-desk tickets.

On a more fundamental level, IT needs to question whether the product truly does what the vendor claims. That requires internal testing and research into other customers' experiences.

Another assumption is that the ROI of a specific product can be calculated in a vacuum, separate from other technologies. But IT products are by nature interdependent: If one software application with a certain expected payoff depends on another technology that may not be up to snuff, the ROI of the new tool won't be realized.

An understanding of your full environment—and what it will take to integrate the new technology—should be factored into the ROI equations.

ROI is really just mathematical equations based on your assumptions, which are the main variables. Often vendors, and sometimes even IT, has a conclusion in mind, and changes the variables until the desired result is returned. Bad idea. Trust your assumptions and let the chips fall where they may.

5. Define the Problem
Many IT tools solve more than one problem, making ROI calculations more challenging. To perform an accurate ROI assessment, you need to be clear about which problem you want a given tool to address, says Christopher W. Urban, a consultant with systems management consultancy Intrinsic Technologies. Urban ran into this exact issue with a 200,000-plus employee client that was interested in SMS. "There's 20 ways to slice-and-dice how SMS impacts your enterprise. It just depends what angle you're looking from, such as asset management, change control, patch management, etc. Each angle has its own set of intricacies and dollars attached to it," explains Urban.

6. Use ROI to Drive a Better Bargain
ROI isn't just a great planning tool; it also offers terrific negotiating leverage. Let's say you're looking at an enterprise Instant Messaging (IM) system. The Microsoft rep would love to sell Live Communications Server 2003 at more than $1,000 per server (with five client access licenses, or CALS) and an extra $25 per end user.

Hopefully you've run your own numbers, and have a detailed ROI analysis that defines total costs, any increase in user productivity, real quantifiable benefits and hard payback numbers.

Microsoft software prices are part of this model. If the numbers are too high, your payback period is too long and the ROI too small. Make the rep give a price that offers the type of payback your company requires.

"Start off with ‘What is the ROI I need, and what is the price I need to get that ROI?'" suggests Nucleus Research's Campbell.

7. ROI Isn't for Everything
Not every technology is appropriate for an ROI analysis. Some things you just need, even if you can't quantify the payback. You don't have to calculate ROI to know that you need personal computers and telephones for your employees. A major upgrade to existing equipment, however, could warrant a cost justification or two.

And sometimes a gut feeling—and understanding of your business' fundamentals—is more telling than ROI. "Instead of looking for projects that we can spend money on to save money, which I kind of see as an ROI game, I look at what projects we have to get done and try and pick the most appropriate tools for the appropriate price. ROI sometimes is considered, but often times it is the best product that meets the project criteria that drives the decision," says Jeff Kohut, lead LAN analyst for a financial services company.

8. Don't Skimp on Measuring
According to Glomark Corp., which sells ROI consulting and tools to vendors, IT predicts ROI for new projects more than half the time. But a paltry 21 percent bother to measure the actual ROI received afterwards.

If you don't know how past projects worked out, it's impossible to predict the future. There should be a common methodology for measuring ROI, and these numbers should be revisited every time you contemplate a new tool. Work to establish a culture of measuring so you can justify IT expenses, and prioritize projects.

Measuring also helps you make decisions about existing investments. Bad results could prompt you to renegotiate, or even let a license lapse.

9. Beware of "Average" and "Cumulative" ROI
Many vendors toss around figures like Average ROI and Cumulative ROI (cROI). Both are imprecise and can be misleading.

Average ROI is like expressing the average height of a room full of jockeys and NBA stars—it's not an accurate reflection of reality.

Average ROI is meant to define what a group of customers have achieved. But who was actually surveyed and who was left off? According to Nucleus, average ROI only tells what may have happened in the past; it can't tell you what gains you may achieve.

cROI is likewise problematic. To derive cROI, you add up all the benefits from a technology, then divide the sum by the overall cost. Vendors have lots of tricks for fiddling with these numbers, like stretching out the time period. Or they can inflate the ROI by looking at the entire lifecycle of a product. Be wary of these methods: after all, a bus that runs routes for 20 years will show an impressive cumulative ROI, even if another form of transportation like an aircraft may have a better three-year ROI and quicker payback.

A product can offer a better payback later in its life, after the costs have been absorbed, than during the early investment years. As a sum of benefits, cROI always looks more impressive than it is.

The main issue with cROI is that the numbers are difficult to compare to more natural measures, such as the average yearly return on an investment or the cost of capital, Nucleus researchers point out.

Redmond's ROI Dictionary

Cost Benefit: Benefits divided by costs equal total cost benefit usually depicted as a ratio. Return on Investment (ROI): ROI focuses on net benefits. It’s usually the average three-year total benefit divided by initial cost. ROI = ((Net Year 1 + Net Year 2 + Net Year 3) 3 / Initial Costs) x 100. Internal Rate of Return (IRR): The effective interest rate of a technology project. IRR should be greater than the interest rate one would get by simply investing the money used to fund the project. Net Present Value (NPV): The value of a project’s benefits discounted back to the current year. NPV factors in the discount rate or interest rate you could get by investing the money elsewhere, the project costs and the yearly benefits. If the current value of the project is less than the current return on an alternative investment, the project shouldn’t be done. An NPV of $50,000 means that the organization expects $50,000 more payback from a technology than it would gain from an alternative financial investment. Annual ROI: Yearly return, usually given as a percentage. Cumulative ROI (cROI): Total ROI commonly calculated over three years. Average ROI: The average return a set of customers have achieved. Total Cost of Ownership (TCO): This could be considered half of the ROI coin, as it calculates the total cost of buying, maintaining and supporting a technology on a yearly basis, but doesn’t address the benefits. Payback Period: The timeframe, in months or years, it takes for the benefits to pay back the initial project cost. Nucleus argues that payback should occur in less than a year. Once payback has occurred, IT is free to consider replacing the technology with something that offers a better return. — Doug Barney

10. Make a List, Check It Twice
If you're strongly considering a technology, a few extra steps will give you the confidence to move ahead.

First, act like you're hiring an employee and check references. Vendors always have reference accounts. Even though these shops have often been given special deals and are chosen for their rosy results, IT folks normally aren't beholden to a vendor. A phone call can reveal the real ROI story, as well as barriers to overcome to achieve good results.

Analyst firms also can help quantify the general value of a technology based on the analyst's interactions with customers.

And always run your own ROI numbers, which can act as an interesting—sometimes stunning—counterpoint to vendor claims.

Run your model and the vendor's past your corporate finance department to see if any pass muster. You may have the utmost confidence in your ROI numbers, but think of how much more comfortable you'll feel if a real numbers jockey carefully reviews it and agrees.

The bottom line is to take all ROI predictions with more than a few grains of salt, and do your homework to nail down the real numbers.