How to Make a Forecast When You Feel Like You're Being Asked to Take a Guess: Sensitivity Analysis

By Chris Bernard, March 22, 2012

This is the latest in an occasional series of posts by Henry Quinn, whose insight is the reason we invite him to blog for us--not the fact that he's our executive director's husband.

Here's a thing that no one should ever say, but that people sometimes do:
 
"I'd like to make a forecast about how some effort is going to impact donations, but there's something important that I'd just be guessing at. Oh well—I'll just guess."
 
The reason you shouldn't say this is what those of us in the business call a ‘sensitivity analysis.’ It's a method for making maximum use of the things you do know, while still allowing for one or more degrees of uncertainty—all without forcing you to commit to an unfounded guess about any of them. Want a simple, single-unknown example?
 
You're planning on-site changes to increase the visibility of your membership renewal page. You know the following:
 
  • In 2011, you had 1M visitors.
  • 4 percent of those saw the membership renewal page.
  • Of those who saw the old page, 46 percent renewed successfully.
  • The average renewal membership is worth $45.
  • Finally, you're going to be implementing these changes in July, and 78 percent of your visits come in the second half of the year.
 
Notice that all these things are pretty easy to measure or find out about 2011, and you actually know a LOT. In fact, there's only one thing, a small, isolated thing, that you don't know. This is key to a sensitivity analysis—you need to isolate the thing that you don't know, and define it very, very precisely. If you throw up your hands at “Well, these efforts are going to do SOMETHING,” you can't use this method. You're also not trying very hard, and quitters never win. The changes in this example aren't going to do SOMETHING, they're going to do something quite SPECIFIC: increase the rate at which visitors to your site visit the renewal page. That is the one and only figure you don't know.
 
And with that in mind, I can tell you EXACTLY how the changes will impact your revenues in 2012*: the revenue impact is going to be equal to the number of visitors times the percentage that see the renewal page times the percentage of those who renew times the value of a renewal times the percentage of potential renewals that will be affected by the changes given when they'll be implemented times some factor representing the increase in the rate at which visitors look at the renewal page. To put a point on it, that's 1M X .04 X .46 X $45 X .78 X a factor, or $645,840 times that factor.
 
What we've done is box in your uncertainty, and determine precisely what impact the unknown is going to have on your eventual estimate—you've determined the SENSITIVITY of your forecast to a single, understandable unknown. The only thing we have to do now is figure out what a reasonable range of guesses for that factor is, and then apply the $646,000 figure to get a range of reasonable impacts. Rather than making a single guess for a large and complex unknown, you can describe a relationship, within a limited range, based on a very specific unknown. That analysis looks like this:
 
If the increase in renewal form view rate is this...
Then the increased revenue will be this.
1 percent
$6,458
2 percent
$12,917
5 percent
$32,292
10 percent
$64,584
20 percent
$129,168
 
Now THAT'S a useful table. How much is it going to cost you to make these changes? If the answer is $10K, I'd go ahead and make them—if they provide even a 2 percent lift in renewal form view rate, they'll pay for themselves in the first year. If the answer is $75K, I might skip them—you'd need a view rate increase between 10 percent and 20 percent, which feels pretty aggressive, to make them worthwhile.
 
At a higher level, I'd make two other points about how this table can help you make better decisions. First, it's easy to communicate, upwards and outwards. It describes a very straightforward relationship, with exactly one moving part, and so discussions about it can be very focused.
 
“Ah,” says your Executive Director. “So you're recommending we go forward with these changes because they'll bring in an extra $32,292? Well, 5 percent sounds like a reasonable level of lift. Let's do it.” It's not that $32K feels high or low—there's too many ingredients in that stew for it to be a simple call. It's the 5 percent lift that's up for discussion, and that's a much simpler consideration. Everything else follows automatically from that.
 
Second, when your forecast eventually breaks, this table makes it easy to diagnose where it broke. Did you see a 5 percent lift in view rate, but miss that $32K? Then something about your assumptions was wrong, but the changes themselves had the desired impact, which was to increase a very specific metric. Did the relationship hold, but the lift in view rate not come in? Then your prediction about the impact of the changes was off on the high side. This is so much more instructive than simply saying, well, we thought it would come in at $32K, and it didn't.
 
(Which really illustrates a third sense in which this is good practice: it establishes, precisely, how you're going to measure the impact of the change, months in advance of its implementation. Think about that—knowing, a year before you'll be asked, what you're going to do when someone asks you, “Did that work?”)
 
So, uncertainty exists. If you believe it's lurking all around you, spread across everything equally, it becomes really hard to predict anything. But if you believe that you know most things pretty well, and that the things that you don't know can be narrowly defined, a sensitivity analysis can be a really helpful method for wrangling it.
 
 
* Yes, yes—you're expecting other changes in 2012, related to other changes in what you do, how you do it, or external circumstances. These changes, they aren't true knowns, but you've probably had to think about, and budget them, as part of some larger planning process. You've already decided what they're going to be, and so they can be applied here as though they were knowns. We're also (secretly) assuming that the mentioned rates are constant throughout the year. If you wanted to, you could easily eliminate this assumption by actually measuring those rates from July to December, or measuring them relative to the first half of 2011 and extrapolating from current measurements in the first half of 2012.

  

Trackback URL for this post:

http://idealware.org/trackback/2721

Comments

Brilliant

by Holly Ross (not verified) - 2012-05-09 13:48

Chris - This is a brilliant post that should help many many decision makers feel more comfortable making tech decisions. There are lots of platitudes about how to get your boss on board. This is real, concrete advice. 

Incorporating Probability

by Sean Power (not verified) - 2012-04-04 11:03

I LOVE this--very useful for fundraisers.

I might add something about using past performance to indicate future performance to add confidence in each scenario.

It's likely that this isn't the first on-site change made by the hypothetical fundraiser. Chances are, they've made several changes over the years. While the goal, scope, and fruitfulness of the intended impact might be different with each on-site change, adding to the mix the performance history of the decision-maker will help that upwards and outwards communication loop you talk about.

Suppose one on-site change per week, big or small, over the past three years (not unreasonable, based on the prominence of online fundraising in some organizations). That's about 150 changes. With each change, the fundraiser can look back and evaluate to what degree they achieved their goal--1 percent lift, 2 percent lift, ...,  20 percent lift, in the observed metric (isolated unknown).

Now the fundraiser has probabilities for each outcome, based on her own decision-making ability. With these probabilities, she can calculate the odds of each outcome actually happening.

For example, the fundraiser might find that in the last 150 decisions she made:

5 decisions created 1% increase in the observed metric.
15 decisions created 2% increase in the observed metric.
75 decisions created 5% increase in the observed metric.
40 decisions created 10% increase in the observed metric.
10 decisions created 20% or more increase in the observed metric.
5 decisions had negative or no effect on the observed metric.

Here, you can see that in her last 150 decisions, 125 resulted in 5% or greater increase in the observed metric (83%). It is therefore likely that the fundraiser will see similar results this time around: there's an 83% chance that the increase in revenue will be $32,292 or greater. If it cost $10K to implement the change, it's likely the project will succeed. Even if the project cost $30K to execute, there's still a 33% chance of the change bringing in $64,584 or more, so it could be worth the risk.

If the numbers were reversed, and 125 of her last 150 decisions resulted in 1-2% or less increase, then there would be an 83% chance that the increase in revenue would be $12,917 or lower. If it cost $10K to implement the change, I might evaluate some of the intangible risks for a project that is likely to have low ROI. If the project cost $75K to implement, it's likely to not happen, based on the low probability of the scenario in which the change creates 20% lift.

I've also posted my response on my <a href="http://advertisinginsights.blogspot.com">marketing and fundraising blog</a>.