In the 1990s it was common for stock options to be the primary long-term incentive. But some investors and governance experts became concerned that the upward drift in the late-century bull market was automatically leading to outsized financial rewards for executives even when their share-price performance meaningfully lagged that of their peers. The experts sought an approach that de-emphasized the overall market trend and would pay more to those who outperformed peers and less to those who lagged.

Performance for shareholders is typically characterized by TSR, or total shareholder return, which combines price appreciation and dividends calculated as a return percentage on the beginning share price. If you invest $1,000 to buy 10 shares at $100 each at the beginning of the year and the share price rises to $110 at the end of the year and you have received $50 in dividends ($5 per share), then the TSR is 15%. That’s the $100 increase in the value of the stock ($10 x 10 shares) plus the $50 in dividends divided by the $1,000 invested.

Although most of us would be pretty happy earning 15% on our money every year, what if the stock we bought was the worst performer in their industry? What if the company’s peers all earned between 20% and 40%, and we were stuck with a mere 15%? Or what if in a different industry a company earned 15% while its peers all earned 0-10%? Should the managements of those latter two companies be rewarded the same?

The governance experts didn’t think so. If you were lucky enough to be in an industry where everything was drifting higher and you benefitted from this even if you didn’t make good decisions, you should make less. Maybe you were in energy exploration and production and the oil price jumped. Did this mean you should cash in options worth a small fortune?

Or what if you were in chemical production and that oil price jump made all your raw materials more expensive and that put pressure on the share price of your company and your peers. Most people felt you should make more when you were at the top of your peers than when you were at the bottom. In other words: Relative performance matters.

To combat the perceived unfairness of these external influences on executive compensation, the governance experts developed a derivative of TSR they named “relative TSR,” which indicates the degree to which the TSR of a company has or has not outperformed an index or a group of companies.

In practice, what typically happens is the TSR of a company over a three-year period is ranked against either a group of peers or the companies in an index. The company’s relative TSR is then described as a percentile rank, say 65th percentile if the company is ranked 65th from the bottom of, say, a 100 companies.

Though there are a variety of ways companies translate this relative TSR into compensation, what usually happens is management is granted a number of performance share units (PSUs), which are provisional shares in the company’s stock.

One way it works is as follows. At the end of the three years, if a company’s relative TSR is 75th percentile or higher, or what is known as top quartile, executives get twice as many shares as indicated by the number of PSUs. From the median, or the 50th percentile, up to the 75th percentile, they get between 100% and 200% and between the 25th and 50th percentile they get between 50% and 100% of the shares. If their percentile ranking is less than 25th percentile, or bottom quartile, they forfeit their PSUs.

Russell 1000 Studies

To evaluate relative TSR more fully, we conducted two studies of the companies in the Russell 1000 that were public for the full period of each study. Our first evaluation showed how the average of a series of nested 3-year relative-TSR performance tests didn’t relate very well to cumulative relative TSR over the period. In other words, the pay for performance linkage was weak.

We studied the 701 Russell 1000 companies that were public from December 2004 through the end of 2016. Ten cycles were studied, with the first being from the end of 2004 to the end of 2007, the next from the end of 2005 to the end of 2008, and so on through the end of 2016.

Each company’s TSR was compared with the whole group and they were assigned a percentile rank in each cycle. We then averaged these percentiles over the 10 cycles as an indication of how management would have been paid, on average, if their TSR was compared with the whole Russell 1000 each cycle.

On average, the overall TSR percentile rank was off by plus or minus 16% versus the cumulative rank across all companies. Running each cycle through the typical PSU vesting logic mentioned above and averaging the vesting over the 10 cycles provided the average “payout.” Comparing this average of the 10 cycles to what the vesting percentage would be based on cumulative performance shows many distortions.

For example, over the full period, NVIDIA’s cumulative TSR was 1,360%, which was 98th percentile (better than 684 of the 700 companies). That was way up in the top quartile and should have generated a 200% payout. But because of the pattern of the cycle-by-cycle relative TSR, NVIDIA’s average relative TSR was only 45th percentile, and its executives would have only averaged an 88% payout. Not great in terms of alignment with shareholders.

And NVIDIA wasn’t alone. Many companies would have had meaningful distortions. Across the whole group of 701 companies, management teams would have either been overpaid or underpaid, on average, by 45% of their target PSU award. Those are pretty big deviations.

What’s more, even within a single cycle the payouts can vary considerably depending on the start and end dates. To demonstrate this we conducted a second study to examine the 850 members of the Russell 1000 that were public from the start of 2013 through the end of 2016. The relative TSR percentile ranking was calculated for each company during 53 three-year cycles ending as of each week in 2016.

If we measured relative TSR versus the Russell 1000 for PPG Industries during the final five weeks of 2016, the executives of that company would have forfeited their PSUs. If their plan ended in July or August, the average PSU vesting would have been about 100%. If the cycle ended during the four weeks from March 26 to April 16, they would have vested in 200% of the PSU shares. Although we’re intending to reward long-term performance for shareholders, in reality it’s like we’ve given management a lottery ticket.

For one out of every fifteen companies, the reward varied all the way from 0% to 200% depending on which week of the year the performance test was measured. And the average difference from the minimum to the maximum vesting was 86%.

This analysis was done versus the whole index. But what if the test was done by just comparing TSR against the company’s sector? The results get worse. At one in thirteen companies, the reward varied from 0% to 200% depending on which week the performance test was measured. The average difference from minimum to maximum vesting was 88%, which is slightly higher than when measured against the whole index.

Although the principle of rewarding management based on relative TSR seems appealing, in practice it doesn’t work very well. Over an extended period, the average vesting in the three-year relative TSR cycles doesn’t relate well to the cumulative TSR over the full period.

Some managements get paid a lot less than their cumulative TSR should justify and some a lot more. Even within a given year, the relative TSR results vary considerably based on which week we determine the performance test. This seems to invalidate the notion that the relative TSR is a useful tool for aligning pay and performance. And it doesn’t get any better if we measure relative TSR against each company’s sector rather than the whole index. Indeed. it gets worse.

The good news is that there’s a better approach. We have found that companies are better off implementing performance tests based on operating performance. For example, a grid that’s based on revenue growth and either improvement in return on capital, improvement in operating profit margin, or improvement in residual cash earnings (a cash flow based measure of economic profit) can be designed so that PSU vesting relates well to TSR among peer companies and, at the same time, gives management a definitive path on how to drive long-term TSR success through tangible operational goals.

Such a fact-based approach to linking TSR to the operations of a business provides a consistent link between results and reward which will be less fickle and volatile. Such an approach is far superior to relative TSR, which seems sound in theory but doesn’t work well in the real world.

Gregory V. Milano, a regular CFO columnist, is the founder and chief executive officer of Fortuna Advisors LLC, a value-based strategic advisory firm.