What’s behind that five-star rating? A new approach to disclosure policies

Key Takeaways

• Rankings, scores, and certifications can be powerful tools for controlling quality in the market.
• Those disclosures regulate quality by guiding consumers toward better products and reducing demand for low-quality ones.
• Disclosure policies must consider firms’ responses to disclosure, including their incentives to manipulate, cheat, or game the system.
• Optimal disclosure policies can often be simple to implement and easy for consumers to understand.

Quality ratings are everywhere. We rate the safety of cars, the achievements of schools, the performance of hospitals, and the safety of neighborhoods. Everything from the milk in your coffee to the planes flying above depends on hundreds of quality evaluations, assurances, and disclosures.

The unprecedented growth in access to information over the last two decades has made quality disclosure an essential part of consumers’ decision-making process and a valuable policy tool for regulators, retailers, distributors, and other market intermediaries. Yet the design of disclosure policies remains widely debated, and their failures plaster the news every so often.

The most common challenge is that firms often react to disclosure in unintended ways.

Columbia University was recently lambasted for misrepresenting its quality to U.S News & World Report for its highly influential ranking of universities. Columbia dropped from No. 2 to No. 18 and then to the abyss of the “unranked.” Class-action lawsuits seeking damages from students followed shortly after, and the U.S. News college ranking system has sustained widespread criticism and boycotts by several schools.

The nursing home industry has faced similar criticisms for decades, with articles claiming that facilities manipulate the ratings given to them by the Centers for Medicare and Medicaid Services (CMS). In 2021, California sued Brookdale Senior Living, one of the largest nursing home systems in the country, for such manipulation. Even Dieselgate — Volkswagen’s emission scandal — falls under this category of manipulation. 

Though harmful, these responses show the signs of an untapped policy lever. Firms clearly value these signals of high quality. Otherwise, they would not risk litigation and public shaming to manipulate them. Therefore, it stands to reason that in a system with accurate quality measurements and the right incentives, they would be willing to invest in true quality improvements rather than face potential losses.

Evidence suggests that most rated firms in food industries, health care, and education already do so, although sometimes not in the magnitudes or the types of qualities we would hope. The evidence points toward a need for new disclosure policies that account for the incentives they create for firms. Fundamentally, these new policies must judicially disclose quality, trading off the value of additional information for consumers’ decision-making and the potential effects on firms’ investment incentives.

Accounting for supply-side responses can lead to very different answers to the disclosure policy problem. Consider an idealized scenario in which a consumer has the option to buy a widget of an unknown quality from a single firm. The regulator measures the quality before any purchase happens and must decide what to disclose. If we ignored supply-side responses, the optimal policy would be to give the consumer all the available information. This way, the consumer never makes a mistake when buying the product.

The problem, however, is that markets are often imperfect, even when consumers are fully informed.

Consider that the socially optimal quality is higher than what the firm would deliver to the fully informed consumer. (We will return to why this happens in the next paragraph.) In this scenario, the optimal policy would be starkly different from the previous solution: a simple sticker on the widget that certifies the product’s quality meets or exceeds the desired standards. We need only to assume that quality is costly for the firm to produce and that consumers know that. A consumer that sees the quality certification label on the product would rationally assume that the firm has invested in precisely the efficient level of quality; after all, the firm has no incentive to invest in a quality that no one will find out about. A consumer who sees an uncertified product would assume the quality is minimal and that the product offers little to no value. As the firm has operating costs to cover, its price for a low-quality product would have to be higher than consumers’ minimal willingness to pay, and no product would be sold. Therefore, through a simple sticker, the regulator has coordinated consumers on higher quality, offsetting the market’s distortion. This simple idea is illustrated in Figure 1.

The fundamental problem raised above is that firms often do not invest socially optimally in quality, even when consumers are fully informed. This can happen for various reasons. First, market power creates a wedge between a firm’s revenue from increasing its quality and the value consumers derive from this improved quality. Intuitively, a firm cannot extract all the value that its quality creates for each consumer (unless it’s the same for everyone or it can perfectly price discriminate) and therefore does not invest as much as the consumers or a welfare regulator would like.

A second common source of distortions is subsidies. If consumers pay subsidized prices as in health care or education markets, or firms have subsidies to investment as in energy markets, the government would bear part of the cost of quality. As firms’ profits are unaffected by the government’s burden, they will often invest excessively. Other scenarios can lead to similar effects. For example, pharmaceutical firms might over-invest in quality because they also benefit from its effect in deterring competitors from researching similar drugs, leading to fewer and more expensive options. Charities might underinvest in raising awareness of a cause, as aware donors might choose to donate to other competing charities. Disclosure policies can marshal demand to offset some of these distortions, leading consumers toward better choices of higher quality.

Theoretically, these ideas about the coordinating power of information have been understood for many years.

Economists have been studying information design, at least dating back to the earlier model of communication and cheap talk (Crawford and Sobel, 1982) and with great developments in the last decade, summarized by Kamenica (2019). Empirically, economists have primarily focused on measuring the effect of disclosure in various markets, including schools (Allende et al., 2019), hospitals (Dranove and Skefas, 2008), nursing homes (Feng Lu, 2012), appliances (Houde, 2018), and grocery items (Barahona et al., 2023). A more general review of the literature on disclosure can be found in Dranove and Jin (2010). However, only recently have the two strands of literature been connected to understand how to design optimal disclosure policies in practice.

Theory to Practice: Scoring Design in Medicare Advantage

My recent work (Vatter, 2022) applied these ideas to study the optimal design of the Medicare Advantage Star Ratings. Medicare Advantage is a large and growing fraction of the Medicare retiree and disabled insurance system, currently serving 28 million people — 48 percent of those enrolled in Medicare. Beneficiaries of Medicare Advantage choose health insurance plans offered by private insurers under highly subsidized premiums and generous mandatory coverage requirements.

Differences across plans in risk protection (such as deductibles, coinsurance rates, and copays) are wide and displayed with great detail for consumers when choosing. However, plans differ vastly in network structure, access to care, preventive medicine practices, management of chronic conditions, and service quality. This heterogeneity in quality has been shown to affect mortality and cost billions to taxpayers (Abaluck et al., 2021; CMS, 2016).

But the complexity of the factors that contribute to this quality heterogeneity — and the fact that plans’ quality is constantly evolving — make it nearly impossible for consumers to assess it properly when enrolling. To help consumers, CMS measures each plan’s quality and scores it on a scale from one to five stars, with halfstar increments; each plan’s score is a mapping from its quality into stars, and the optimal disclosure policy problem is to find the map that would maximize total welfare in the market.

Figure 1. Regulating a monopolist with certification.

Notes: This figure exemplifies the effect of regulating a monopolist producer by controlling consumers’ information about product quality. The monopolist chooses a quality level on the horizontal axis and then sets a price optimally (not shown), resulting in the blue profit curve. The green line shows the total welfare generated from these choices. The panel on the left shows the two informational extrema; in both cases, quality is underprovided. On the right, the regulator shows consumers a one-star rating when quality is inefficiently low and two stars when it’s at least efficient. This disclosure leaves the monopolist with two options: Invest zero or the efficient amount. The efficient outcome is achieved as long as the monopolist’s profits are higher at the efficient point than at zero.

An invaluable feature of Medicare Advantage is that its disclosure policy has varied yearly since 2009. In acts of experimentation and attempts to improve its disclosure system, CMS has changed the mapping from quality to stars, revealing both demand and supply responses to scores.

Using data about new enrollees, I found that consumers are about 20 percent more likely to choose a five-star plan than a financially equivalent two-star one. Using data on each plan’s measured quality and changes to the disclosure policy, I found that insurers respond quickly to changes in the policy, investing more aggressively when changes put them at a higher risk of losing scores. Insurers also invested more in quality dimensions (access to care, clinical outcome quality, preventive care, etc.) with a more significant weight or reward in the scoring system. The evidence revealed that Medicare Advantage is an ideal setting for disclosure policy: Consumers value higher-scoring products, and insurers react to scores meaningfully and in ways coherent with their profit incentives.

Using this variation in design, data on enrollees and plans, and an economic model of the market, I set to assess the market frictions a new optimal policy must tackle. The findings revealed that in the 2009- 2015 period, the Star Ratings were only moderately informative. Consumers would have been willing to pay about three monthly premiums in advance to obtain full information about plan quality and avoid suboptimal choices. The most significant driver of this distortion was misclassification: Some plans that consumers preferred were given lower scores. This distortion occurs because while both CMS and consumers agree that a better plan in every dimension of quality should be scored higher, they disagree on how to rank plans with mixed qualities. In practice, CMS tends to prioritize plans with good chronic conditions management, while consumers would prefer them to score higher plans with access to good hospitals (in terms of clinical outcomes).

The second key distortion uncovered was in quality provision. Medicare Advantage markets are highly concentrated, with UnitedHealthcare, Humana, Blue Cross Blue Shield, and Aetna/CVS Health accounting for more than 70 percent of the national market share. This concentration leads firms to under-invest relative to the social optimum.

The results revealed that while investment was, on average, insufficient, it was sometimes excessive on quality dimensions that were given a high reward in the scoring design but that consumers value little. Therefore, the supply distortions mirror the demand ones, and both are connected to how the disclosure rewards different quality dimensions and how different qualities are assigned stars.

Creating a New Policy

The paper develops a new methodology by which these measurements and data are used to solve for a new optimal policy. This novel empirical scoring design methodology bridges the gap between theory and practice and delivers policies that trade off alleviating consumers’ asymmetric information and regulating quality. The new policy uncovered is estimated to increase consumer surplus by nearly two and a half monthly premiums, generating billions in welfare gains, all using a simpler scheme than what is currently in place. The new design can be considered a simple weighted average of quality, with four cutoffs, similar to a school grade or a discrete five-star system. To address the under-provision of quality, the new design sets the threshold to obtain two stars (or a passing grade) much higher than it currently is.

Under the status quo, the new design would fail nearly 62.6 percent of all plans. However, estimates of firms’ responsiveness to changes in scores suggest that, in practice, only 26.5 percent would receive one star. The others would be pushed to invest more, increasing quality substantially. The four additional star ratings would partition the higher quality range, leading most consumers toward the higher-scoring products. The weights used in this weighted-average scheme are the most important feature of the design. These weights are carefully chosen to balance consumers’ preferences for different quality dimensions and firms’ costs of producing those dimensions.

This new policy reveals several lessons for disclosure policy design for Medicare Advantage and beyond. First, it shows that quality disclosure can be considered a quality regulation policy. This is because more than half of the welfare gains of the design stem from the effect on quality rather than on information. This would be further augmented if one considered the possibility that consumers might have difficulty processing complex quality information or understanding what scores mean. In this case, scores can coordinate consumers to offset supply-side distortions but can do little to alleviate consumers’ asymmetric information.

Second, it shows no fundamental tension between the design of policies under a more behavioral perspective on consumers’ decision-making and a purely rational one. In this article, consumers are fully rational, yet the optimal design is extremely simple. Strikingly, a binary certification of quality (like a USDA organic label) — which would be the simplest design from a cognitive standpoint — would generate 94 percent of the optimal welfare gains.

This leads to the third observation, which is that the most important feature of a scoring system is often how it aggregates quality dimensions rather than how finely they partition this into stars, letters, or any other label. This challenges a longstanding view that a score with more distinct values is more informative. Consumers are better off in a world with fewer scores that rank products similarly to their preferences and coordinate them with other consumers to create proper incentives for firms to invest.

Finally, the results provide a word of caution about paternalism in policy design. The wedge between how CMS rewards quality in scores and consumers’ preferences could be considered an attempt to nudge consumers toward what CMS thinks are better choices. For example, the over-representation of quality measures associated with managing chronic conditions could be due to a perception that consumers do not pay enough attention to the deterioration of their diabetes, osteoporosis, or heart disease.

Designing quality scores that reward such aspects could be well intended but, in practice, it erodes the value consumers would otherwise place in the scores, often leading them to ignore the ratings altogether. To make things worse, if consumers ignore scores, firms have little incentive to obtain high ratings, and thus this nudging results in a significant loss of information and quality for consumers. This loss is so significant that a subsidy to quality investment that resulted in insurers spending only five cents of every dollar given would outperform this attempt to nudge consumers with information.

These results highlight the great potential of optimal disclosure policies. Scores that account for supply-side responses and with consumers’ preferences in mind can alleviate considerable barriers to information and act as robust quality regulations — all of this at virtually no cost to the regulator beyond the quality measurement. Regulators have a great advantage in implementing disclosure, as measuring quality often requires a combination of legal authority and scale.

Moreover, for scores to be credible to consumers, whoever discloses them should not have conflicting incentives with consumers. Putting these ideas into practice is often difficult, but new methodologies are emerging. Improvements in computation, economic theory, and empirical methods are paving the way toward solutions for increasingly challenging settings, including those with manipulation, voluntary disclosure, or competing sources of information. Simple, welldesigned scores can change the landscape, alleviating the burden of difficult choices and stimulating greater quality investments from firms.

About the Author

Benjamin Vatter is a SIEPR postdoctoral fellow working on topics at the intersection of industrial organization and health economics. His work currently focuses on quality competition in health insurance, the optimal design of informational policies, and merger effects in health care markets.

References

Crawford, V. P., and Sobel, J. (1982). Strategic information transmission. Econometrica: Journal of the Econometric Society, 1431-1451.

Kamenica, E. (2019). Bayesian persuasion and information design. Annual Review of Economics, 11, 249-272.

Allende, C., Gallego, F., and Neilson, C. (2019). Approximating the equilibrium effects of informed school choice (No. 2019-16).

Dranove, D., and Sfekas, A. (2008). Start spreading the news: a structural estimate of the effects of New York hospital report cards. Journal of health economics, 27(5), 1201-1207.

Feng Lu, S. (2012). Multitasking, information disclosure, and product quality: Evidence from nursing homes. Journal of Economics & Management Strategy, 21(3), 673-705.

Houde, S. (2018). How consumers respond to product certification and the value of energy information. The RAND Journal of Economics, 49(2), 453-477.

Barahona, N., Otero, C., and Otero, S. (2023). Equilibrium effects of food labeling policies. Econometrica, 91(3), 839-868.

Dranove, D., and Jin, G. Z. (2010). Quality disclosure and certification: Theory and practice. Journal of economic literature, 48(4), 935-963.

Vatter, Benjamin (2022). Quality Disclosure and Regulation: Scoring Design in Medicare Advantage. Available at SSRN: https://ssrn.com/ abstract=4250361 or http://dx.doi.org/10.2139/ssrn.4250361.

Abaluck, J., Caceres Bravo, M., Hull, P., and Starc, A. (2021). Mortality Effects and Choice Across Private Health Insurance Plans. The Quarterly Journal of Economics, 136(3), 1557–1610.

CMS (2016). Quality Strategy. Technical report.