Abstract

A major contribution to the debate over managed care in the 1990s was a survey carried out by Louis Harris & Associates for The Commonwealth Fund. It was instrumental in casting doubt on the efficiency and effectiveness of manage care. However, there are fundamental problems with the survey which cast doubt on its results. These problems are discussed.

The debate over managed care is conducted in the media, in scholarly journals, in the halls of state legislatures, and in Congress. One of the most hotly argued topic is choice: Does the person insured have a choice of plans and, once in a plan, a choice of primary care physician? Many persons take the position that choice of primary care physician is the single most important factor. It is argued that if one can choose a primary care physician, then the person will perceive the care to be of high quality and will be satisfied with access to that care. Most important, if the perceived care drops in quality, then the person can choose another primary care physician. Freedom of choice of primary care physician is championed as one of the fundamental rights in the US society.

A major contribution to the debate was a study carried out by Louis Harris and Associates for The Commonwealth Fund. It was published by the Fund1 and reported in the literature.2 Generally it presents the argument that freedom of choice is the most important consideration for users of health care insurance. The study reports that persons who have fee for service plans clearly prefer their plans over persons who have managed care plans. It presents a number of other findings which mainly portray managed care plans in a poor light.

The press gave attention to the publication of the results of the study. The Los Angeles Times (circulation of 1,104,651) on July 21, 1995, reported: "Working Americans in managed-care plans are unhappier with their health coverage than are those with traditional fee-for-service insurance.... Leaders of The Commonwealth Fund...said the results raise a cautionary flag about the rush to get more Americans into managed care."3 The Boston Globe (circulation of 500,33) on July 20, 1995, using associated press wire reports, described people in managed care plans as "not as happy with their health coverage as those with traditional fee-for-service insurance...." The story repeated the "cautionary flag" comment.4 The Boston Herald (circulation of 312,779) also on July 20, 1995, reported that the study "casts doubts about how well managed-care health plans are serving their members...."5 A similar story was reported in the July 20, 1995, edition of the Atlanta Constitution (circulation 302,616).6 It was also reported in the July 26, 1995, edition of the Chicago Tribune (circulation 684,366).7 The Globe, the Herald, the Constitution, and the Tribune, it should be mentioned, also carried quotes by persons who said The Commonwealth Fund study was misleading or conflicted with other studies.

Problems

There are three major problems with The Commonwealth Fund study. First, there is a question about how the sample was chosen. It may well be that this method was the only one possible and that it did not bias the results. Second, there is a problem with how the results were presented (using percentages) which may be cleared up if the original numbers (not percentages) are used. And third, the first question asked seemingly destined the outcome of the entire study. These three problems will be reviewed in order.

The Sample

Louis Harris and Associates conducted 25 minute telephone interviews from January 12 to March 27, 1994, with a random sample of 3,348 adults between the ages of 18 and 64 in Boston, Los Angeles, and Miami who had health insurance through their employers. According to Louis Harris and Associates, these three cities represent "mature" managed care markets. The sample was reduced to 1,000 from each city (a total of 3000) with half of them in managed care (67% of them had a choice between managed care and fee for service) and the other half in fee for service (100% of them had a choice). According to Louis Harris Associates,8 this method permitted them to make the comparisons they wanted to make. The methodology was discussed further in what they called Appendix A.

The document published by The Commonwealth Fund1 is an extraordinary document. It contains summaries of the findings, the actual questionnaire with frequency counts in percentages, a copy of the article reporting and discussing the results,2 and even some press clippings about the publication of the survey results. Unfortunately, Appendix A to the original Louis Harris report (which discusses the methodology) was not included.

Davis, Collins, Schoen, & Morris,2 however, do present a rationale for this methodology. They explain that the comparison in the study was to be between "fee-for-service plans, which offer enrollees a relatively unrestricted choice of providers but at potentially higher cost, with managed care plans, which restrict enrollees' choice of providers but offer lower out-of-pocket costs." It is contended that this way of stratifying the sample allowed a comparison of people facing future higher costs with people facing present lower costs while at the same time determining the role of choice. It appeared to clearly present a comparison between cost and choice. Even in the face of future higher costs would people still prefer to select an option (fee for service) in which they were free to choose any primary care physician? And would people with lower present costs still prefer to select the option of fee for service and choose any primary care physician even if it meant higher costs?

There is a problem with this rationale. It conflicts with the findings of researchers who discovered that many persons radically discount potential, future health costs because they are in good physical shape at that moment. They do not anticipate higher future health costs.9-12 In addition it fails to deal with the fact that at times the premiums for fee for service plans and for managed care plans fluxuated in such a manner that at different times they each provided cheaper insurance than the other. It also fails to consider that there are other factors that influence the choice between managed care and fee for service.2,13-21 The fact that fee for service persons all had a choice between types of plans and two thirds of the managed care persons had a choice also complicates matters. The rationale for the way in which the sample was constructed may have biased the results or it may not. The answer is not clear. The fact that the study is six years old or older means that it may never be possible to resolve this issue.

The Presentation with Percentages

The executive summary in the Commonwealth Fund study states: "Overall, the findings of this survey suggest that adults in managed care plans are less satisfied with their health plans and...they would be less inclined to recommend their plans to others."22 It goes on to cite some differences in the level of satisfaction of the two groups. Clearly the thrust of the summary is that fee for service plans are the preferred choice of the respondents.

Under "The Major Findings of the Survey" in the Commonwealth Fund study23 it is stated: "One-fifth (21%) of managed care members consider their health insurance plans fair or poor overall, compared to only 14% of those in fee-for-service plans." An experienced researcher might automatically question if this difference is statistically significant. No statement is made of significance, but only an inference is drawn that it must be a real, not a random, difference.

As Table One indicates, taking the percentages as given there is no statistically significant difference between the two types of plans and the ratings of the plans.

Table One: Plan Rating by Type of Plan Using Percentages

Chi Square = 1.70 with 1 d.f., p = 0.19 (NS)

The original, entire questionnaire is reproduced in the Commonwealth Fund publication24 with the percentages. The percentages for each of the categories (Excellent, Good, Fair, Poor) for this question ("Health insurance plan overall") are given.25 Using these percentages Table Two shows that there is still no statistically significant difference in the two plans. Are Louis Harris and Associates and The Commonwealth Fund distorting the truth when they state that fee for service plans are preferred to managed care plans? No, but they are not careful in their presentation of the results.

Table Two: Expanded Plan Rating by Type of Plan Using Original Percentages

Chi Square = 2.78 with 3 d.f., p = 0.43 (NS)

There are 3000 respondents to the survey, but not everyone answered each question. Since only 99% of the respondents answered this question the sample size for each type of plan is 1485 or 2970 overall. Using the actual numbers (not the percentages) produces Table Three. The relationship between the ratings of the two types of plans are statistically significant in their differences.

Table Three: Expanded Plan Rating by Type of Plan Using the Number of Respondents

Chi Square = 40.91 with 3 d.f., p < 0.000005
Gamma = - 0.20

There are two problems with using the results of Table Three to support the Commonwealth Fund's conclusion about managed care. The first problem is that as the sample size increases, the likelihood of a statistically significant Chi Square result increases. Using a sample of 2970 one would expect a statistically significant result. The second problem is the interpretation of the relationship evidenced by the Gamma statistic.

Goodman and Kruskal's Gamma is a well accepted measure of the strength of relationships in cross tabulation tables. In this case the Gamma equals - 0.20 which means that there is a negative relationship. A negative relationship (for Table Three) is the type which the study says exists. As the relationship moves from poor to excellent, the type of plan moves from managed care to fee for service. In other words, there is an association between being in a managed care plan and rating it lower. At the same time there is an association between being in a fee for service plan and rating it higher.

The sign indicates how the association changes. The number (in this case 0.20) indicates the strength of the association. The Gamma can vary between 0.00 and 1.00. If it is 1.00, there is a perfect (positive or negative) association between the two variables. If it is 0.00 there is no association between the two variables. Although there is no commonly accepted interpretation of the strength of association, generally a 0.20 would be considered not a very strong one. In fact, it would probably be described as a low relationship by most persons. So there is a statistically significant difference, but it is not very strong. A fact which the study does not point out.

The executive summary and the major findings also say that persons in managed care plans are less likely to recommend the plan to their friends. As with the previous case, using the percentages there is no statistically significant difference between the two types of plans. When actual numbers of respondents are included, the Chi Square probability is less than 0.00005, a statistically significant result. The Gamma is - 0.22 which indicates the direction of the association as the executive summary and the major findings indicate. However, again the strength of the association is low, a fact which the study does not point out.

Many of the other relationships which the study presents as establishing the inferiority of managed care plans can be analyzed in the same manner. Does this fact make any difference? Yes, it is well known by researchers that if a large enough sample is obtained, almost any relationship can be found to be statistically significant. That is why it is often preferable to use percentages which are statistically significant in their differences. A larger sample is used in order to have enough respondents to break down the results into smaller and smaller categories. In this case the percentages by themselves were not statistically significant in their differences. Even when the actual number of respondents were used and statistical significant differences were found, the levels of association were not particularly high. The presentation of the results, when analyzed in this manner, do not lend great credence to the main findings of the study.

It should be pointed out that Davis, Collins, Schoen, & Morris26 do list statistical significance in their tables and the use of Chi Square tests, but they do not provide levels of association. Not all of their results, as they note, are statistically significant.

The First Question in the Survey

The study has a more serious flaw, however. The very first question asked of respondents was phrased in such a way as to bias the rest of the replies. The first question was: "Do you have a fee-for-service plan that allows you to go to almost any doctor or hospital and then reimburses you for all or part of the cost OR do you have an HMO or PPO or other type of plan that significantly limits your choice of doctors and hospitals?"27 (emphasis in the original) Since freedom of choice in almost any area is a cherished principle in the US, this phrasing immediately painted managed care in a pejorative light. This flaw is a serious one and the results must be viewed in this light.

In addition to this flaw, many managed care plans actually do have a wide choice of physicians and hospitals. It is the possibility (among other things) that the choice may, in the future, be limited which produces much of the opposition to managed care. In no way can this question be described as fair phrasing and produces a bias for the remainder of the questions.

There are choices offered a person who joins a managed care plan. Harvard Pilgrim Health Care with approximately 982,791 members in Massachusetts is one of the most established HMOs in the country. Out of a total of some 16,000 physicians in the Harvard Pilgrim Health Care network there are nearly 5,500 primary care physicians some of whom are also specialists. A choice of primary care physician is offered when a person enrolls. Existing members can switch among primary care physicians.28,29 Plan members can also self-refer to dermatology, ophthalmology, allergy, mental health, obstetrics, and gynecology.

The Community Medical Alliance in Boston is under contract with the state to provide medical care for persons with severe disabilities. They have about 300 persons with disabilities enrolled in their plan and have 50 primary care physicians. When someone is referred to them the person usually has a primary care physician who does the referral. If that physician is part of their network, there is no change. If the physician is not part of their network, then the person usually contacts another managed care provider where their primary care physician is a member.30,31 According to Kodmur,30 who interviews every referral, only "one in the last two months" had to change from their existing primary care physician. Managed care does not mean lack of choice although caution is advised34 and problems do occur.35 Nonetheless, not only is the wording in the very first survey question slanted, but the factual basis for it is often wrong.

Davis, Collins, Schoen, & Morris32 present policy recommendations based upon the study. Many of these recommendations appear to be quite sound, but they are not necessarily backed up by the study for The Commonwealth Fund. Perhaps another study without the problems cited in this article would support these recommendations.

Postscript

A study funded by The Commonwealth Fund which puts managed care in a pejorative light is quite ironic. Harvard Pilgrim Health Care - or Harvard Community Health Plan as it was originally named - was begun by the late Dr. Robert H. Ebert with a $500,000 grant from The Commonwealth Fund.33 It was the definitive experiment which proved (as did others) that managed care plans can give quality care and cost effective care. Many things changed since Ebert started Harvard Community Health Plan, but the experiment in managed care funded by The Commonwealth Fund stands in stark contrast to the survey of managed care plan enrollees funded by The Commonwealth Fund.

Endnotes

1. Commonwealth Fund, Patient Experience with Managed Care: A Survey (New York: 1995).

2. K. Davis, K.S. Collins, C. Schoen, and C. Morris, "Choice Matters: Enrollee's Views of Their Health Plans," Health Affairs, (Summer 1995): 99-112.

3. Commonwealth Fund, op. cit., p. 108.

4. Ibid, p. 110.

5. Ibid, p. 111.

6. Ibid, p. 112.

7. "Pollsters: Managed Care Users Less Pleased with Coverage," Chicago Tribune, July 26, 1995: 7.

8. Commonwealth Fund, op. cit., p. 39.

9. R.K. Homan, G.L. Glandon, and M.A. Counte, "Perceived Risk: The Link to Plan Selection and Future Utilization," Journal of Risk and Insurance, 56(1,1989): 67-82.

10. R. A. Connor, Consumer Discount Rates for Health and Money in Health Care Cost Effectiveness Analysis, doctoral dissertation, University of Pennsylvania, 1990.

11. T.A. Sheldon, "Discounting in Health Care Decision-Making: Time for a Change?" Journal of Public Health and Medicine, 14(3,1992): 250-56.

12. D.A. Redelmeier and D.N. Heller, "Time Preference in Medical Decision Making and Cost-Effectiveness Analysis," Medical Decision Making, 13(3,1993): 212-17.

13. H.S. Luft, "Trends in Medical Care Costs: Do HMOs Lower the Rate of Growth?" Medical Care, 18(1,1980): 1-16.

14. N. Lythcott, Variables Associated with Health Maintenance Organization Increase in Enrollment, doctoral dissertation, University of California, Los Angeles, 1983.

15. M. Reisler, "Business in Richmond Attacks Health Care Costs," Harvard Business Review, 63(1,1985): 145-55.

16. G.R. Wilensky and L.F. Rossiter, "Patient Selection in HMOs," Health Affairs, 5(1986): 66.

17. F.J. Hellinger, "Selection Bias in Health Maintenance Organizations: Analysis of Recent Evidence," Health Care Financing Review, 9(2,1987): 55-63.

18. W.E. Scheckler and R. Schulz, "Rapid Change to HMO Systems: Profile of the Dane County, Wisconsin, Experience," Journal of Family Practice, 24(4,1987): 417-24.

19. W.D. Paley, "Overview of the HMO Movement," Psychiatric Quarterly, 64(1,1993): 5-12.

20. F.J. Hellinger, "Selection Bias in HMOs and PPOs: A Review of the Evidence," Inquiry, 32(1995): 135-42.

21. A.K. Taylor, K.M. Beauregard, and J.P. Vistnes, "Who Belongs to HMOs: A Comparison of Fee-for-Service Versus HMO Enrollees," Medical Care Research and Review, 52(3,1995): 389-408.

22. Commonwealth Fund, op. cit., p. 42.

23. Ibid, p. 44.

24. Ibid, p. 56-87.

25. Ibid, p. 70.

26. Davis, Collins, Schoen, & Morris, op. cit., p. 102 and note 5.

27. Commonwealth Fund, p. 65.

28. P. Embry-Tautenhen, Personal communication, February 12, 1996. Ms. Embry-Tautenhen is the Manager of Media Relations in the Public Affairs Office of Harvard Pilgrim Health Care.

29. D. Pfeiffer, Personal experience of a 31 year member of the Harvard Pilgrim Health Care network who is also a person with a disability, 2001.

30. L. Kodmur, Personal communication, February 9, 1996. Ms. Kodmur, Member Services Manager, interviews all new enrollees at the Community Medical Alliance in Boston which is under contract with the state to handle Medicaid eligible persons with disabilities.

31. C. Taniguchi, Personal communication, February 12, 1996. Ms. Taniguchi is Contracts and Credentialing Coordinator of the Community Medical Alliance in Boston.

32. Davis, Collins, Schoen, & Morris, pp. 110-12.

33. R.A. Knox, "Dr. Robert H. Ebert, 81; Founded HMO, Was Medical School Dean," The Boston Globe, January 31, 1996: 21.

34. B. Griss, Disability Litmus Test for New Jersey Medicaid Managed Care, People with Disabilities [Magazine], October 2000: 36-39.

35. Lewin, T. Disabled Patients Win Sweeping Changes From H.M.O., New York Times, April 13, 2001; http://www.nytimes.com/2001/04/13/national/13DISA.html.