Kaiser Permanente study finds health care quality report falls short on estimates of risk of post-surgery heart problems
Health care quality measurements are widely used to assess and improve patient care. But a new Kaiser Permanente study suggests the data used by a national group to measure an individual hospital’s quality may lead to inaccurate results.
The study was published in Population Health Management.
The research team started the study after Kaiser Permanente Northern California (KPNC) surgical quality and safety committees reviewed a routine report that showed their hospitals had a higher-than-average rate of patients who developed heart problems after surgery.
“We undertook this analysis because the report we received from the National Surgical Quality Improvement Project (NSQIP) suggested that several of our hospitals had higher risk than others,” said the study’s senior author Lisa Herrinton, PhD, a research scientist at the Kaiser Permanente Division of Research. “Our electronic medical records allowed us to capture information on a much larger group of patients over a longer period of time than NSQIP. Using that rich data set, we found that the complication rate in our hospitals were similar from one hospital to the next, suggesting that care had been standardized, which was reassuring to our surgeons.”
The study included 157,075 adults age 65 and over who had elective, urgent, or emergency surgery not due to a heart problem between January 2017 and December 2019 at 20 Kaiser Permanente Northern California (KPNC) hospitals. Of these, 137,722 (88%) had elective surgery and 19,353 had urgent surgery (within 48 hours) or emergency surgery (within 24 hours). Through an electronic medical record review, the researchers identified every patient who had a heart attack (blood clot blocking an artery to the heart), cardiac arrest (heart stoppage), or similar heart-related problem after surgery.
The analyses showed that after accounting for patient characteristics such as age, gender, smoking history, body-mass index, and chronic conditions, the risk of a post-surgical heart problem was about the same at each of the 20 hospitals.
By looking at every case, and over a longer period of time, we could provide a fuller picture of what was occurring in our hospitals.
— Edward Yap, MD
Next, the researchers averaged odds ratios for post-surgical heart risk from the 3 annual reports they had received from NSQIP from 2017 to 2019. They found that the odds ratios the researchers calculated did not differ significantly from one hospital to the next. In addition, the calculated odds ratios differed significantly — and were lower than — what from NSQIP reported. The researchers said this suggests that risk adjustment performed by NSQIP may be inadequate.
“Cardiac-related problems after surgery are rare outcomes,” said the study’s lead author Edward Yap, MD, an anesthesiologist with The Permanente Medical Group. “Studies have shown they occur in about 1% to 5% of all patients who have moderate to high-risk surgery. Because they are so rare, if you just pick a random sample of cases and you are looking at a small time period — which is what NSQIP does — you might happen to select surgeries where these events occurred, which could provide an inaccurate assessment. By looking at every case, and over a longer period of time, we could provide a fuller picture of what was occurring in our hospitals.”
The Department of Veterans Affairs (VA) developed NSQIP in 1994 to evaluate causes of surgical deaths in VA hospitals, in an effort to improve patient outcomes. After it was shown to be successful, NSQIP was expanded to other hospitals in the mid-2000s. NSQIP remains the only national system to evaluate and report a hospital’s surgical quality and the data it collects is used to identify opportunities for improvement, develop training programs, and set benchmarks for hospitals nationwide.
More than 700 hospitals are currently participating in NSQIP. (Hospital participation is voluntary.) NSQIP reports on multiple aspects of hospital care. The study focused on heart attacks and other heart problems that develop after surgery because, although not common, they are a leading cause of illness and death after surgery.
“When we initially saw the NSQIP report, our first impulse was to make substantial changes to improve outcomes, but we now know that the report was not completely accurate,” said Yap, who is a member of the Delivery Science and Applied Research (DARE) Anesthesia Research Collaborative. “This suggests to me that perioperative and surgical leaders should compare the reports they receive from NSQIP with their own data before implementing any changes. We need programs like NSQIP, but we also need to be aware of their limitations.”
The study was funded by The Permanente Medical Group’s Delivery Science and Applied Research Program.
Co-authors include Jennifer R. Dusendang, MPH, Matthew Solomon, MD, PhD, and Douglas A. Corley, MD, PhD, of the Division of Research; and Kevin P. Ng, MD, Hermant V. Keny, MD, and Bradley R. Cohn, MD, of The Permanente Medical Group.
# # #
About the Kaiser Permanente Division of Research
The Kaiser Permanente Division of Research conducts, publishes and disseminates epidemiologic and health services research to improve the health and medical care of Kaiser Permanente members and society at large. It seeks to understand the determinants of illness and well-being, and to improve the quality and cost-effectiveness of health care. Currently, DOR’s 600-plus staff is working on more than 450 epidemiological and health services research projects. For more information, visit divisionofresearch.kaiserpermanente.org or follow us @KPDOR.
This Post Has 0 Comments