Kaiser Permanente researchers develop tool that could automatically pull relevant factors already available from a patient’s health record
For patients over 65 who may need emergency surgery, clinicians often need to make difficult judgment calls, since older patients tend to have more surgical complications. A new predictive model developed by Kaiser Permanente researchers could help clinicians assess risk more objectively and quickly.
The Assessment of Geriatric Emergency Surgery (AGES) model was described in a study published in BMC Anesthesiology. AGES was developed using machine learning, a type of artificial intelligence that takes advantage of massive computing power to process large amounts of patient data to create a model that works well — it has strong predictive value.
The model calculates a risk score based on up to 76 predictors on a particular patient. A doctor can take that risk score into account along with all the medical, social, and other factors used to decide whether to recommend emergency surgery for an older patient.
“Our previous research showed the patients with highest risk of a bad outcome were those getting emergency surgery, and those 65 or older had the worst outcomes,” said lead author Edward Yap, MD, an anesthesiologist with The Permanente Medical Group. “We thought, wouldn’t it be great to have a tool that is integrated into the electronic health record, provides valuable information, and saves doctors’ time.”
“A score that predicts risk of downstream health effects doesn’t necessarily preclude the patient from having surgery,” Yap continued, “but it gives the clinicians an objective data point to discuss options with the patient. It’s better for shared decision-making between clinicians and patients.”
The model performed well in testing, producing a predictive value of about 0.8. A value of 0.5 is like flipping a coin, Yap said; anything over 0.7 is good and over 0.8 is great. AGES tested similarly across races and ethnicities, and across male and female patients.
“We think AGES performed great in our testing,” said Yap, who is a member of the Anesthesia Research Collaborative, a specialty research network with the Delivery Science and Applied Research group housed at the Kaiser Permanente Division of Research (DOR). “How it would perform in the real world depends on how it is deployed.”
The model is not currently incorporated into Kaiser Permanente medical care, though it has great potential, the study authors said. “Our integrated setting is the ideal place to both develop and implement this kind of innovative tool to support patient care,” said senior author Mary Reed, DrPH, a DOR research scientist who directs its Virtual Care Research Program. “This tool is well-designed by clinicians using their everyday experience, and its development reflects feedback from the views of several medical specialties’ leaders.”
Many factors taken into account
Among the 76 factors that can be automatically pulled from the patient record and included in the calculation were age, sex, body mass index, certain medical diagnoses, delirium score, lab results, and vital signs.
AGES was designed to predict an array of potential post-surgical conditions in the following 30 days, including heart attack, stroke, pulmonary embolism, renal failure, sepsis, and death.
The model was developed using data from 66,262 patients age 65 or older (average age 78) who had urgent or emergency non-cardiac surgery at one of 21 KPNC hospitals between 2017 and 2020. Data from 80% of the patients was used to design the model, and it was tested against data from the other 20%.
AGES could be used efficiently by doctors if it was embedded in the medical record and functioned automatically or with a simple request, the authors said. Currently, doctors may use other external calculators on a separate website, which takes time to type in patient information to get a risk score. Also, other surgical risk scores may not be developed specifically for older patients, or be able to predict multiple outcomes beyond mortality or cardiac arrest.
“When you have to spend an extra 3 to 5 minutes inputting data about a patient with an urgent medical problem, busy surgeons and clinicians may be reluctant to do that,” Yap said. Without a calculated risk score, doctors rely on their own experience, which may involve biases and produce inconsistent choices from one doctor to the next, he said.
Implementation of predictive models — moving them from the theoretical to the real world — is often challenging given the complexities of hospital care with many medical specialists, nurses, and other clinicians needing to align behind care decisions, Yap said. Also, predictive models and algorithms require tending to maintain their predictive value over time and as conditions change. Any health care organization adopting a predictive model must commit to maintaining it. Kaiser Permanente has adopted other predictive models that have resulted in life-saving hospital processes.
“Kaiser Permanente is an ideal setting for physicians to carefully design the implementation of these models into clinical care pathways, and to make sure we are monitoring patient outcomes,” Reed added.
The model is flexible in terms of what kind of risk score it produces for the clinician to use, Yap said. “It depends on whatever the clinicians want. We could set the output as a percentage risk of a bad outcome, or as a relative risk comparing this patient with a pool of other similar patients.”
The predictive model’s development is tied to a KPNC initiative called the Senior Surgical Care program, which focuses on 4 key areas of geriatric surgical care: goals and decision-making, avoiding delirium, preventing functional deterioration, and nutrition and hydration. The program, in place at 4 KPNC hospitals, has received a high-level geriatric surgery designation from the American College of Surgeons.
“One of the program’s objectives is to have better shared decision-making,” Yap said. “We hope this tool could be part of that.”
The study was funded by The Permanente Medical Group Delivery Science and Applied Research program.
Additional co-authors were Jie Huang, PhD, of the Division of Research; and Robert W. Chang, MD, Bradley Cohn, MD, Joshua Chiu, MD, and Judith C.F. Hwang, MD, MBA, of The Permanente Medical Group.
# # #
About the Kaiser Permanente Division of Research
The Kaiser Permanente Division of Research conducts, publishes and disseminates epidemiologic and health services research to improve the health and medical care of Kaiser Permanente members and society at large. It seeks to understand the determinants of illness and well-being, and to improve the quality and cost-effectiveness of health care. Currently, DOR’s 600-plus staff is working on more than 450 epidemiological and health services research projects. For more information, visit divisionofresearch.kaiserpermanente.org or follow us @KPDOR.
Comments (0)