Project Presentation Flashcards
Risk prediction models are necessary to reduce the number of intraoperative complications that occur – up to 44% of patients undergoing surgery experience a complication as a result
Up to 44% of patients undergoing major surgery experience a complication as a result of this. This is why risk prediction models are necessary, as they can identify – based on other related data – the patients that are at risk of experiencing a complication as a result of surgery. This prediction is key for the shared decision-making process between patient and anaesthetist, as surgery can be decided against/precautionary measures can be taken if the risk is deemed high.
Current tools use regression-based modelling
Out of the risk prediction tools that are currently available to use, multivariate logistic regression is the method use to generate the predictions. This is sufficient for the modelling of simple datasets that don’t contain too many features, but for datasets with more complex non-linear relationships between features – machine learning, also known as ML, should be used.
ML can model more complex datasets but carries issues with implementation
Saying this, there are problems with the implementation of ML models into clinical settings – mainly associated with the interpretability of the predictions made by the model. Without sufficient reasoning, patients and clinicians alike have trouble trusting the decisions of such models. For this reason, the performance of more interpretable regression-based approaches should be compared to the black-box ML models to determine whether the transparency issues are outweighed by exceptional performance.
Current tools don’t focus on intraoperative complications
All of the currently implemented tools focus on 30-day mortality as an outcome and none focus on intraoperative complications that can be managed by anaesthetists during the operation. Intraoperative risk prediction is something that holds substantial potential. For example, if intraoperative hypotension can be predicted, then drugs can be administered that counter the effects – stopping the problem at the root rather than letting it manifest itself into something more serious.
Clear opportunity for research
This project aims to fill a gap and build risk predictors (that utilise both regression and ML based approaches) capable of predicting the chance of a patient experiencing at least one intraoperative complication, using preoperative data.
Currently implemented tools include P-POSSUM, NELA, SORT and ACS NSQIP
Currently implemented risk prediction models include Portsmouth Physiological and Operative Severity Score for the enUmeration of Mortality and Morbidity (P-POSSUM), National Emergency Laparotomy Audit Calculator (NELA), Surgical Outcome Risk Tool (SORT), American College of Surgeons National Surgical Quality Improvement Project Universal Surgical Risk Calculator (ACS NSQIP).
Mainly focus on 30-day mortality
All of these tools consider varying preoperative variables, such as age and haemoglobin levels as you can see in the P-POSSUM tool, to predict 30-day mortality. ACS NSQIP in addition to mortality also predicts 13 other post-operative outcomes such as renal failure and pneumonia. ACS NSQIP and P-POSSUM gives estimates for general surgical procedures, while NELA specifically focuses on emergency bowel surgery and SORT is exclusively for non-cardiac, non-neurological inpatient surgery. P-POSSUM was adapted from POSSUM to improve performance, while NELA built upon P-POSSUM as it overestimated mortality on the NELA data when predicted risk was over 15%.
ML approaches have been tested but none have been implemented yet
ML approaches have been used to develop tools that haven’t been implemented yet. Most of these focused on postoperative outcomes with very few making use of intraoperative variables to predict, in real time, intraoperative complications. Intraoperative outcomes that were predicted included bradycardia and hypotension, none of the literature focused on the prediction of intraoperative complications using solely preoperative data, however.
Random forests and gradient boosting were the models that performed best – AUROC of above 0.9 for best models
Random forests and gradient boosting models were the two models that were tested in the literature the most, with the best models of this type achieving an area under the receiver operator curve (AUROC) of above 0.9. Some studies utilising ML made comparisons to simple regression-based approaches, the majority of which reported superior ML performance. A couple, however, reported similar performance between the two approaches, highlighting the need for comparison. Many of the ML models lacked external validation, something which is necessary before they can be implemented into a clinical setting. Additionally, many of the current ML studies are reported poorly, failing to fully adhere to transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines.
The activity survey, conducted as part of NAP7, gave rise to a dataset that could be exploited for the prediction of intraoperative complications by ML methods
The royal college of anaesthetists conducted their 7th National Audit Project (NAP7), with the overall focus on cardiac arrest. As part of this, an activity survey was undertaken over 4 days in all participating hospitals, to provide context to the case registry, another part of NAP7. The first page of the activity survey is shown here to demonstrate how it was structured, it’s not necessary to read all of the text though.
Dataset contains 62 features for 24,172 patients
This activity survey gave rise to a dataset containing 62 features for a total 24,172 patients, some of these relate to intraoperative complications experienced by each patient including septic shock and major haemorrhage, others are supporting information such as age, gender and BMI to name a few. According to the survey, 1 in 18 cases had some sort of intraoperative complication demonstrating a need for the forecasting of intraoperative complications. This dataset holds promise for the building and training of ML based models tasked with predicting intraoperative complications.
Comparison of ML and regression-based prediction of at least one intraoperative complication
The size of the dataset suggests that non-linear relationships between features will be present, highlighting the need for ML approaches. As previously mentioned, there are problems with the implementation of ML models due to their black-box nature. Hence, the performance of ML based risk predictors for predicting at least one intraoperative complication using preoperative data should be compared to that of simple regression-based models to determine whether the lack of interpretability is negated by a superiority in performance, if there is a superiority at all.
ML approaches that will be tested will include random forest, gradient boosting and artificial neural networks
Numerous ML approaches will be tested including random forests, gradient boosting and deep learning techniques such as artificial neural networks.
Scope for prediction of specific complications (such as cardiac complications) and also predicting probability of experiencing more than one complication in a procedure
After testing the performance of both ML and regression approaches at predicting at least one intraoperative complication, there is a scope for comparing the performance of intraoperative risk prediction using both preoperative and intraoperative data with that using just the former. Alternative avenues to explore are the prediction of more specific complications such as cardiac complications as well as building models that can determine the probability of experiencing more than one intraoperative complication.
What is my plan for addressing the problem?
I will spend the first 3 weeks cleaning the data and preparing it for analysis. Weeks 4 and 5 will be spent building the initial models for the prediction of at least one intraoperative complication with the following 2 weeks being spent on the development of the adapted models previously described. I will spend week 8 and into week 9 analysing the findings from each of the different models and drawing conclusions about their performance and I’ll begin writing the report alongside this to ensure that I adhere to time constraints.
