When the COVID-19 pandemic took hold across the world, it promptly became clear that testing would be critical in reducing the spread of the virus. Swabbing for lateral flow tests eventually became part of our daily lives. But the tests are expensive, wasteful and are not always accurate. Experts have estimated that 39 tonnes of plastic waste was produced in England each week between May and November in 2021 from lateral flow test kits. Artificial intelligence researchers saw this as an opportunity to explore how machine learning (ML) models could be used to detect COVID-19 in a cheaper, easier and less wasteful way.
The use of sound recordings, known as audio signals, in detecting respiratory illnesses like COVID-19 is widely being researched because it offers many advantages. Its efficiency and scalability make it perfect for monitoring breakouts of respiratory viruses. As researchers, we, wanted to investigate how this technology could be used to identify COVID-19, but first we needed access to a large number of audio recordings.
The UK Health Security Agency (UKHSA), who were part of our research team, worked with NHS Test and Trace (who collected mostly COVID-positive samples from those following the UK Government's testing scheme) and the REACT-1 study (who collected mostly COVID-negative samples through random population screening) to collate audio samples from 67,842 people who had taken a PCR COVID-19 test. Participants were asked to record a single cough and then a triple cough, a sharp exhalation, and to read a sentence out loud.
How can machine learning identify positive COVID-19 cases?
In order to train a ML model – in this case to identify positive cases of COVID-19 – we collect the relevant data and, after cleaning, split it into three sets. We then use these sets to train the model to perform the task and evaluate its performance. Using this system in our research, we were able to accurately classify a large proportion of positive COVID-19 cases, on par with other research groups which used smaller and less reliable datasets.
But as we continued to analyse the results, it appeared that the accuracy was likely due to an effect in statistics called confounding – where models learn other variables which correlate with the true signal, as opposed to the true signal itself. This is because almost all people in our dataset with COVID-19 have some symptoms, and so the model is learning whether you have symptoms from the audio as a proxy for COVID-19 infection. This confounding is caused by a phenomenon called recruitment bias, since Test and Trace was a pathway accessible only to those with symptoms.
We adjusted our study to correct for this and check whether these results were biased. We matched participants of the same gender and of similar ages, and reporting the same audible COVID-19 symptoms, however only one of each pair was positive for COVID-19. The process of pairing participants is called matching, and it removes the effect of confounding. When we evaluated these models on the matched data, the models failed to perform well, and so we conclude that our models cannot detect a COVID-19 bio-acoustic marker from this data.
We did find there was a slight increase in accuracy if this method was used alongside a symptom tracker, for example within an app where someone could input the symptoms they were experiencing along with an audio recording. But there wasn’t a significant enough improvement for this technology to make a meaningful difference and replace the use of lateral flow tests.
Whilst our search for a ML model that detects COVID-19 from audio alone was not successful, it is still possible that technologies like this could work in the future. Recent publications show progress in detecting sleep apnea and chronic obstructive pulmonary disease (COPD) from audio recordings. But in order to be certain that this is really working as we hope, it is vital that these models, alongside the data, go through equally robust model development and testing.
To find out more about the research, read our pre-print papers we've published.
