Denoising ECG signals with ensemble of filters
An automated command line application useful for time-series signal data with utility services like preprocessing raw data, signal-to-noise (SNR or S/N) ratio estimation, filtering noise from raw data, etc.
TL;DR: ECG CLI application code
TL;DR: denoising demo notebook
Hello friends,
In this tutorial, we will peek into an ECG dataset, and use a bunch of different filters to remove noise from the signals using an ensemble approach. Also, we will see how wavelet denoising can be helpful to filter noise. To start with, we will see how noise can be determined at the very first place, and then see how to filter out noise from raw time-series signal data illustrated with a hands-on demo example. Also, at the end of this tutorial, I have provided my developed Command Line Interface (CLI) application program, which user’s can use straightaway to process their own raw data, estimate signal-to-noise ratio using the different algorithms, and denoise (filter noise) at ease with simple single liner low-code solution.
ECG timeseries data
An ECG signal can be described as a non-stationary time series that presents some irregularities in the waveform. Electrocardiography (ECG) signals are used to monitor the health of human heart that results in a time series signal which is cyclic in nature, and represents distinctive heart activity patterns. For comparison of signal vs. time-series check this blog here.
Noise in time-series signal data
The intuitive way to interpret noise with context to timeseries simply refers to some random fluctuations in the time series from its typical pattern which is random (unwanted) and carry no useful information. The goal of eliminating such random or not useful information while preserving useful information is coined as Denoising.
What is Signal-to-Noise Ratio (SNR or S/N) ?
Signal-to-noise ratio is simply a measure that compares the level of a desired signal to the level of background noise. SNR is an estimate of the signal quality. It is represented in decibel (dB).
Related state-of-the-art works (Literature Survey) on SNR estimation methods
The table below lists out most of the related works done towards ECG signal quality evaluations. In the table, corresponding method for calculating SNR is given, along with the needed prerequisites in terms of clean/reconstructed data. In most of the works as seen in the table below, there is a need of clean signal data (reference signal data) or alteratively the original data has to be reconstructed to estimate the quality.
My Methodology
The psuedocode for my proposed alternative approach for SNR estimation is as given below. Firstly, we start with removing the outlier data points. Next, we will estimate the mean of all the signals, and substract it from every individual data point. Then, we compress the signal with Karhunen–Loève transform (KLT) followed by calculating signal-to-noise ratio, and finally converting it to the desired unit.
Other alternative ways to detect/filter out noise from data
Here, we will see the Machine Learning based techniques to remove noise from raw Signal/Data. One can certainly use Deep Learning techniques to denoise data, also leverage the dimensional reduction techniques for denoising, and can also stick with the classic signal processing techniques.
Few random techniques are as listed below just for reference.
1) Deep denoising auto-encoding method
2) PCA (Principal Component Analysis)
3) Contrastive PCA
4) Fourier Transform Technique / Fast Fourier Transform Technique and many more…
What are filters?
Filters in signal processing can be basically a device (hardware) or a process (software) that removes some unwanted components or features from the raw signal. For more information on signal & the different types of medical signals check this blog here.
How ensemble of different filters can help in effective denoising?
There are many filters proposed to solve the noise reduction problem. The most commonly used filters are low-pass, high-pass, band-pass and band-stop. In general, filters can be classified into two broad categories, i.e., Low Pass Filter (that passes signals with a frequency lower than a certain cut-off frequency, and attenuates signals with frequencies higher than the cut-off frequency) & High Pass Filter (that passes signals with a frequency higher than a certain cut-off frequency, and attenuates signals with frequencies lower than the cut-off frequency). We use the combination of butter high/low pass filter, IIR or FIR filter, band-stop filter, etc. to reduce background noise. For intuitive explanation of the signal filters do check-out this material here.
The snippet code below applies different filters towards denoising the raw signals by combining the techniques to produce a singular composite effect.
Demo illustration
Now, that we have seen all the related theoretical concepts. Let’s quickly get started with an example. Here, we will use the data that I had received from one of the leading smart garment and textile companies that manufactures Sports Infrared Performance and Recovery Sportswear for fitness enthusiasts. The data provided includes ECG recordings logged from the sportwear tested for different activities such as Sitting, Walking, Standing, etc.
Please Note:- For data privacy and data security reasons, I would like to keep the actual data source anonymous, and also just provide readers with a subset of actual data (sample data) from the complete dataset just for sake of following this tutorial.
We will see how “Deep-XF” a python library can also be used intuitively for denoising signals. For quick introduction & all the other use-case applications that is supported by the deep-xf package in general is listed in the blog post here. Also, take a look at the Forecasting, Nowcasting blogs with hands-on demo tutorial from here & here service provided using the same package. Further, checkout how deep-xf can be also used to learn signal similarity representations from here.
For library installation follow these steps here & for manual prerequisites installation check here.
Once the data is imported, next we input the parameters such as sampling frequency (fs), order, cutoff thresholds, etc. with set_parameters method. Once the custom user-input parameters are set, then we remove noise from raw data with ensembling of the different filters such as butter high pass filter, butter low pass filter, and notch filter followed by plotting & outputting the filtered data with plot_and_write method.
CLI ECG APPLICATION
Finally, an ECG command line application is developed covering all the milestones/deliverables bundled as all-in-one CLI program for users to leverage the facility at ease, where the users can input the raw data, choose the particular algorithms, provide a path to save the denoised data, estimate & print the SNR values for each ECG signals in the data, save plots, etc. The code is placed in the folder “cli_ecg_application” along with the README file from here. The complete application code can be found here.
It includes two algorithm choices to estimate SNR (classical way to estimate noise, using KLT method), and two algorithm choices to denoise the raw signals (with ensemble of filters, using DWT method). The snippet code for estimating signal-to-noise (SNR or S/N) is as shown below.
Similarily, the filtering of noise from the raw signal is done using two choices, i.e., either with ensembling of filters, or using the discrete wavelet transform (DWT) technique as shown in the snippet below.
To run the application follow the usage steps as shown in the snippet below that takes in the command line inputs such as raw dataset, output path to store filtered dataset, sampling frequency, algorithm choice for snr estimation, and choice for selecting the filtering algorithm. The complete application code can be found here.
Conclusion
Through this post, we have walked through some basics of signal, and talked about the measure of estimating signal quality. We have seen how the raw ECG signals can be processed to filter out noise from the actual signals. We also saw how to estimate the signal quality. Finally, we glimsed through an end-to-end ECG CLI software program that can be used by the end-users to evaluate the signal quality, filter-out noise from raw signals, visualizing with graph plots, and outputting the filtered data to disk, etc. via command line.
Let’s get connected
You can reach me at ajay.arunachalam08@gmail.com; Contact via Linkedin
Keep Learning, Cheers :)
About Author
I am an AWS Certified Machine Learning Specialist & Cloud Solution Architect. From my experience working on real-world problems, I fully acknowledge that finding good representations is the key in designing the system that can solve interesting challenging real-world problems, that go beyond human-level intelligence, and ultimately explain complicated data for us that we don’t understand. In order to achieve this, I envision learning algorithms that can learn feature representations from both unlabelled and labelled data, be guided with and/or without human interaction, and that are on different levels of abstractions in order to bridge the gap between low-level data and high-level abstract concepts. I also truly believe that Opacity in AI Systems is the need of the hour. With this in mind, I have always strived to democratize AI, and be more inclined towards building Interpretable/Explainable Models. My interest is in building real-time AI Solutions, Machine Learning/Deep Learning at scale, Productionalising Explainable Models, Deep Reinforcement Learning, Computer Vision, and Natural Language Processing, specifically learning good representations.
References
