Keywords
neurophysiological data
machine learning methods
deep learning neural networks
convolutional neural networks
recurrent neural networks
Abstract
Introduction. Recently, machine learning methods, which are core components of artificial intelligence, have gained popularity in analyzing neurophysiological data. Functional near-infrared spectroscopy (fNIRS) is actively used to study neurocognitive mechanisms. This technology for recording hemodynamic data has a number of advantages, including spatial resolution, non-invasiveness, and the feasibility to conduct studies in natural settings, which has made the technology popular among researchers. Theoretical justification. The analysis of fNIRS results relies on the sequence and selected methods for preliminary processing of raw data, as well as on the classification models employed. This review evaluates various preprocessing methods and examines the approaches to classifying fNIRS data. An essential aspect of preprocessing involves detecting and eliminating physiological artifacts from raw data, utilizing algorithms such as filtering, signal whitening, principal component analysis (PCA) and independent component analysis (ICA), short-channels removal. Methods such as wavelet filtering, spline interpolation, and Kalman filtering are employed to address motion artifacts. Discussion. The review aims to provide an in-depth exploration of machine learning methods, specifically recurrent neural networks (RNN) and convolutional neural networks (CNN), which have been used in various studies for analyzing fNIRS data. The review highlights that leveraging deep learning neural networks can streamline signal preprocessing while achieving higher accuracy compared to traditional approaches in processing neurocognitive data.
References
Селивёрстов, Ю. А., Селивёрстова, Е. В., Коновалов, Р. Н., Котенкова, М. В., & Иллариошкин, С. Н. (2014). Функциональная магнитно-резонансная томография покоя: возможности и будущее мето-да. Бюллетень Национального общества по изучению болезни Паркинсона и расстройств движений, (1), 16–19.
Asgher, U., Khalil, K., Khan, M. J., Ahmad, R., Butt, S. I., Ayaz, Y., ... & Nazir, S. (2020). Enhanced accuracy for mul-ticlass mental workload detection using long short-term memory for brain–computer interface. Frontiers in neuroscience, 14, 584. https://doi.org/10.3389/fnins.2020.00584
Barker, J. W., Aarabi, A., & Huppert, T. J. (2013). Autoregressive model-based algorithm for correcting motion and serially correlated errors in fNIRS. Biomedical optics express, 4(8), 1366–1379.
Benerradi, J., A. Maior, H., Marinescu, A., Clos, J., & L. Wilson, M. (2019, November). Exploring machine learning approaches for classifying mental workload using fNIRS data from HCI tasks. In Proceedings of the Halfway to the Future Symposium 2019 (pp. 1–11). https://doi.org/10.1145/3363384.3363392
Blanco, B., Molnar, M., & Caballero-Gaudes, C. (2018). Effect of prewhitening in resting-state functional near-infrared spectroscopy data. Neurophotonics, 5(4), 040401–040401. https://doi.org/10.1117/1.NPh.5.4.040401
Brigadoi, S., & Cooper, R. J. (2015). How short is short? Optimum source–detector distance for short-separation channels in functional near-infrared spectroscopy. Neurophotonics, 2(2), 025005–025005. https://doi.org/10.1117/1.NPh.2.2.025005
Chen, W. L., Wagner, J., Heugel, N., Sugar, J., Lee, Y. W., Conant, L., ... & Whelan, H. T. (2020). Functional near-infrared spectroscopy and its clinical application in the field of neuroscience: advances and future direc-tions. Frontiers in neuroscience, 14, 724. https://doi.org/10.3389/fnins.2020.00724
Chhabra, H., Shajil, N., & Venkatasubramanian, G. (2020). Investigation of deep convolutional neural network for classification of motor imagery fNIRS signals for BCI applications. Biomedical Signal Processing and Con-trol, 62, 102133. https://doi.org/10.1016/j.bspc.2020.102133
Chiarelli, A. M., Croce, P., Merla, A., & Zappasodi, F. (2018). Deep learning for hybrid EEG-fNIRS brain–computer in-terface: application to motor imagery classification. Journal of neural engineering, 15(3), 036028. https://doi.org/10.1088/1741-2552/aaaf82
Cohen, D. (1968). Magnetoencephalography: evidence of magnetic fields produced by alpha-rhythm cur-rents. Science, 161(3843), 784–786. https://doi.org/10.1126/science.161.3843.784
Cooney, C., Folli, R., & Coyle, D. (2021). A bimodal deep learning architecture for EEG-fNIRS decoding of overt and imagined speech. IEEE Transactions on Biomedical Engineering, 69(6), 1983–1994. https://doi.org/10.1109/TBME.2021.3132861
Cordes, D., Haughton, V. M., Arfanakis, K., Carew, J. D., Turski, P. A., Moritz, C. H., ... & Meyerand, M. E. (2001). Fre-quencies contributing to functional connectivity in the cerebral cortex in “resting-state” data. American Jour-nal of Neuroradiology, 22(7), 1326–1333.
Cui, Z., Chen, W., & Chen, Y. (2016). Multi-scale convolutional neural networks for time series classification. arXiv preprint arXiv:1603.06995. https://doi.org/10.48550/arXiv.1603.06995
Dargazany, A. R., Abtahi, M., & Mankodiya, K. (2019). An end-to-end (deep) neural network applied to raw EEG, fNIRs and body motion data for data fusion and BCI classification task without any pre-/post-processing. arXiv preprint arXiv:1907.09523. https://doi.org/10.48550/arXiv.1907.09523
Dolmans, T. C., Poel, M., van’t Klooster, J. W. J., & Veldkamp, B. P. (2021). Perceived mental workload classification using intermediate fusion multimodal deep learning. Frontiers in human neuroscience, 14, 609096. https://doi.org/10.3389/fnhum.2020.609096
Eastmond, C., Subedi, A., De, S., & Intes, X. (2022). Deep learning in fNIRS: a review. Neurophotonics, 9(4), 041411. https://doi.org/https://doi.org/10.1117/1.NPh.9.4.041411
Erdoĝan, S. B., Özsarfati, E., Dilek, B., Kadak, K. S., Hanoĝlu, L., & Akın, A. (2019). Classification of motor imagery and execution signals with population-level feature sets: implications for probe design in fNIRS based BCI. Journal of neural engineering, 16(2), 026029. https://doi.org/10.1088/1741-2552/aafdca
Funane, T., Sato, H., Yahata, N., Takizawa, R., Nishimura, Y., Kinoshita, A., ... & Kiguchi, M. (2015). Concurrent fNIRS-fMRI measurement to validate a method for separating deep and shallow fNIRS signals by using multidis-tance optodes. Neurophotonics, 2(1), 015003–015003. https://doi.org/10.1117/1.NPh.2.1.015003
Gagnon, L., Yücel, M. A., Boas, D. A., & Cooper, R. J. (2014). Further improvement in reducing superficial contami-nation in NIRS using double short separation measurements. Neuroimage, 85, 127–135. https://doi.org/10.1016/j.neuroimage.2013.01.073
Gao, Y., Chao, H., Cavuoto, L., Yan, P., Kruger, U., Norfleet, J. E., ... & Intes, X. (2022). Deep learning-based motion artifact removal in functional near-infrared spectroscopy. Neurophotonics, 9(4), 041406–041406. https://doi.org/10.1117/1.NPh.9.4.041406
Gao, Y., Yan, P., Kruger, U., Cavuoto, L., Schwaitzberg, S., De, S., & Intes, X. (2020). Functional brain imaging relia-bly predicts bimanual motor skill performance in a standardized surgical task. IEEE Transactions on Biomedi-cal Engineering, 68(7), 2058–2066. https://doi.org/10.1109/TBME.2020.3014299
Ghonchi, H., Fateh, M., Abolghasemi, V., Ferdowsi, S., & Rezvani, M. (2020а). Deep recurrent–convolutional neural network for classification of simultaneous EEG–fNIRS signals. IET Signal Processing, 14(3), 142–153. https://doi.org/10.1049/iet-spr.2019.0297
Ghonchi, H., Fateh, M., Abolghasemi, V., Ferdowsi, S., & Rezvani, M. (2020b). Spatio-temporal deep learning for EEG-fNIRS brain computer interface. In 2020 42nd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC) (pp. 124–127). IEEE. https://doi.org/10.1109/EMBC44109.2020.9176183
Glorot, X., Bordes, A., & Bengio, Y. (2011, June). Deep sparse rectifier neural networks. In Proceedings of the four-teenth international conference on artificial intelligence and statistics (pp. 315–323). JMLR Workshop and Conference Proceedings.
Graves, A. (2012). Long Short-Term Memory. In: Supervised Sequence Labelling with Recurrent Neural Networks. Studies in Computational Intelligence, 385, 37–45. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-642-24797-2_4
Hakimi, N., Jodeiri, A., Mirbagheri, M., & Setarehdan, S. K. (2020). Proposing a convolutional neural network for stress assessment by means of derived heart rate from functional near infrared spectroscopy. Computers in Biology and Medicine, 121, 103810. https://doi.org/10.1016/j.compbiomed.2020.103810
Hamid, H., Naseer, N., Nazeer, H., Khan, M. J., Khan, R. A., & Shahbaz Khan, U. (2022). Analyzing classification per-formance of fNIRS-BCI for gait rehabilitation using deep neural networks. Sensors, 22(5), 1932. https://doi.org/10.3390/s22051932
Ho, T. K. K., Gwak, J., Park, C. M., & Song, J. I. (2019). Discrimination of mental workload levels from multi-channel fNIRS using deep leaning-based approaches. Ieee Access, 7, 24392–24403. https://doi.org/10.1109/ACCESS.2019.2900127
Hochreiter, S., & Schmidhuber, J. (1997). Long short-term memory. Neural computation, 9(8), 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Huppert, T. J. (2016). Commentary on the statistical properties of noise and its implication on general linear mod-els in functional near-infrared spectroscopy. Neurophotonics, 3(1), 010401–010401. https://doi.org/10.1117/1.NPh.3.1.010401
Hyvärinen, A., & Oja, E. (2000). Independent component analysis: algorithms and applications. Neural net-works, 13(4–5), 411–430. https://doi.org/10.1016/s0893-6080(00)00026-5
Izzetoglu, M., Chitrapu, P., Bunce, S., & Onaral, B. (2010). Motion artifact cancellation in NIR spectroscopy using discrete Kalman filtering. Biomedical engineering online, 9, 1–10. https://doi.org/10.1186/1475-925X-9-16
Jahani, S., Setarehdan, S. K., Boas, D. A., & Yücel, M. A. (2018). Motion artifact detection and correction in function-al near-infrared spectroscopy: a new hybrid method based on spline interpolation method and Savitzky–Golay filtering. Neurophotonics, 5(1), 015003–015003. https://doi.org/10.1117/1.NPh.5.1.015003
Kim, M., Lee, S., Dan, I., & Tak, S. (2022). A deep convolutional neural network for estimating hemodynamic re-sponse function with reduction of motion artifacts in fNIRS. Journal of Neural Engineering, 19(1), 016017. https://doi.org/10.1088/1741-2552/ac4bfc
Kohno, S., Miyai, I., Seiyama, A., Oda, I., Ishikawa, A., Tsuneishi, S., ... & Shimizu, K. (2007). Removal of the skin blood flow artifact in functional near-infrared spectroscopic imaging data through independent component analysis. Journal of Biomedical Optics, 12(6), 062111–062111. https://doi.org/10.1117/1.2814249
Kwon, J., & Im, C. H. (2021). Subject-independent functional near-infrared spectroscopy-based brain–computer interfaces based on convolutional neural networks. Frontiers in Human Neuroscience, 15, 646915. https://doi.org/10.3389/fnhum.2021.646915
Light, G. A., Williams, L. E., Minow, F., Sprock, J., Rissling, A., Sharp, R., ... & Braff, D. L. (2010). Electroencephalog-raphy (EEG) and event‐related potentials (ERPs) with human participants. Current protocols in neurosci-ence, 52(1), 6–25. https://doi.org/10.1002/0471142301.ns0625s52
Liu, Y., Ayaz, H., & Shewokis, P. A. (2017). Multisubject “learning” for mental workload classification using concur-rent EEG, fNIRS, and physiological measures. Frontiers in human neuroscience, 11, 389. https://doi.org/10.3389/fnhum.2017.00389
Lu, J., Yan, H., Chang, C., & Wang, N. (2020). Comparison of machine learning and deep learning approaches for decoding brain computer interface: an fNIRS study. In Intelligent Information Processing X: 11th IFIP TC 12 In-ternational Conference, IIP 2020, Hangzhou, China, July 3–6, 2020, Proceedings 11 (pp. 192-201). Springer International Publishing. https://doi.org/10.1007/978-3-030-46931-3_18
Luke, R., Larson, E. D., Shader, M. J., Innes-Brown, H., Van Yper, L., Lee, A. K., ... & McAlpine, D. (2021). Analysis methods for measuring passive auditory fNIRS responses generated by a block-design paradigm. Neuropho-tonics, 8(2), 025008. https://doi.org/10.1117/1.NPh.8.2.025008
Ma, T., Lyu, H., Liu, J., Xia, Y., Qian, C., Evans, J., ... & He, S. (2020). Distinguishing bipolar depression from major depressive disorder using fnirs and deep neural network. Progress In Electromagnetics Research, 169, 73–86. https://doi.org/10.2528/PIER20102202.
Ma, T., Wang, S., Xia, Y., Zhu, X., Evans, J., Sun, Y., & He, S. (2021). CNN-based classification of fNIRS signals in motor imagery BCI system. Journal of Neural Engineering, 18(5), 056019. https://doi.org/10.1088/1741-2552/abf187
Molavi, B., and Dumont, G. A. (2012). Wavelet-based motion artifact removal for functional near-infrared spec-troscopy. Physiological Measurement, 33, 259–270. https://doi.org/10.1088/0967-3334/33/2/259
Naseer, N., Qureshi, N. K., Noori, F. M., & Hong, K. S. (2016). Analysis of different classification techniques for two-class functional near-infrared spectroscopy-based brain-computer interface. Computational Intelligence and Neuroscience, 2016. https://doi.org/10.1155/2016/5480760
Nguyen, H. D., Yoo, S. H., Bhutta, M. R., & Hong, K. S. (2018). Adaptive filtering of physiological noises in fNIRS da-ta. Biomedical Engineering Online, 17, 1–23. https://doi.org/10.1186/s12938-018-0613-2
Orrù, G., Monaro, M., Conversano, C., Gemignani, A., & Sartori, G. (2020). Machine learning in psychometrics and psychological research. Frontiers in Psychology, 10, 2970. https://doi.org/10.3389/fpsyg.2019.02970
Ortega, P., & Faisal, A. (2021, May). HemCNN: deep learning enables decoding of fNIRS cortical signals in hand grip motor tasks. In 2021 10th International IEEE/EMBS Conference on Neural Engineering (NER) (pp. 718-721). IEEE. https://doi.org/10.1109/NER49283.2021.9441323
Ortega, P., & Faisal, A. A. (2021). Deep learning multimodal fNIRS and EEG signals for bimanual grip force decod-ing. Journal of Neural Engineering, 18(4), 0460e6. https://doi.org/10.1088/1741-2552/ac1ab3
Osharina, V., Ponchel, E., Aarabi, A., Grebe, R., & Wallois, F. (2010). Local haemodynamic changes preceding inter-ictal spikes: a simultaneous electrocorticography (ECoG) and near-infrared spectroscopy (NIRS) analysis in rats. Neuroimage, 50(2), 600–607. https://doi.org/10.1016/j.neuroimage.2010.01.009
Pinti, P., Aichelburg, C., Gilbert, S., Hamilton, A., Hirsch, J., Burgess, P., & Tachtsidis, I. (2018). A review on the use of wearable functional near‐infrared spectroscopy in naturalistic environments. Japanese Psychological Re-search, 60(4), 347–373. https://doi.org/10.1111/jpr.12206
Quaresima, V., & Ferrari, M. (2019, August). A mini-review on functional near-infrared spectroscopy (fNIRS): where do we stand, and where should we go?. Photonics, 6(3). https://doi.org/10.3390/photonics6030087
Robertson, F. C., Douglas, T. S., & Meintjes, E. M. (2010). Motion artifact removal for functional near infrared spec-troscopy: a comparison of methods. IEEE Transactions on Biomedical Engineering, 57(6), 1377–1387. https://doi.org/10.1109/TBME.2009.2038667
Rojas, R. F., Romero, J., Lopez-Aparicio, J., & Ou, K. L. (2020). Pain assessment based on fNIRS using bidirectional LSTMs. arXiv preprint arXiv:2012.13231. URL: http://arxiv.org/abs/2012.13231
Saadati, M., Nelson, J., & Ayaz, H. (2019, October). Mental workload classification from spatial representation of fnirs recordings using convolutional neural networks. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP) (pp. 1-6). IEEE. https://doi.org/10.1109/MLSP.2019.8918861
Scholkmann, F., Kleiser, S., Metz, A. J., Zimmermann, R., Pavia, J. M., Wolf, U., & Wolf, M. (2014). A review on con-tinuous wave functional near-infrared spectroscopy and imaging instrumentation and methodology. Neu-roimage, 85, 6-27. https://doi.org/10.1016/j.neuroimage.2013.05.004
Scholkmann, F., Spichtig, S., Muehlemann, T., & Wolf, M. (2010). How to detect and reduce movement artifacts in near-infrared imaging using moving standard deviation and spline interpolation. Physiological measure-ment, 31(5), 649. https://doi.org/10.1088/0967-3334/31/5/004
Sirpal, P., Kassab, A., Pouliot, P., Nguyen, D. K., & Lesage, F. (2019). fNIRS improves seizure detection in multimodal EEG-fNIRS recordings. Journal of Biomedical Optics, 24(5), 051408–051408. https://doi.org/10.1117/1.jbo.24.5.051408
Sitnikova, M. A., & Malykh, S. B. (2021). Functional near-infrared spectroscopy applications in developmental cog-nitive neuroscience. I.P. Pavlov Journal of Higher Nervous Activity, 71(4), 485–499.
Sun, Z., Huang, Z., Duan, F., & Liu, Y. (2020). A novel multimodal approach for hybrid brain–computer inter-face. IEEE Access, 8, 89909–89918. https://doi.org/10.1109/ACCESS.2020.2994226
Tanveer, M. A., Khan, M. J., Qureshi, M. J., Naseer, N., & Hong, K. S. (2019). Enhanced drowsiness detection using deep learning: an fNIRS study. IEEE access, 7, 137920–137929. https://doi.org/10.1109/ACCESS.2019.2942838
Trakoolwilaiwan, T., Behboodi, B., Lee, J., Kim, K., & Choi, J. W. (2018). Convolutional neural network for high-accuracy functional near-infrared spectroscopy in a brain–computer interface: three-class classification of rest, right-, and left-hand motor execution. Neurophotonics, 5(1), 011008–011008. https://doi.org/10.1117/1.nph.5.1.011008
Van Houdt, G., Mosquera, C., & Nápoles, G. (2020). A review on the long short-term memory model. Artificial In-telligence Review, 53, 5929–5955. https://doi.org/10.1007/s10462-020-09838-1
Wang, R., Hao, Y., Yu, Q., Chen, M., Humar, I., & Fortino, G. (2021). Depression analysis and recognition based on functional near-infrared spectroscopy. IEEE Journal of Biomedical and Health Informatics, 25(12), 4289–4299. https://doi.org/10.1109/JBHI.2021.3076762
Wang, Z., & Oates, T. (2015). Spatially encoding temporal correlations to classify temporal data using convolu-tional neural networks. arXiv preprint arXiv:1509.07481. https://doi.org/10.48550/arXiv.1509.07481
Wickramaratne, S. D., & Mahmud, M. S. (2020, November). A Ternary Bi-Directional LSTM Classification for Brain Activation Pattern Recognition Using fNIRS. In 2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS) (pp. 202-207). IEEE. https://doi.org/10.48550/arXiv.2101.05892
Wickramaratne, S. D., & Mahmud, M. S. (2021). Conditional-GAN based data augmentation for deep learning task classifier improvement using fNIRS data. Frontiers in big Data, 4, 659146. https://doi.org/10.3389/fdata.2021.659146
Woo, S. W., Kang, M. K., & Hong, K. S. (2020). Classification of finger tapping tasks using convolutional neural net-work based on augmented data with deep convolutional generative adversarial network. In 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob). IEEE. https://doi.org/10.1109/BioRob49111.2020.9224386
Xu, L., Choy, C. S., & Li, Y. W. (2016, September). Deep sparse rectifier neural networks for speech denoising. In 2016 IEEE International Workshop on Acoustic Signal Enhancement (IWAENC) (pp. 1-5). IEEE. https://doi.org/10.1109/IWAENC.2016.7602891
Xu, L., Geng, X., He, X., Li, J., & Yu, J. (2019). Prediction in autism by deep learning short-time spontaneous hemo-dynamic fluctuations. Frontiers in Neuroscience, 13, 1120. https://doi.org/10.3389/fnins.2019.01120
Yang, D., Huang, R., Yoo, S. H., Shin, M. J., Yoon, J. A., Shin, Y. I., & Hong, K. S. (2020). Detection of mild cognitive impairment using convolutional neural network: temporal-feature maps of functional near-infrared spec-troscopy. Frontiers in Aging Neuroscience, 12, 141. https://doi.org/10.3389/fnagi.2020.00141
Yücel, M. A., Lühmann, A. V., Scholkmann, F., Gervain, J., Dan, I., Ayaz, H., ... & Wolf, M. (2021). Best practices for fNIRS publications. Neurophotonics, 8(1), 012101–012101. https://doi.org/10.1117/1.NPh.8.1.012101
Zhao, Q., Li, C., Xu, J., & Jin, H. (2019, July). FNIRS based brain-computer interface to determine whether motion task to achieve the ultimate goal. In 2019 IEEE 4th International Conference on Advanced Robotics and Mechatronics (ICARM) (pp. 136–140). IEEE. https://doi.org/10.1109/ICARM.2019.8833883
Zhang, X., Noah, J. A., & Hirsch, J. (2016). Separation of the global and local components in functional near-infrared spectroscopy signals using principal component spatial filtering. Neurophotonics, 3(1), 015004–015004. https://doi.org/10.1117/1.NPh.3.1.015004
Zheng, Y., Liu, Q., Chen, E., Ge, Y., & Zhao, J. L. (2014). Time series classification using multi-channels deep con-volutional neural networks. In Web-Age Information Management: 15th International Conference, WAIM 2014, Macau, China, June 16-18, 2014. Proceedings 15. Springer International Publishing. https://doi.org/10.1007/978-3-319-08010-9_33
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright (c) 2024 Rustam G. Asadullaev, Maria А. Sitnikova, Aleksandr А. Sletov, Andrey V. Sitnikov, Sergey B. Malykch