ieee research paper on face recognition

Face Recognition

Featured article, related topics, top conferences on face recognition, top videos on face recognition.

Xplore Articles related to Face Recognition

Periodicals related to face recognition, e-books related to face recognition, courses related to face recognition, top organizations on face recognition, most published xplore authors for face recognition.

Help | Advanced Search

Computer Science > Computer Vision and Pattern Recognition

Title: deep face recognition: a survey.

Abstract: Deep learning applies multiple processing layers to learn representations of data with multiple levels of feature extraction. This emerging technique has reshaped the research landscape of face recognition (FR) since 2014, launched by the breakthroughs of DeepFace and DeepID. Since then, deep learning technique, characterized by the hierarchical architecture to stitch together pixels into invariant face representation, has dramatically improved the state-of-the-art performance and fostered successful real-world applications. In this survey, we provide a comprehensive review of the recent developments on deep FR, covering broad topics on algorithm designs, databases, protocols, and application scenes. First, we summarize different network architectures and loss functions proposed in the rapid evolution of the deep FR methods. Second, the related face processing methods are categorized into two classes: "one-to-many augmentation" and "many-to-one normalization". Then, we summarize and compare the commonly used databases for both model training and evaluation. Third, we review miscellaneous scenes in deep FR, such as cross-factor, heterogenous, multiple-media and industrial scenes. Finally, the technical challenges and several promising directions are highlighted.

Submission history

Access paper:.

• Other Formats

References & Citations

• Google Scholar
• Semantic Scholar

DBLP - CS Bibliography

Bibtex formatted citation.

Bibliographic and Citation Tools

Code, data and media associated with this article, recommenders and search tools.

• Institution

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs .

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 24 May 2023

A study on computer vision for facial emotion recognition

• Zi-Yu Huang 1 ,
• Chia-Chin Chiang 1 ,
• Jian-Hao Chen 2 ,
• Yi-Chian Chen 3 ,
• Hsin-Lung Chung 1 ,
• Yu-Ping Cai 4 &
• Hsiu-Chuan Hsu 2 , 5  

Scientific Reports volume  13 , Article number:  8425 ( 2023 ) Cite this article

18k Accesses

23 Citations

2 Altmetric

Metrics details

• Health care
• Health occupations

Artificial intelligence has been successfully applied in various fields, one of which is computer vision. In this study, a deep neural network (DNN) was adopted for Facial emotion recognition (FER). One of the objectives in this study is to identify the critical facial features on which the DNN model focuses for FER. In particular, we utilized a convolutional neural network (CNN), the combination of squeeze-and-excitation network and the residual neural network, for the task of FER. We utilized AffectNet and the Real-World Affective Faces Database (RAF-DB) as the facial expression databases that provide learning samples for the CNN. The feature maps were extracted from the residual blocks for further analysis. Our analysis shows that the features around the nose and mouth are critical facial landmarks for the neural networks. Cross-database validations were conducted between the databases. The network model trained on AffectNet achieved 77.37% accuracy when validated on the RAF-DB, while the network model pretrained on AffectNet and then transfer learned on the RAF-DB results in validation accuracy of 83.37%. The outcomes of this study would improve the understanding of neural networks and assist with improving computer vision accuracy.

Similar content being viewed by others

Four-layer ConvNet to facial emotion recognition with minimal epochs and the significance of data diversity

Image-based facial emotion recognition using convolutional neural network on emognition dataset

Estimation of continuous valence and arousal levels from faces in naturalistic conditions

Introduction.

In human communications, facial expressions contain critical nonverbal information that can provide additional clues and meanings to verbal communications 1 . Some studies have suggested that 60–80% of communication is nonverbal 2 . This nonverbal information includes facial expressions, eye contact, tones of voice, hand gestures and physical distancing. In particular, facial expression analysis has become a popular research topic 3 . Facial emotional recognition (FER) has been applied in the field of human–computer interaction (HCI) in areas such as autopilot, education, medical treatment, psychological treatment 4 , surveillance and psychological analysis in computer vision 5 , 6 .

In psychology and computer vision, emotions are classified as categorical or dimensional (valence and arousal) models 7 , 8 , 9 . In the categorical model, Ekman et al . 7 defined basic human emotions as happiness, anger, disgust, fear, sadness, and surprise. In the dimensional model, the emotion is evaluated by continuous numerical scales for determination of valence and arousal. FER is an important task in computer vision that has numerous practical applications and the number of studies on FER has increased in recent years 10 , 11 , 12 , 13 , benefiting from the advances provided by deep neural networks. In particular, convolutional neural networks (CNNs) have attained excellent results in terms of extracting features. For example, He et al . 14 proposed the residual neural network (ResNet) architecture in 2015, which added residual learning to a CNN to resolve the issues of vanishing gradient and decreasing accuracy of deep networks.

Several authors have applied neural network models to classify emotions according to categorical models 15 , 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 and dimensional models 15 , 23 , 24 , 25 , 26 . Huang 27 applied a residual block architecture to a VGG CNN to perform emotion recognition and obtained improved accuracy. Mao et al . 28 proposed a new FER model called POSTER V2, which aims to improve the performance of the state-of-the-art technique and reduce the required computational cost by introducing window-based cross attention mechanism and facial landmarks’ multi-scale features. To incorporate more information into the automatic emotion recognition process, some recent studies have fused several modalities, such as the temporal, audio and visual modalities 10 , 17 , 18 , 23 , 25 , into the algorithm. Moreover, attention mechanisms have been adopted by several studies 17 , 18 , 19 , 20 , 22 , 25 for FER tasks. Zhang et al . 19 applied class activation mapping to analyze the attention maps learned by their model. It was found that the model could be regularized by flipping its attention map and randomly erasing part of the input images. Wang et al. 22 introduced an attention branch to learn a face mask that highlights the discriminative parts for FER. These studies show that attention mechanisms play a critical role in FER. Several approaches for FER utilize self-attention mechanisms to capture both local and global contexts through a set of convolutional layers for feature extraction 29 , 30 , 31 . The extracted features are then used as the inputs of a relation attention module, which utilizes self-attention to capture the relationships between different patches and the context.

However, the practical deployment of facial recognition systems remains a challenging task, as a result of the presence of noise, ambiguous annotations 32 , and complicated scenes in the real-world setting 33 , 34 , 35 . Since attention modules have been proven effective for computer vision tasks, applying attention modules to FER tasks is of great interest. Moreover, in psychology, the facial features for FER by human have been analyzed. The results presented by Beaudry et al . 35 suggest that the mouth is the major landmark when observing a happy emotion and that the eyes are the major landmarks when observing a sad emotion. Similarly, the DNN model extracts discriminative features for FER. It is beneficial to apply class activation mapping to identify the discriminative features learned by the network at each layer. It has been shown that the class activation mapping method can be utilized for localization recognition around the eyes for movement analysis purposes 37 , 38 . The produced feature maps could provide a better understanding of the performance of the developed model.

In this study, the squeeze-and-excitation module (SENet) was used with ResNet-18 to achieve a relatively light model for FER. This model has fewer trainable parameters (approximately 11.27 million) than the approximately 23 million parameters required for ResNet-50 and the approximately 86 million parameters of the vision transformer. The effectiveness of the proposed approach was evaluated on two FER datasets, namely, AffectNet and the Real-World Affective Faces Database (RAF-DB). Both datasets contain a great quantity of facial emotion data, including those from various cultures and races. The number of images in AffectNet is about 20 times than that of RAF-DB. The images in AffectNet are more diverse and wilder than those in RAF-DB. The neural network was trained to extract emotional information from AffectNet and RAF-DB. A cross-database validation between the AffectNet dataset and the RAF-DB was conducted. The results show that a training accuracy of 79.08% and a validation accuracy of 56.54% were achieved with AffectNet. A training accuracy of 76.51% and a validation accuracy of 65.67% were achieved with RAF-DB. The transfer-learning was applied on RAF-DB with pretrained weight obtained with AffectNet. The prediction accuracy after transfer-learning increases dramatically on the RAF-DB dataset. The results suggest that transfer learning can be conducted for smaller dataset with a particular culture, region, or social setting 36 for specific applications. Transfer-learning enables the model to learn the facial emotions of a particular population with a smaller database and achieve accurate results. Moreover, the images in AffectNet and RAF-DB with softmax score exceeding 90% were selected to identify the important facial landmarks that were captured by the network. It is found that in the shallow layers, the extracted dominant features are fine lines, whereas in the deep layers, the regions near mouth and nose are more important.

Database and model

The AffectNet database contains 456,349 images of facial emotions obtained from three search engines, Google, Bing and Yahoo, in six different languages. The images were labeled with the following 11 emotions: neutrality, happiness, sadness, surprise, fear, disgust, anger, contempt, none, uncertain, and nonface. Among these emotions, “uncertain” means that the given image cannot be classified into one of the other categories, and “nonface” means that the image contains exaggerated expressions, animations, drawings, or watermarks. Mollahosseini et al . 15 hired annotators to manually classify emotions defined in AffectNet. In addition, AffectNet is heavily imbalanced in terms of the number of images of each emotion category. For example, the number of images representing “happy” is almost 30 times higher than the number of images representing “disgust”. The number of images for each category is shown in Table 1 . Figure  1 shows sample images for the 11 emotions contained in AffectNet. In this study, we use seven categories, surprise, fear, disgust, anger, sadness, happiness and neutrality, in AffectNet.

Image categories of the faces contained in the AffectNet database 12 .

The RAF-DB is provided by the Pattern Recognition and Intelligent System Laboratory (PRIS Lab) of the Beijing University of Posts and Telecommunications 39 . The database consists of more than 300,000 facial images sourced from the internet, which are classified into seven categories: surprise, fear, disgust, anger, sadness, happiness and neutrality. Each of the images contains 5 accurate landmark locations and 37 automatic landmark locations. The RAF-DB also contains a wide variety of information in terms of ages, races, head gestures, light exposure levels and blocking. The training set contains five times as many images as the test set. Figure  2 shows sample images for the seven emotions contained in the RAF-DB. Table 1 shows the number of images used in this article for each emotion from each database.

Image categories of the faces contained in the RAF-DB database 37 .

SENet is a new image recognition architecture developed in 2017 40 . The network reinforces critical features by comparing the correlations among feature channels to achieve increased classification accuracy. Figure  3 shows the SENet architecture, which contains three major operations. The squeeze operation extracts global feature information from the previous convolution layer and conducts global average pooling on the feature map to obtain a feature tensor (Z) of size 1 × 1 ×  \({\text{C}}\) (number of channels), in which the \({\text{c}} - {\text{th}}\) element is calculated by:

where \(F_{sq}\) is the global average pooling operation, \(u_{c}\) is the \({\text{c}} - {\text{th}}\) 2-dimensional matrix, W × H represents the dimensions of each channel, and C is the number of channels.

The schema of the SENet inception module.

Equation ( 1 ) is followed by two fully connected layers. The first layer reduces the number of channels from \({\text{C}}\) to \({\text{C}}/{\text{r}}\) to reduce the required number computations (r is the compression rate), and the second layer increases the number of channels to \({\text{C}}\) . The excitation operation is defined as follows:

where \({\upsigma }\) is the sigmoid activation function, \(\delta\) is the rectified linear unit (ReLU) excitation function, and \(W_{1}\) and \(W_{2}\) are the weights for reducing and increasing the dimensionality, respectively.

The scale operation multiplies the feature tensor by the excitation. This operation captures the significance of each channel via feature learning. The corresponding channel is then multiplied by the gained weight to discern the major and minor information for the computer 38 . The formula for the scale operation, which is used to obtain the final output of the block, is shown as follows.

where the dot is the channelwise multiplication operation and \(S_{c}\) is the output of the excitation operation.

ResNet was proposed by He et al . 11 to solve the vanishing gradient problem in a deep network. ResNet introduces a residual block to a conventional CNN. Figure  4 shows the residual block in the ResNet architecture. The concept of a residual block is to combine the output from the previous convolutional layer with the next convolutional layer in the ResNet. It has been shown in several studies that the residual blocks relieve the vanishing gradient issue encountered by a deeper network. Therefore, the residual blocks have been adopted in several architectures 37 , 38 .

Residual block of the ResNet architecture.

SE-ResNet combines the SENet and ResNet architectures presented above and adds the SE block from SENet to ResNet. The SE block is used to capture the significance of each channel to determine whether it contains major or minor information. The feature information from the previous convolutional layer is then combined with the next layer by the residual block. This method can mitigate the decreasing accuracy caused by the vanishing gradient problem that occurs while increasing the network layers. Figure  5 shows the network architecture of SE-ResNet.

The schema of the SE-Resnet module.

Experimental method

In this study, we extracted seven categories from AffectNet to ensure that AffectNet and the RAF-DB were validated with identical categories. The SE-ResNet architecture was adopted as the neural network model for training and testing. A comparison and cross-database validation were conducted between RAF-DB and AffectNet. To achieve better performance, the transfer learning technique was used. The model trained on AffectNet was applied as the pretrained model to train RAF-DB.

The feature maps derived from each SE block were printed to determine which facial landmarks contain major information for the network. Only facial emotion images with softmax score exceeding 90% were adopted to ensure objectivity and accuracy. Examples of the feature maps printed from AffectNet are shown in Fig.  6 . The feature maps printed from the RAF-DB are shown in Fig.  7 .

Feature maps of different SE block layers (AffectNet).

Feature maps of different SE block layers (RAF-DB).

In this experiment, the training hardware was an NVIDIA TITAN RTX 24-GB GPU. The input image size was 256 × 256 pixels with data augmentation. For the training process, the tones of the input images were changed. The images were randomly rotated between + / − 30 degrees, and cropped according to the four corners and the center into five images of the size 224 × 224 pixels. For validation purposes, the input images were cropped from the center to a final size of 224 × 224 pixels. The optimization algorithm and loss function were stochastic gradient descent and the cross entropy loss function, respectively. Twenty epochs were used, and the initial learning rate was set to 0.01. The momentum was 0.9 and the batch size for training was 100.

Results and discussion

Cross-database validation.

The AffectNet dataset and the RAF-DB were cross-database validated in this study. The model trained on AffectNet was used to predict the RAF-DB, and the model trained on the RAF-DB was used to predict AffectNet. The results are shown in Table 2 . Because AffectNet exhibits more diversity in terms of facial emotion data and more images, when the model trained on AffectNet predicted the RAF-DB, an accuracy of 77.37% was achieved, which was significantly higher than the accuracy achieved by directly training on the RAF-DB (65.67%). In contrast, low accuracy (42.6%) was obtained for AffectNet predicted by the model trained on the RAF-DB. The difference can be understood by the fact that the images in AffectNet are more in quantity and more complex.

The accuracies achieved on AffectNet and RAF-DB by SE-ResNet were compared in this study. RAF-DB results in a higher accuracy than AffectNet, as shown in Table 3 . However, this was expected since the RAF-DB dataset exhibits more constrained images. The accuracy of the proposed model on AffectNet is 56%, which is slightly lower than the 58% accuracy obtained in the original paper 19 that proposed AffectNet. However, as mentioned in the original paper 15 , the agreement between two human annotators was 60% over 36,000 images. Our result is comparable to this agreement rate.

Additionally, we performed transfer learning by pretraining the model on AffectNet, followed by training on the RAF-DB. As shown in Table 4 , the validation accuracy on the RAF-DB increased by 26.95% ([(accuracy with pretrained model—accuracy without pretrained model)/accuracy without pretrained model = (83.37–65.67) / 65.67] × 100%) and was higher than that of the model trained directly with the RAF-DB. Compared to the accuracy of 76.73% obtained in 21 by multi-region ensemble CNN, transfer learning with a single network performs better than the ensemble CNN that utilizes global and local features. This result indicates that AffectNet provides useful pretrained weights because of the wide diversity of the dataset. The diverse cultural and racial backgrounds of the images in the AffectNet dataset provides a more representative and inclusive training set, leading to a more robust and accurate recognition system. The result highlights the significance of considering the diversity of data and transfer learning in the development and deployment of FER algorithms.

The normalized confusion matrices predicted by the model trained on AffectNet for AffectNet and RAF-DB are shown in Fig.  8 a and b, respectively. The normalized confusion matrices predicted by the model after transfer learning for RAF-DB is given in Fig.  8 c. Figure  8 a and b show that the model tends to falsely classify images as “neutral”. It suggests the discriminative features learned from AffectNet are similar between “neutral” and other categories. Moreover, the comparison between Fig.  8 b and c shows that after transfer learning, the model classifies the emotions in the RAF-DB in a more accurate and even manner.

Normalized confusion matrix for AffectNet and RAF-DB ( a ) AffectNet, ( b ) RAF-DB and ( c ) RAF-DB with pretrained model.

It can be seen from the normalized confusion matrices that the classification accuracy is positively correlated with the number of images in the dataset, as given in Table 1 . In Fig.  8 a, the AffectNet dataset contains the least number of “disgust” images, which results in the lowest accuracy in the normalized confusion matrix. In contrast, the number of images of the “happy” category is the most in AffectNet and, therefore, yields the highest accuracy in the normalized confusion matrix for this category. The same conclusion can be obtained from Fig.  8 b and c for RAF-DB.

Feature maps

This study examines the important features that the network learns to classify facial emotions. The feature maps in AffectNet with softmax scores (P) exceeding 90% are visualized in Fig.  9 . It is shown that mouth, nose, and other facial lines are major information, while the eyes and ears for minor information. This is similar to the results found in Beaudry et al . 35 that the mouth is the major landmark when the neural network predicts a happy emotion. The feature maps of misclassified images are also visualized in Fig.  10 for comparisons with those that were correctly classified. By observing the feature maps of misclassified images, it is evident that the important features in the images are similar to those in the correctly classified images. It can be observed from Figs. 9 and 10 that the network tends to detect edges and lines in shallow layers and focuses more on local features, like mouth and nose, in deeper layers.

Feature maps with a softmax score greater than 90% (AffectNet).

Misclassified feature maps (AffectNet).

Asian facial emotion

The Asian facial emotion dataset 41 consists of images of 29 actors aged from 19 to 67 years old. The images were taken from frontal, 3/4 sideways and sideways angles. Figure  11 shows some example images from the Asian facial emotion dataset. The number of images of each class are given in Table 5 . There are only six labeled categories in this dataset. The “neutrality” category is not provided in this dataset. Therefore, in the output layer of the model, which was trained to predict the probabilities of 7 categories, the probability for “neutrality” was specified as zero.

Example images from the Asian facial emotion dataset 39 .

The Asian facial emotion dataset was tested with the model trained on AffectNet. The images were resized to 256 × 256 pixels and then cropped to 224 × 224 pixels with their faces centered. The derived average accuracy was 61.99%, which was slightly higher than that of AffectNet. Similar to the validation results of AffectNet, the “happy” category yielded the highest score, while “fear” and “disgust” had the lowest scores. The normalized confusion matrix is shown in Fig.  12 , and the feature maps are shown in Fig.  13 . In contrast with the feature maps of AffectNet, the discriminative locations were not centered around the mouth and nose but were located more on the right half of the face. It shows that the model lacked generalizability for Asian faces in the laboratory setting. This experiment shows that the model trained on AffectNet has limited prediction performance on other datasets.

Normalized confusion matrix produced for the Asian facial emotion dataset tested with the model trained on AffectNet.

Feature maps produced for the Asian facial emotion dataset.

The process of interpreting facial expressions is also subject to cultural and individual differences that are not considered by the model during the training phase. The feature maps in Figs. 9 and 10 show that the proposed model focused more on the mouth and nose but less on the eyes. To obtain correct FER results, subtle features such as wrinkles and eyes may also be critical. However, the proposed model does not capture features that are far from the mouth or nose. The test results obtained on the Asian face emotion dataset shows that the discriminative regions are skewed toward the right half of the face. This finding indicates that the limited generalizability of the model to Asian faces in the laboratory setting. Although AffectNet is a diverse dataset containing representations from various cultures and races, it is still limited to a tiny portion of the global population. In contrast, the RAF-DB contains similar ethnic groups and settings similar to AffectNet. The validation results obtained on the RAF-DB (77.37%) is better than that on the Asian face emotion dataset. The results show that for datasets with similar ethnic groups, the model trained on a more diverse and wilder dataset (AffectNet) performs better prediction on a more constrained dataset (the RAF-DB in this work).

This study addresses how the neural network model learns to identify facial emotions. The features displayed on emotion images were derived with a CNN, and these emotional features were visualized to determine the facial landmarks that contains major information. Conclusions drawn based on the findings are listed below.

A cross-database validation experiment was conducted for AffectNet and RAF-DB. An accuracy of 77.37% was achieved when the RAF-DB was predicted by the model trained on AffectNet. The accuracy is comparable to the result in 21 . An accuracy of 42.6% was achieved when AffectNet was predicted by the model trained on RAF-DB. These results agree with the fact that AffectNet exhibits more diversity than RAF-DB in terms of facial emotion images. Moreover, transfer learning dramatically increases the accuracy by 26.95% for RAF-DB. The finding highlights the significance of using transfer learning to improve the performance of FER algorithms by training the associated models on AffectNet for pretrained weights.

The visualized emotion feature maps show that the mouth and nose contain the major information, while the eyes and ears contain the minor information when the neural network learns to perform FER. This paradigm is similar to how human observes emotions.

When comparing the feature maps that were correctly classified (those with softmax scores exceeding 90%) with those that were incorrectly classified, it can be seen that the network model focuses on similar features with no major differences. This result indicates that FER requires the observation of large patches near distinctive areas on a face.

Data availability

The datasets applied in this study are available with authorization from the following websites for AffectNet ( http://mohammadmahoor.com/affectnet/ ), the Real-World Affective Faces Database (RAF-DB; http://www.whdeng.cn/raf/model1.html ) and the Asian facial emotion dataset ( http://mil.psy.ntu.edu.tw/ssnredb/logging.php?action=login ). However, restrictions apply to the availability of these data, which were used under license for the current study and thus are not publicly available. The data are, however, available from the authors upon reasonable request and with permission from AffectNet, the RAF-DB and the Asian facial emotion dataset. The training and analysis processes are discussed in the research methodology.

Vo, T. H., Lee, G. S., Yang, H. J. & Kim, S. H. Pyramid with super resolution for in-the-wild facial expression recognition. IEEE Access 8 , 131988–132001 (2020).

Article   Google Scholar  

Mehrabian, A. Nonverbal communication (Aldine Transaction, 2007).

Ekman, P. Darwin, deception, and facial expression. Ann. N. Y. Acad. Sci. 1000, 205–2 (Kortli & Jridi, 2020) (2006).

Farzaneh, A. H. & Qi, X. Facial expression recognition in the wild via deep attentive center loss in 2021 IEEE winter conference on applications of computer vision (WACV) 2401–2410 (IEEE, 2021).

Alnuaim, A. A. et al. Human-computer interaction for recognizing speech emotions using multilayer perceptron classifier. J. Healthc. Eng. 2022 , 6005446 (2022).

Article   PubMed   PubMed Central   Google Scholar  

Kumari, H. M. L. S. Facial expression recognition using convolutional neural network along with data augmentation and transfer learning (2022).

Ekman, P., Dalgleish, T. & Power, M. Handbook of cognition and emotion (Wiley, 1999).

Ekman, P. Are there basic emotions?. Psychol. Rev. 99 , 550–553 (1992).

Article   CAS   PubMed   Google Scholar  

Russell, J. A. A circumplex model of affect. J. Pers. Soc. Psychol. 39 , 1161–1178 (1980).

Goodfellow, I. J. et al. Challenges in representation learning: A report on three machine learning contests in Neural information processing (eds. Lee, M., Hirose, A., Hou, Z. & Kil, R) 117–124 (Springer, 2013).

Maithri, M. et al. Automated emotion recognition: Current trends and future perspectives. Comput. Method Prog. Biomed. 215 , 106646 (2022).

Article   CAS   Google Scholar  

Li, S. & Deng, W. Deep facial expression recognition: A survey. IEEE Trans. Affect. Comput. 13 , 1195–1215 (2022).

Canal, F. Z. et al. A survey on facial emotion recognition techniques: A state-of-the-art literature review. Inf. Sci. 582 , 593–617 (2022).

He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition in 2016 IEEE conference on computer vision and pattern recognition (CVPR) 770–778 (IEEE, 2016).

Mollahosseini, A., Hasani, B. & Mahoor, M. H. AffectNet: A database for facial expression, valence, and arousal computing in the wild. IEEE Trans. Affect. Comput. 10 , 18–31 (2019).

Schoneveld, L. & Othmani, A. Towards a general deep feature extractor for facial expression recognition in 2021 IEEE international conference on image processing (ICIP) 2339–2342 (IEEE, 2021).

Rajan, V., Brutti, A. & Cavallaro, A. Is cross-attention preferable to self-attention for multi-modal emotion recognition? in ICASSP 2022–2022 IEEE international conference on acoustics, speech and signal processing (ICASSP) 4693–4697 (IEEE, 2022).

Zhuang, X., Liu, F., Hou, J., Hao, J. & Cai, X. Transformer-based interactive multi-modal attention network for video sentiment detection. Neural Process. Lett. 54 , 1943–1960 (2022).

Zhang, Y., Wang, C., Ling, X. & Deng, W. Learn from all: Erasing attention consistency for noisy label facial expression recognition in Lecture notes in computer science (eds. Avidan, S., Brostow, G., Cissé, M., Farinella, G. M. & Hassner T.) 418–434 (Springer, 2022).

Savchenko, A. V., Savchenko, L. V. & Makarov, I. Classifying emotions and engagement in online learning based on a single facial expression recognition neural network. IEEE Trans. Affect. Comput. 13 , 2132–2143 (2022).

Fan, Y., Lam, J. C. K. & Li, V. O. K. Multi-region ensemble convolutional neural network for facial expression recognition in Artificial neural networks and machine learning—ICANN 2018 (eds. Kůrková, V., Manolopoulos, Y., Hammer, B., Iliadis, L. & Maglogiannis, I.) 84–94 (Springer International Publishing, 2018).

Wang, Z., Zeng, F., Liu, S. & Zeng, B. OAENet: Oriented attention ensemble for accurate facial expression recognition. Pattern Recognit. 112 , 107694 (2021).

Schoneveld, L., Othmani, A. & Abdelkawy, H. Leveraging recent advances in deep learning for audio-Visual emotion recognition. Pattern Recognit. Lett. 146 , 1–7 (2021).

Article   ADS   Google Scholar  

Hwooi, S. K. W., Othmani, A. & Sabri, A. Q. M. Deep learning-based approach for continuous affect prediction from facial expression images in valence-arousal space. IEEE Access 10 , 96053–96065 (2022).

Sun, L., Lian, Z., Tao, J., Liu, B. & Niu, M. Multi-modal continuous dimensional emotion recognition using recurrent neural network and self-attention mechanism in Proceedings of the 1st international on multimodal sentiment analysis in real-life media challenge and workshop 27–34 (ACM, 2020).

Allognon, S. O. C., de S. Britto, A. & Koerich, A. L. Continuous emotion recognition via deep convolutional autoencoder and support vector regressor in 2020 international joint conference on neural networks (IJCNN) 1–8 (IEEE, 2020).

Huang, C. Combining convolutional neural networks for emotion recognition in 2017 IEEE MIT undergraduate research technology conference (URTC) 1–4 (IEEE, 2017).

Mao, J. et al. POSTER V2: A simpler and stronger facial expression recognition network. arXiv preprint arXiv:2301.12149 (2023).

Le, N. et al. Uncertainty-aware label distribution learning for facial expression recognition in 2023 IEEE/CVF winter conference on applications of computer vision (WACV) 6088–6097 (IEEE, 2023).

Singh, S. & Prasad, S. V. A. V. Techniques and challenges of face recognition: A critical review. Proc. Comput. Sci. 143 , 536–543 (2018).

Kortli, Y., Jridi, M., Falou, A. A. & Atri, M. Face recognition systems: A survey. Sensors (Basel, Switzerland) 20 , 342 (2020).

Article   ADS   PubMed   Google Scholar  

Shirazi, M. S. & Bati, S. Evaluation of the off-the-shelf CNNs for facial expression recognition in Lecture notes in networks and systems (ed. Arai, K.) 466–473 (Springer, 2022).

Chen, D., Wen, G., Li, H., Chen, R. & Li, C. Multi-relations aware network for in-the-wild facial expression recognition. IEEE Trans. Circuits Syst. Video Technol. https://doi.org/10.1109/tcsvt.2023.3234312 (2023).

Heidari, N. & Iosifidis, A. Learning diversified feature representations for facial expression recognition in the wild. arXiv preprint arXiv:2210.09381 (2022).

Beaudry, O., Roy-Charland, A., Perron, M., Cormier, I. & Tapp, R. Featural processing in recognition of emotional facial expressions. Cogn. Emot. 28 , 416–432 (2013).

Article   PubMed   Google Scholar  

Bhattacharyya, A. et al. A deep learning model for classifying human facial expressions from infrared thermal images. Sci. Rep. 11 , 20696 (2021).

Article   ADS   CAS   PubMed   PubMed Central   Google Scholar  

Alp, N. & Ozkan, H. Neural correlates of integration processes during dynamic face perception. Sci. Rep. 12 , 118 (2022).

Siddiqi, M. H. Accurate and robust facial expression recognition system using real-time YouTube-based datasets. Appl. Intell. 48 , 2912–2929 (2018).

Li, S., Deng, W. H. & Du, J. P. Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild in 2017 IEEE conference on computer vision and pattern recognition (CVPR) 2584–2593 (IEEE, 2017).

Hu, J., Shen, L. & Sun, G. Squeeze-and-excitation networks in 2018 IEEE/CVF conference on computer vision and pattern recognition 7132–7141 (IEEE, 2018).

Chen, C. C., Cho, S. L. & Tseng, R. Y. Taiwan corpora of Chinese emotions and relevant psychophysiological data-Behavioral evaluation norm for facial expressions of professional performer. Chin. J. Psychol. 55 , 439–454 (2013).

Google Scholar  

Download references

Acknowledgements

This work was funded in part by National Science and Technology Council (project number MOST 111-2635-E-242-001 -).

Author information

Authors and affiliations.

Department of Mechanical Engineering, National Kaohsiung University of Science and Technology, Kaohsiung, Taiwan

Zi-Yu Huang, Chia-Chin Chiang & Hsin-Lung Chung

Graduate Institute of Applied Physics, National Chengchi University, Taipei, Taiwan

Jian-Hao Chen & Hsiu-Chuan Hsu

Department of Occupational Safety and Hygiene, Fooyin University, Kaohsiung, Taiwan

Yi-Chian Chen

Department of Nursing, Hsin Sheng Junior College of Medical Care and Management, Taoyuan, Taiwan

Yu-Ping Cai

Department of Computer Science, National Chengchi University, Taipei, Taiwan

Hsiu-Chuan Hsu

You can also search for this author in PubMed   Google Scholar

Contributions

Z.-Y. Huang contributed to writing the manuscript. C.-C. Chiang contributed to overseeing and finalizing the paper. J.-H. Chen conducted all computations and contributed equally as the first author. Y.-C. Chen contributed to designing the research and editing the manuscript. H.-L. Chung contributed to editing the manuscript. Y.-P. C. assessed the emotion classification field and contributed to the literature review. H.-C. H. designed the study and provided conceptual guidance. All authors discussed and reviewed the manuscript.

Corresponding authors

Correspondence to Yi-Chian Chen or Hsiu-Chuan Hsu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Huang, ZY., Chiang, CC., Chen, JH. et al. A study on computer vision for facial emotion recognition. Sci Rep 13 , 8425 (2023). https://doi.org/10.1038/s41598-023-35446-4

Download citation

Received : 08 December 2022

Accepted : 18 May 2023

Published : 24 May 2023

DOI : https://doi.org/10.1038/s41598-023-35446-4

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

Comprehensive comparison between vision transformers and convolutional neural networks for face recognition tasks.

• Marcos Rodrigo
• Carlos Cuevas
• Narciso García

Scientific Reports (2024)

Accuracy is not enough: a heterogeneous ensemble model versus FGSM attack

• Reham A. Elsheikh
• M. A. Mohamed
• Mohamed Maher Ata

Complex & Intelligent Systems (2024)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

• Artificial Intelligence
• Computer Science
• Face Recognition

Face recognition based attendance system using machine learning with location identification

• 18(1):1029-1035
• This person is not on ResearchGate, or hasn't claimed this research yet.

Abstract and Figures

Discover the world's research

• 25+ million members
• 160+ million publication pages
• 2.3+ billion citations

• Mahmoud A. Moussa
• Mohamed Hassan M. Mahmoud

• Sayali Barsagade
• Sakshi Moon
• Dhyaneshwari Itnakr
• Prof. Dhananjay Dumbere
• Samrat Dutta

• Ghalib Al-Muhaidhri

• Ananya Mani
• Shruti Bharuka
• Sakshi Kaushik

• Vikas Yadav
• G. P. Bhole
• E. Varadharajan
• S. Jeevitha
• S. Hemalatha
• Divya Pandey
• Recruit researchers
• Join for free
• Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

• IEEE Xplore Digital Library
• IEEE Standards
• IEEE Spectrum

Join Public Safety Technology

Ethical Considerations in the Use of Facial Recognition for Public Safety

The implementation of facial recognition technology for public safety purposes has sparked intense debate among policymakers, technologists, and civil liberties advocates. This powerful tool, capable of identifying individuals from digital images or video frames, offers potential benefits for law enforcement and security operations. However, its use raises significant ethical questions that society must grapple with.

At its core, facial recognition technology presents a double-edged sword. On one side, it promises enhanced capabilities to prevent crime, locate missing persons, and respond swiftly to security threats. Proponents argue that leveraging this technology could save lives and make communities safer. On the other hand, critics warn of the potential for abuse, erosion of privacy, and the risk of creating a surveillance state that undermines fundamental freedoms. Both aspects have been repeatedly demonstrated in multiple countries around the world, from western democracies to dictatorships.

The ethical considerations surrounding facial recognition are multifaceted. One primary concern is the potential for this technology to be used in ways that infringe on individual privacy and autonomy. There are valid fears that widespread deployment of facial recognition systems in public spaces could lead to a chilling effect on free expression and association. Citizens may feel constantly watched and modify their behavior accordingly, even when engaging in perfectly legal activities.

Learn about other technologies covering a wide range of Public Safety applications at the 2025 World Forum on Public Safety Technology (WF-PST) – Sign up for Alerts.

Another critical ethical issue is the potential for bias and discrimination in facial recognition algorithms. Research has shown that many current systems exhibit lower accuracy rates for certain demographic groups, particularly women and people of color. This raises serious questions about fairness and equal treatment under the law if such systems are used to make consequential decisions about individuals. 

The use of facial recognition also brings up questions of consent and transparency. In many cases, individuals may be unaware that their biometric data is being captured and analyzed. This lack of informed consent challenges notions of personal autonomy and control over one's own image and identity.

As society continues to keep up with these complex ethical considerations, it is crucial to foster open dialogue between technologists, policymakers, ethicists, and the public. Striking the right balance between leveraging the potential benefits of facial recognition for public safety while safeguarding civil liberties and human rights will be an ongoing challenge that requires careful thought and robust democratic debate.

Privacy Concerns and Individual Rights

The widespread adoption of facial recognition technology has ignited a fierce debate surrounding privacy and individual rights. As governments and private entities increasingly deploy these systems in public spaces, questions arise about the extent to which people can expect privacy in an age of ubiquitous surveillance.

Facial recognition technology fundamentally impacts individual privacy by enabling the large-scale collection and analysis of biometric data. Unlike other forms of identification, such as ID cards or passwords, facial features are inherently public and difficult to conceal. This means that individuals may be identified and tracked without their knowledge or consent simply by moving through public spaces where cameras are present.

The implications for data protection and online privacy are profound. As facial recognition systems become more sophisticated and widespread, they create vast databases of personal information. These databases can potentially be used to track individuals' movements, associations, and behaviors over time. This level of surveillance raises concerns about the right to anonymity in public spaces and the potential for such data to be used for purposes beyond public safety, such as targeted advertising or social control.

Measures to protect privacy in public surveillance do exist, but their effectiveness is often questioned. Some jurisdictions have implemented regulations requiring clear signage to inform the public when facial recognition is in use. Others have mandated strict data retention policies, limiting how long biometric information can be stored. However, critics argue that these measures are insufficient given the scale and power of modern facial recognition systems.

One approach to balancing public safety needs with privacy concerns is the use of privacy-enhancing technologies. For example, some systems employ encryption techniques to ensure that facial data is anonymized and cannot be easily linked to individual identities without proper authorization. Another approach is to limit the use of facial recognition to specific, high-risk areas or events, rather than deploying it universally.

The risks of data breaches in facial recognition systems are particularly concerning. Unlike passwords or credit card numbers, biometric data cannot be changed if compromised. A breach of a facial recognition database could potentially expose individuals to identity theft, stalking, or other forms of harassment. Moreover, the sensitive nature of biometric data means that its exposure could have long-lasting consequences for affected individuals.

Regulations addressing privacy in facial recognition vary widely across different jurisdictions. In the European Union, the General Data Protection Regulation (GDPR) classifies biometric data as a special category of personal information, subject to strict protections. This includes requirements for explicit consent and limitations on data processing. In the United States, regulation is more fragmented, with some states implementing strict controls on facial recognition use while others have few restrictions. 

At the federal level in the US, there is ongoing debate about the need for comprehensive legislation to address the privacy implications of facial recognition. Proposed measures include requiring warrants for law enforcement use of the technology, mandating regular audits of facial recognition systems, and giving individuals the right to know when their biometric data has been collected and how it is being used.

As facial recognition technology continues to evolve, so too must the legal and ethical frameworks governing its use. Balancing the potential benefits of enhanced public safety with the fundamental right to privacy remains a critical challenge. It requires ongoing dialogue between policymakers, technologists, civil liberties advocates, and the public to ensure that the deployment of facial recognition does not come at the cost of essential individual rights and freedoms.

Ultimately, the goal must be to create a regulatory environment that allows for the responsible use of facial recognition technology while providing robust protections for individual privacy and data security. This may involve a combination of technological safeguards, legal restrictions, and public oversight mechanisms to ensure that the power of facial recognition is harnessed in a way that respects fundamental human rights and democratic values.

Accuracy and Bias in Facial Recognition Technology

The efficacy and fairness of facial recognition systems hinges on their accuracy and potential for bias. As these technologies become increasingly integrated into public safety and law enforcement operations, understanding the factors that influence their performance and addressing inherent biases has become a pressing concern for technologists, policymakers, and civil rights advocates.

The accuracy of facial recognition systems is influenced by a complex interplay of factors. At the most basic level, the quality and quantity of training data used to develop these algorithms play a crucial role. Systems trained on diverse, representative datasets tend to perform better across different demographic groups. However, many existing face recognition systems have been developed using datasets that overrepresent certain populations, leading to disparities in accuracy.

Environmental factors also significantly impact accuracy. Lighting conditions, camera angles, and image resolution can all affect a system's ability to correctly identify individuals. In real-world applications, such as mass surveillance in public spaces, these variables can be difficult to control, potentially leading to unreliable results.

The issue of bias in facial recognition technology has garnered significant attention in recent years. Multiple studies have demonstrated that many current systems exhibit lower accuracy rates for certain demographic groups, particularly women and people of color. This bias can stem from various sources, including underrepresentation in training data, algorithmic design choices, and even the underlying physics of how different skin tones interact with camera sensors.

The implications of these biases for public safety applications are profound. False positives in facial recognition systems can lead to wrongful accusations, arrests, or other adverse actions against innocent individuals. This risk is particularly concerning for already marginalized communities that may be disproportionately affected by biased algorithms.

Addressing bias in facial recognition systems is an ongoing challenge that requires a multifaceted approach. One strategy involves diversifying the datasets used to train these algorithms, ensuring they include a representative sample of faces across different ages, genders, ethnicities, and other demographic factors. Some researchers are also exploring the use of artificial intelligence techniques to identify and mitigate bias in existing systems.

Another approach focuses on improving the transparency and accountability of facial recognition algorithms. This includes developing standardized testing protocols to assess accuracy and bias across different demographic groups and making these results publicly available. Some advocates argue for the implementation of "algorithmic impact assessments" before deploying facial recognition systems in sensitive applications like law enforcement.

The question of whether facial recognition systems can be trained to completely eliminate bias remains open. While significant progress has been made in improving accuracy across different demographic groups, achieving perfect fairness may be an elusive goal given the complexity of human facial features and the inherent limitations of machine learning algorithms.

As the debate over facial recognition and mass surveillance continues, it is crucial to consider not only the technical aspects of accuracy and bias but also the broader societal implications. Even a highly accurate and unbiased system raises concerns about privacy, civil liberties, and the potential for abuse when deployed at scale.

Ultimately, addressing the challenges of accuracy and bias in facial recognition technology requires ongoing collaboration between technologists, policymakers, and civil society. It involves not only technical improvements but also the development of robust governance frameworks to ensure these powerful tools are used responsibly and ethically.

As facial recognition systems continue to evolve and improve, it is essential to maintain a critical perspective on their limitations and potential risks. While these technologies offer promising capabilities for enhancing public safety, their deployment must be carefully balanced against the fundamental rights and freedoms that form the foundation of democratic societies.

Regulation and Governance of Facial Recognition

The rapid advancement and widespread deployment of facial recognition technology have outpaced the development of comprehensive legal frameworks to govern its use. As a result, the regulation of facial recognition, particularly in the context of public safety applications, varies significantly across jurisdictions and remains a subject of intense debate among policymakers, legal experts, and civil liberties advocates.

In the United States, there is currently no federal law specifically regulating the use of facial recognition technology. Instead, a patchwork of state and local laws has emerged to address concerns about privacy, bias, and potential misuse. Some cities, such as San Francisco and Boston, have enacted outright bans on the use of facial recognition by government agencies, citing concerns about civil liberties and the technology's potential for abuse.

At the state level, regulations range from comprehensive restrictions to more limited oversight measures. Illinois, for example, has implemented the Biometric Information Privacy Act (BIPA), which requires companies to obtain explicit consent before collecting or using biometric data, including facial recognition. Other states have focused on specific applications, such as restricting the use of facial recognition in schools or limiting law enforcement access to driver's license databases for facial recognition searches.

In contrast to the fragmented approach in the U.S., the European Union has taken a more unified stance on regulating facial recognition and personal information . The General Data Protection Regulation (GDPR) classifies facial recognition data as a special category of personal data, subject to stringent protection requirements. Additionally, the proposed AI Act seeks to establish a risk-based regulatory framework for artificial intelligence applications, including facial recognition systems used in public spaces.

Other countries have adopted varying approaches to facial recognition regulation. China, for instance, has embraced the technology for public safety and social governance purposes, with fewer restrictions on government use. However, it has recently introduced regulations requiring consent for facial recognition in commercial settings. In contrast, Canada has taken a more cautious approach, with its Privacy Commissioner calling for strict limits on the use of facial recognition by law enforcement and intelligence agencies.

Global standards for facial recognition use are still in their infancy. Organizations such as the National Institute of Standards and Technology (NIST) in the U.S. have developed testing protocols for assessing the accuracy and bias of facial recognition algorithms. However, these standards primarily focus on technical performance rather than ethical or governance issues.

The development of international norms and standards for facial recognition is an ongoing process, with bodies like the United Nations and the Organization for Economic Co-operation and Development (OECD) working to establish guidelines for the responsible development and use of AI technologies, including facial recognition.

Legal frameworks to protect citizens from misuse of facial recognition technology typically focus on several key areas:

• Consent and Transparency: Many regulations require explicit consent from individuals before their biometric data can be collected or processed. They also mandate clear disclosure of when and where facial recognition systems are in use.
• Data Protection: Laws often stipulate strict requirements for the security, storage, and deletion of facial recognition data to prevent unauthorized access or misuse.
• Accuracy and Fairness: Some legal frameworks require regular auditing of facial recognition systems to ensure they meet minimum standards for accuracy and do not exhibit bias against particular demographic groups.
• Limitations on Use: Regulations may restrict the purposes for which facial recognition can be used, often requiring a compelling public interest or court order for law enforcement applications.
• Accountability and Redress: Legal frameworks often include mechanisms for individuals to challenge decisions made based on facial recognition and seek redress for any harm caused by misuse of the technology.

As facial recognition technology continues to evolve, so too must the legal and regulatory frameworks governing its use. The challenge lies in striking a balance between harnessing the potential benefits of this technology for public safety while safeguarding individual rights and civil liberties.

Moving forward, it is likely that we will see increased calls for harmonized global standards and more comprehensive national regulations. These efforts will need to address not only the technical aspects of facial recognition but also the broader ethical and societal implications of its widespread use.

Ultimately, effective regulation and governance of facial recognition technology will require ongoing collaboration between policymakers, technologists, legal experts, and civil society. It must be flexible enough to adapt to rapid technological advancements while providing robust protections for individual privacy and fundamental rights.

Balancing Public Safety and Civil Liberties

The deployment of facial recognition technology in the name of public safety has ignited a contentious debate about the balance between security measures and civil liberties. As governments and law enforcement agencies increasingly turn to this powerful tool, questions have arisen about its effectiveness in preventing crime and the potential for abuse that could infringe on individual rights and freedoms.

Proponents of facial recognition argue that it can be a highly effective tool for homeland security and crime prevention. The technology has been credited with aiding in the identification and apprehension of suspects in various criminal cases, from petty theft to more serious offenses. In high-security environments such as airports, facial recognition systems can quickly process large numbers of individuals, potentially identifying known threats or persons of interest.

However, measuring the true effectiveness of facial recognition in preventing crime is challenging. While there are certainly success stories, critics argue that the deterrent effect of such systems may be overstated. They point out that determined criminals may find ways to circumvent facial recognition, such as wearing disguises or avoiding monitored areas. Additionally, the prevalence of false positives in many systems raises questions about how many innocent individuals might be wrongly flagged or inconvenienced for every genuine threat detected.

The potential for abuse of facial recognition technology by authorities is a significant concern for civil liberties advocates. Government surveillance , if unchecked, could lead to a chilling effect on free speech and assembly. There are fears that facial recognition could be used to track individuals' movements, monitor political dissent, or harass marginalized communities. Historical precedents of government overreach in surveillance activities serve as cautionary tales about the risks of unchecked monitoring power. One example is the United States Federal Bureau of Investigation’s (FBI) COINTELPRO program, a series of covert and illegal projects conducted between 1956 and 1971. COINTELPRO was aimed at surveilling, infiltrating, discrediting, and disrupting American political organizations that the FBI perceived as subversive. The program was exposed by the Church Committee, a US Senate select committee formed in 1975 to investigate abuses by US Intelligence Agencies.

The impact of facial recognition on vulnerable populations is particularly concerning. Studies have shown that many current facial recognition systems exhibit higher error rates for certain demographic groups, particularly women and people of color. This bias could lead to disproportionate scrutiny or false accusations against already marginalized communities. Moreover, facial recognition could be used to target specific groups, such as undocumented immigrants or political activists, potentially exacerbating existing social inequalities.

The question of whether facial recognition can be used without infringing on civil liberties is complex and contentious. Some argue that with proper safeguards and oversight, the technology can be a valuable tool for public safety without compromising individual rights. Proposed measures include strict limitations on data retention, requirements for judicial warrants before using facial recognition for identification purposes, and regular audits to ensure system accuracy and detect potential bias.

Others contend that the risks to privacy and civil liberties are inherent in the technology itself and that its use in public spaces is fundamentally incompatible with the principles of a free society. They argue that the mere presence of such systems creates a panopticon effect, where individuals modify their behavior due to the perception of constant surveillance.

The debate over facial recognition also touches on broader questions about the nature of privacy in the digital age. As technology continues to advance, traditional notions of privacy in public spaces are being challenged. The ability to remain anonymous in public, once taken for granted, is increasingly under threat as sophisticated surveillance technologies become more pervasive.

Striking the right balance between public safety and civil liberties in the context of facial recognition requires careful consideration of several factors:

• Transparency: Clear policies and public disclosure about when and how facial recognition is being used are essential for maintaining public trust and enabling democratic oversight.
• Accountability: Robust mechanisms for auditing facial recognition systems and holding authorities accountable for misuse are crucial.
• Proportionality: The use of facial recognition should be proportionate to the security threat and limited to specific, high-risk scenarios rather than deployed as a blanket surveillance tool.
• Data protection: Strict controls on the collection, storage, and sharing of biometric data are necessary to prevent misuse and protect individual privacy.
• Consent and opt-out options: Wherever possible, individuals should have the ability to consent to or opt out of facial recognition systems, particularly in non-critical applications.

As society continues to grapple with these issues, it is clear that the use of facial recognition technology for public safety purposes will remain a contentious topic. The challenge lies in developing frameworks that allow for the responsible use of this powerful tool while robustly protecting civil liberties and democratic values. This will require ongoing dialogue, careful policy-making, and a commitment to upholding fundamental rights in the face of evolving technological capabilities.

Future Implications of Widespread Facial Recognition Use

As facial recognition technology continues to advance and proliferate, its potential future implications for society are both far-reaching and profound. The evolution of this technology is likely to shape various aspects of public and private life, raising new ethical challenges and fundamentally altering the way we interact with our environment and each other.

In the years to come, facial recognition is expected to become increasingly sophisticated and ubiquitous. Advancements in artificial intelligence and machine learning will likely lead to more accurate and efficient systems capable of recognizing individuals in a wider range of conditions and contexts. This could lead to the integration of facial recognition into an ever-expanding array of devices and services, from smartphones and smart home systems to public transportation and retail environments.

One potential long-term societal impact of ubiquitous facial recognition is the erosion of anonymity in public spaces. As these systems become more prevalent, the ability to move through society without being identified and tracked may become increasingly difficult. This could fundamentally alter social dynamics and behaviors, potentially leading to a society where individuals are always conscious of being observed and identified.

The collection and use of biometric information on such a massive scale could also have profound implications for personal privacy and data security. As facial recognition databases grow, they become increasingly valuable targets for cybercriminals and state actors. The potential for this sensitive data to be breached, stolen, or misused could have long-lasting consequences for individuals and society as a whole.

For future generations, growing up in a world where facial recognition is commonplace may reshape expectations of privacy and personal identity. Young people may develop different attitudes towards sharing their biometric data and may be more accustomed to constant identification in public spaces . This could lead to shifts in social norms and behaviors, with potential impacts on everything from dating and socializing to political activism and artistic expression.

The advancement of facial recognition technology could also lead to new forms of social stratification and discrimination. If access to certain spaces or services becomes contingent on facial recognition, those who choose not to participate or are unable to do so due to technical limitations may face exclusion. This could create new forms of digital divides and exacerbate existing social inequalities.

On the other hand, the evolution of facial recognition could also bring about positive changes. In healthcare, for instance, advanced facial recognition could aid in early diagnosis of certain conditions or help monitor patient well-being. In education, it could enable personalized learning experiences and enhance campus security. The technology could also play a role in combating human trafficking and locating missing persons.

However, these potential benefits come with significant ethical challenges. As facial recognition capabilities advance, questions about consent, data ownership, and the right to remain anonymous will become increasingly complex. The ability to instantly identify individuals, potentially combined with other data sources, raises concerns about the creation of comprehensive personal profiles that could be used for surveillance, manipulation, or control.

The integration of facial recognition with other emerging technologies, such as augmented reality, could create new paradigms of human interaction. Imagine a world where looking at someone instantly brings up their social media profile, criminal record, or credit score. Such capabilities could dramatically alter social dynamics and raise new questions about privacy, judgment, and the nature of human relationships.

Another ethical challenge that may arise from advanced facial recognition capabilities is the potential for emotional and behavioral analysis. As these systems become more sophisticated, they may be able to infer an individual's emotional state, intentions, or even thoughts based on subtle facial expressions. This level of insight into personal states raises significant ethical concerns about mental privacy and autonomy.

The use of facial recognition in law enforcement and national security will likely continue to be a contentious issue. While the technology may enhance public safety efforts, the risk of creating a surveillance state that infringes on civil liberties will remain a serious concern. Striking the right balance between security and freedom in an age of advanced facial recognition will be an ongoing challenge for societies worldwide.

As facial recognition technology becomes more prevalent in public spaces, there may also be a growing movement for "facial recognition-free" zones. These could be areas where individuals can expect to move about without being identified or tracked, similar to current concepts of phone-free or Wi-Fi-free spaces. The designation and protection of such zones could become an important aspect of urban planning and public policy.

Ultimately, the future implications of widespread facial recognition use will depend on how societies choose to regulate and deploy this powerful technology. The decisions made in the coming years about the legal, ethical, and social frameworks governing facial recognition will play a crucial role in shaping its impact on future generations.

As we navigate this complex landscape, it will be essential to foster ongoing public dialogue about the role of facial recognition in society. Engaging diverse perspectives from technologists, ethicists, policymakers, and the general public will be crucial in developing approaches that harness the benefits of this technology while safeguarding fundamental human rights and values.

The challenge ahead lies in creating a future where facial recognition enhances our lives and society without compromising the essence of human dignity, privacy, and freedom. This will require careful consideration, robust democratic processes, and a commitment to ethical principles that place the well-being of individuals and communities at the forefront of technological advancement.

The integration of facial recognition technology into public safety measures presents a complex landscape of opportunities and challenges. As this powerful tool continues to evolve and proliferate, society finds itself at a critical juncture, forced to grapple with the delicate balance between enhancing security and preserving fundamental rights and liberties.

The ethical considerations surrounding facial recognition are multifaceted and far-reaching. From privacy concerns and potential biases to questions of consent and data security, the implementation of this technology raises significant issues that demand careful thought and robust public debate. The potential for abuse and the risk of creating a surveillance state that chills free expression and association cannot be overlooked.

At the same time, the potential benefits of facial recognition for public safety and various other applications are substantial. When used responsibly and with proper safeguards, this technology has the power to aid in crime prevention, enhance security in high-risk areas, and contribute to various positive societal outcomes.

As we look to the future, the implications of widespread facial recognition use are profound and far-reaching. The technology has the potential to reshape social norms, alter our understanding of privacy in public spaces, and create new paradigms of human interaction. It also presents new challenges in terms of data protection, consent, and the right to anonymity.

Moving forward, it is crucial that the development and deployment of facial recognition technology be guided by strong ethical principles, robust legal frameworks, and ongoing public dialogue. Policymakers, technologists, civil liberties advocates, and citizens must work together to create governance structures that harness the benefits of this technology while steadfastly protecting individual rights and societal values.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Recommended Articles
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

A method for the pattern recognition of acoustic emission signals using blind source separation and a cnn for online corrosion monitoring in pipelines with interference from flow-induced noise.

1. Introduction

2. theory and method, 2.1. the proposed model, 2.2. signal-separation module, 2.3. feature-extraction module, 2.4. acoustic source-recognition module, 3. ae monitoring experimental setup, 3.1. experimental platform, 3.2. dataset construction, 4. results and discussion, 4.1. blind source separation of signals, 4.2. feature extraction and model training, 4.3. comparison of model effects, 4.4. hyperparameter optimization analysis, 5. conclusions, author contributions, institutional review board statement, informed consent statement, data availability statement, conflicts of interest.

• Gao, Y.; Piltan, F.; Kim, J.-M. A Hybrid Leak Localization Approach Using Acoustic Emission for Industrial Pipelines. Sensors 2022 , 22 , 3963. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Suyama, F.M.; Delgado, M.R.; da Silva, R.D.; Centeno, T.M. Deep neural networks based approach for welded joint detection of oil pipelines in radiographic images with Double Wall Double Image exposure. Ndt E Int. 2019 , 105 , 46–55. [ Google Scholar ] [ CrossRef ]
• Askari, M.; Aliofkhazraei, M.; Afroukhteh, S. A comprehensive review on internal corrosion and cracking of oil and gas pipelines. J. Nat. Gas Sci. Eng. 2019 , 71 , 102971. [ Google Scholar ] [ CrossRef ]
• Luo, J.-H.; Li, L.-F.; Zhu, L.-X.; Zhang, L.; Wu, G.; Zhao, X.-W. Oil-Pipe Cracking and Fitness-for-Service Assessment. Metals 2022 , 12 , 1236. [ Google Scholar ] [ CrossRef ]
• Ravanbod, H. Application of neuro-fuzzy techniques in oil pipeline ultrasonic nondestructive testing. Ndt E Int. 2005 , 38 , 643–653. [ Google Scholar ] [ CrossRef ]
• Scislo, L. Single-Point and Surface Quality Assessment Algorithm in Continuous Production with the Use of 3D Laser Doppler Scanning Vibrometry System. Sensors 2023 , 23 , 1263. [ Google Scholar ] [ CrossRef ]
• Scislo, L.; Szczepanik-Scislo, N. Quantification of Construction Materials Quality via Frequency Response Measurements: A Mobile Testing Station. Sensors 2023 , 23 , 8884. [ Google Scholar ] [ CrossRef ]
• Erlinger, T.; Kralovec, C.; Schagerl, M. Monitoring of Atmospheric Corrosion of Aircraft Aluminum Alloy AA2024 by Acoustic Emission Measurements. Appl. Sci. 2023 , 13 , 370. [ Google Scholar ] [ CrossRef ]
• Jha, S.K.; Narayanan, S. On the acoustic emission characteristics of airfoils with different trailing edge configurations. Proc. Inst. Mech. Eng. Part G-J. Aerosp. Eng. 2023 , 237 , 2008–2026. [ Google Scholar ] [ CrossRef ]
• Zafar, T.; Kamal, K.; Mathavan, S.; Hussain, G.; Alkahtani, M.; Alqahtani, F.M.; Aboudaif, M.K. A Hybrid Approach for Noise Reduction in Acoustic Signal of Machining Process Using Neural Networks and ARMA Model. Sensors 2021 , 21 , 8023. [ Google Scholar ] [ CrossRef ]
• Kim, S.J.; Kim, K.; Hwang, T.; Park, J.; Jeong, H.; Kim, T.; Youn, B.D. Motor-current-based electromagnetic interference de-noising method for rolling element bearing diagnosis using acoustic emission sensors. Measurement 2022 , 193 , 110912. [ Google Scholar ] [ CrossRef ]
• Suwansin, W.; Phasukkit, P. Deep Learning-Based Acoustic Emission Scheme for Nondestructive Localization of Cracks in Train Rails under a Load. Sensors 2021 , 21 , 272. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Bhange, P.; Joshi, D.K.; Pandu, S.K.; Mankari, K.; Acharyya, S.G.; Sridhar, K.; Acharyya, A. Real-time fatigue crack growth rate estimation methodology for structural health monitoring of ships. IEEE Sens. J. 2022 , 22 , 19729–19738. [ Google Scholar ] [ CrossRef ]
• Megid, W.A.; Mejia, J.; Modwel, R.; Hay, D.R. Monitoring welded joints of steel pressure vessels using acoustic emission: Case study. Trans. Indian Inst. Met. 2022 , 75 , 2199–2209. [ Google Scholar ] [ CrossRef ]
• Kietov, V.; Mandel, M.; Krueger, L. Combination of Electrochemical Noise and Acoustic Emission for Analysis of the Pitting Corrosion Behavior of an Austenitic Stainless Cast Steel. Adv. Eng. Mater. 2019 , 21 , 1800682. [ Google Scholar ] [ CrossRef ]
• Wu, K.; Kim, J.-Y. Acoustic emission monitoring during open-morphological pitting corrosion of 304 stainless steel passivated in dilute nitric acid. Corros. Sci. 2021 , 180 , 109224. [ Google Scholar ] [ CrossRef ]
• Rai, A.; Ahmad, Z.; Hasan, M.J.; Kim, J.-M. A Novel Pipeline Leak Detection Technique Based on Acoustic Emission Features and Two-Sample Kolmogorov-Smirnov Test. Sensors 2021 , 21 , 8247. [ Google Scholar ] [ CrossRef ]
• Thang Bui, Q.; Kim, J.-M. Crack detection and localization in a fluid pipeline based on acoustic emission signals. Mech. Syst. Signal Process. 2021 , 150 , 107254. [ Google Scholar ]
• Kafle, M.D.; Fong, S.; Narasimhan, S. Active acoustic leak detection and localization in a plastic pipe using time delay estimation. Appl. Acoust. 2022 , 187 , 108482. [ Google Scholar ] [ CrossRef ]
• Liang, Z.; Wang, A.; Yu, Y.; Yang, P. Research on early weak structural damage detection of aeroengine intershaft bearing based on acoustic emission technology. Struct. Health Monit.-Int. J. 2021 , 20 , 3113–3122. [ Google Scholar ] [ CrossRef ]
• Lin, Q.; Lyu, F.; Yu, S.; Xiao, H.; Li, X. Optimized Denoising Method for Weak Acoustic Emission Signal in Partial Discharge Detection. IEEE Trans. Dielectr. Electr. Insul. 2022 , 29 , 1409–1416. [ Google Scholar ] [ CrossRef ]
• Yu, A.; Liu, X.; Fu, F.; Chen, X.; Zhang, Y. Acoustic Emission Signal Denoising of Bridge Structures Using SOM Neural Network Machine Learning. J. Perform. Constr. Facil. 2023 , 37 , 04022066. [ Google Scholar ] [ CrossRef ]
• Yang, W.; Li, X.; Wang, Y.; Zheng, Y.; Guo, P. Novel method for detecting weak acoustic emission signals based on the similarity of time-frequency spectra. Geophysics 2022 , 87 , V143–V154. [ Google Scholar ] [ CrossRef ]
• Skal’s’kyi, V.R.; Nazarchuk, Z.T.; Dolins’ka, I.Y.; Yarema, R.Y.; Selivonchyk, T.V. Acoustic-emission diagnostics of corrosion defects in materials (a survey). Part. 2. corrosion cracking of metals. applied aspects of application of the method. Mater. Sci. 2018 , 53 , 431–443. [ Google Scholar ] [ CrossRef ]
• Liu, H.; Zhang, H.; Huang, X.; Kong, Z.; Yang, J.; Yang, Y. Research on noise source separation and sound quality prediction for electric powertrain. Appl. Acoust. 2022 , 199 , 109034. [ Google Scholar ] [ CrossRef ]
• Lin, J.; Zhang, A.M. Fault feature separation using wavelet-ICA filter. Ndt E Int. 2005 , 38 , 421–427. [ Google Scholar ] [ CrossRef ]
• Bandara, S.; Rajeev, P.; Gad, E.; Sriskantharajah, B.; Flatley, I. Damage detection of in service timber poles using Hilbert-Huang transform. Ndt E Int. 2019 , 107 , 102141. [ Google Scholar ] [ CrossRef ]
• Zhang, P.; Gao, D.; Lu, Y.; Kong, L.; Ma, Z. Online chatter detection in milling process based on fast iterative VMD and energy ratio difference. Measurement 2022 , 194 , 111060. [ Google Scholar ] [ CrossRef ]
• Lee, S.-K.; Lee, H.; Back, J.; An, K.; Yoon, Y.; Yum, K.; Kim, S.; Hwang, S.-U. Prediction of tire pattern noise in early design stage based on convolutional neural network. Appl. Acoust. 2021 , 172 , 107617. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Share and Cite

Wang, X.; Xu, S.; Zhang, Y.; Tu, Y.; Peng, M. A Method for the Pattern Recognition of Acoustic Emission Signals Using Blind Source Separation and a CNN for Online Corrosion Monitoring in Pipelines with Interference from Flow-Induced Noise. Sensors 2024 , 24 , 5991. https://doi.org/10.3390/s24185991

Wang X, Xu S, Zhang Y, Tu Y, Peng M. A Method for the Pattern Recognition of Acoustic Emission Signals Using Blind Source Separation and a CNN for Online Corrosion Monitoring in Pipelines with Interference from Flow-Induced Noise. Sensors . 2024; 24(18):5991. https://doi.org/10.3390/s24185991

Wang, Xueqin, Shilin Xu, Ying Zhang, Yun Tu, and Mingguo Peng. 2024. "A Method for the Pattern Recognition of Acoustic Emission Signals Using Blind Source Separation and a CNN for Online Corrosion Monitoring in Pipelines with Interference from Flow-Induced Noise" Sensors 24, no. 18: 5991. https://doi.org/10.3390/s24185991

Article Metrics

Article access statistics, further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.