Objectives: In recent years, the demand for automatic emotion recognition systems has increased for use in various fields, including efforts to modify an individual's mood to improve mental health, as well as assisting in identifying the emotions of children with autism spectrum disorder who struggle to express their emotional states. Deep learning networks are linked with face Detection algorithms. If a deep learning network is used alone, any object detected in the image will be considered as a face and will be processed to determine whether it has emotions or not. This results in high computational complexity, very high response time, and low accuracy if there is more than one face in the same image. Methods: In this research, images were first introduced into the Fer-Net convolutional neural network after performing some preprocessing operations. The Fer-Net was selected after experimenting with several other CNN. Facial features were then extracted from the network, and these extracted features were classified into four basic emotions. Additionally, several standard databases were tested individually, such as FER-2013 and AffectNet, for training and evaluating the network. Subsequently, the previous databases were merged with other databases like RAF-DB and CK+ to increase the number of training samples and evaluation samples in order to avoid the issue of overfitting. Finally, we linked facial detection with the classification network obtained from the trained model using the MTCNN algorithm to identify the faces present in the image before analyzing the facial features and determining the emotions extracted from them. Results: First, Data Augmentation was implemented on the data in standard database (Fer-2013), and the overfitting problem was resulted. The experimental results showed that the (Train Accuracy) value during the training epochs did not exceed more than 0.7, while the (Val Accuracy) value did not exceed more than 0.55 during the evaluation phase. Moreover, the value of error (Train Loss), it started with values above 2 and then decreased until it reached 0.8, while the error value during the evaluation phase (Val Loss) maintained large values until it reached a value of 1.2. When we replaced the Fer-2013 dataset with the AffectNet dataset, the error value during the evaluation phase (Val Loss) exceeded 4, which is a very large value. Finally, we merged several standard emotion recognition datasets (Fer-2013, AffectNet, RAF-DB, CK+) where the images for each class were grouped together, thus increasing the number of samples for training and evaluation. The results showed an increase in classification accuracy of over 0.95 and a decrease in error to approximately 0.15. Conclusions: Experimental results demonstrated the proposed system’s effectiveness and capability to detect the primary emotion through facial expression, achieving a higher accuracy than other related studies.