View of Recognition of Human Sentiment from Image using Machine Learning

(1)

1802 http://annalsofrscb.ro

Recognition of Human Sentiment from Image using Machine Learning

Gagandeep Kaur Saini

¹

, Harshit Chouhan

²

, Sharlabh Kori

³

, Akanksha Gupta

⁴

, Mohammad Shabaz

^5*

, Vishal Jagota

⁶

, Bhupesh Kumar Singh

⁷

1Department of Computer Science and Engineering, Lovely Professional University, India.

5*Department of Computer Science and Engineering, Lovely Professional University, India.

E-mail: [email protected]

6Madanpalle Institute of Technology & Science, India.

7Arba Minch University, Ethiopia.

ABSTRACT

Human Sentiment is an important area of work to improve communication between human and machine. Emotional tension makes adoption very difficult. Quondam activities are proposed to capture sentiments using non-modem-based methods such as facial expressions, speech or voice input only. In this paper, we tried to recognize human sentiment from image using machine learning with tensor flow. The dataset which is used in the model is human images which are categorized into different human sentiments using machine learning and it tells the prediction score for each sentiment that is made by a human in the image.

KEYWORDS

Human Sentiments, Emotions, Expressions, Machine Learning, Tensor Flow.

Introduction

The most common expression of the concept of emotions can be found "as a natural state of the intellect of human nature, its emotions, or relationships with others". Missing showing the ability to motivate all possible motives negative or neutral. This is the most important information for understand emotions as a clever agent. It's too much and it is difficult to find emotions and distinguish among them. Before decades or two feelings began to be as disturbing as an important addition in relation to the world of modern technology. It inspires hope the dawn of intelligence services. Think of the earth there machines hear what people need or want. By file a special kind of calculation at that time that machine could do predicting other results and humanity can avoid critical situations and much more. Humans have great power and intelligence because of emotional enhancement but less effective than machines. But what if the machines acquire these special human features? It will be the strongest addition to the technology ever made and making dreams come true is the first step; train the system to recognize and recognize emotions. This the beginning of a wise plan. Intelligent Systems exist efficiency in prediction and classification decision on various aspects of active health. In particular, emotional awareness through deep learning has become a thing of the past an exciting area for research on its wisdom which works. This method mainly contains to get feelings for the different types of input taken in the conduct of a person's condition. A technology i.e. the neural network receives sensations with in-depth reading. Because of its stated problem previously, a system for stellar emotional awareness efficiency and accuracy are required.

Emotions play an important role when it comes to art. Unintentionally or not, when an art piece was created, the artist introduces this emotional component to his work and that makes us feel different. When they see art, for example during a novel reading, people can feel emotions, because they are drawn to stories that portray characters who act and feel, who have desires and fears, who achieve success or fail. Myths readers have rich emotional experiences and better empathy skills than those who do not read books.

This view has two major implications for communication between literature and human emotions. First, literature requires that we use our senses to understand it, or better yet, we should use our knowledge of human emotions to understand the feelings and feelings of fictional characters. Second, the emotional experience we find in literature is

(2)

the same kind we have in real life, making books a valid source of expression for human emotions.

All of the above means that emotions are deeply entrenched in the artwork, and therefore need to be studied in this context not only by human scholars but also by psychologists, because research in this way can benefit their understanding of both art and emotion.

Although one of the first jobs in computerized computer therapy came from the field of artificial intelligence (AI), it was only a few years later that the first work of computer-assisted emotional modelling in literature was published. It concluded that the "emotional tone" of the story could be in the reader's interest. The results of the research suggest that large-scale “emotional tone” analysis of text collection is possible with the help of a computer program. There are three implications for these findings. First, they suggested that by identifying the emotional tones of text verses one could show the relevant patterns of a image, which could be used to challenge or test existing image ideas.

Second, their approach to modelling shows that literary style structures can be defined on the basis of their emotional interest and not just linguistic features. And finally, they suggest that active images (speeches, memos, advertising, emotions) can be carried out through an emotional analysis program to assess whether it will have the intended effect or not.

Machine Learning and Tensor Flow

Machine learning (ML) is a branch of Artificial Intelligence that furthers the idea that, by providing access to proper data, machines can read on their own how to solve the file specific problem. By using complex calculations as well mathematical tools, ML provides capable machines independently of intellectual functions that have been traditionally solved by humans.

Tensor Flow is an open source library for research and creative machine. Tensor Flow provides beginner and professional APIs for desktop, mobile, web, and cloud desktop applications. The best use of Google Tensor Flow are excellent applications for in-depth learning. In-depth reading is typically for example machine information / recognition, and is linked to photos, video, audio, and voice, content and timeline information. It collects and collects such information with superficial accuracy. This can be done by accepting various items, for example, Ball, Cat, Bottle, and Car and so on. It can use Android as its component to use a mobile camera to prepare data pointers and see various objects in an ongoing process.

The Tensor Flow model was trained to classify images into 1,000 categories. The ML.NET model uses part of the Tensor Flow model in its pipeline to train the image separation model. Training a photo segmentation model from scratch requires setting up millions of parameters, a ton of training details with a label and a large number of computer resources (hundreds of hours of GPU). While it doesn't work like training a custom model from scratch, transferring learning allows you to cut through this process by working with thousands of images compared to millions of labeled images and building a customized model quickly (within an hour on a machine without a GPU).

Learning transfer is the process of applying the knowledge while solving the problem and applying it to a different problem which is somewhat related to the previous problem. In-depth learning is a piece of Learning Machines, which transforms areas such as Computer Vision and Speech Recognition.

In-depth learning models are trained using large labeled data sets and neural networks that contain multiple learning layers. In-depth learning: Do better with other tasks like Computer Vision and requires a greater amount of training details. Image Separation is a standard Machine Learning function that allows us to automatically classify images into categories such as: Finding a person's face in a photo.

Application of Emotion and Sentiment Analysis

Robots and Artificial Intelligence (AI)

While there are major breakthroughs between robots and AI, the first is primarily an engineering field dealing with the construction and use of robots, while the latter focuses on its own functioning including but not limited to decision making, problem solving, and consultation. This includes emotional intelligence, as more and more robots

(3)

are being developed today that serve not only tangible principles (e.g., cleaning, appliances) but also social ones. The impetus for computer programming in robotics and AI, therefore, is to build robots and realistic human-like agents through communication and imagination.

Robots and visual agents that have been able to detect and express emotions have been part of the robotic field and artificial intelligence for decades, both at the intellectual and experimental levels. Other activities focus on the emotional impact of emotional robots that work on the basic argument that such a thing is possible. Many closely related studies address the moral and ethical implications of artificial intelligence when it comes to independent robots, which can make decisions that conflict with human moral standards.

Another thriving line related to AI and emotions related to emotional integration modeling using robotic and virtual agent. For example, a new approach to emotional modeling is based on comparing computer model behavior to human behavior and using standard clinical diagnostic tools.

Human-computer communication (HCI) can be regarded as a subset of artificial intelligence. And it showed more interest in emotions. For example, basic issues related to the release of emotions by the user include psychological and linguistic analysis.

Computer Games and Real Reality

As video games become more complex and enticing, research in the field of AI game is gaining more popularity. The focus of the research is different but the focus of our discussion is to dig up player data and improve the character of non- players. A great incentive for researchers from this field is to learn what makes players enjoy or hate the game and as a result make the playing experience more enjoyable.

On the other hand, the recognition and enhancement of user feelings through player data mining (e.g., face recognition and keystroke patterns, chat messaging analysis) have few applications in the game development field.

For example, by recognizing in a timely and accurate manner the player's emotional state the system can react to it by changing the game environment (changing speed, color scheme, or even suggesting that the player take a break). It can participate in educational games by customizing the learning process. On the other hand, games that can generate player response, such as fear or excitement, are very immersive, and are thought to make it easier to flow, which is a great fun and immersive experience at work. Therefore, for the play process to be attractive and realistic it is important that the player encounters artificial characters that play emotionally and intelligently and respond well to the player's emotions.

Sentiments and Mining Analysis of Vision

Requests for emotional analysis and archeology were not included in this study. Opinion mines are concerned with the automatic tracking and classification of ideas expressed by humans. An idea in the smallest sense is understood as an examination or attitude toward something. Although mining analysis and theories are often used consistently in literature, ideas are not emotions: While emotions are motivated by emotions, opinions judgment is based on defining the purpose or automation of a topic that has nothing to do with emotion.

Based on emotional analysis and the exploration of ideas about people's attitudes and feelings about anything, they receive applications in many areas, for example business and social. A popular application for demographic analysis is automated review analysis, which is usually done in a large collection of updates that can come from any domain, for example movie reviews, product reviews (books, electronics, DVDs, etc.), restaurant product reviews and tourism. The purpose of the digging of ideas in this context is to classify positive or negative reviews into different levels of granularity.

Opinion mines are not limited to reviews. Computer science has also seen a growing interest in automated emotional analysis, for example, in the political arena. The purpose of these studies is not only to gain an understanding of human preferences, but also to gain an understanding of how these options are created and distributed through social media.

(4)

A significant amount of research is related to the automatic analysis of social media posts (usually, Twitter). The purpose here is to enable automatic detection of user posts in relation to feelings and ideas. This can be helpful in automatically recognizing the social media for emergency reports, violent language, and the spirit of certain user groups.

Methodology

The Methodology depicts the implementation of the model which tells about the human sentiments. The methodology is implemented by following points:-

1. Data is split into training and evaluation sets.

2. Data set consist of different human images which is processed and analyzed.

3. The model is trained using data set.

4. Use test data to evaluate the model performance.

5. The trained model is then used on data to generate the prediction score, log loss value and human sentiment.

The following flow chart represents the Image analysis done in the model and how the Sentiment is recognized.

Machine learning split the datasets into three sets:

Training Validation Test

To train the model training set is used. The validation set is used to help tune the hyper parameters of the model, which leads to better performance. The dataset that incorporates a wide variety of data to accurately judge the performance of the model is test set. Test sets are used to compare multiple models, including the same models at different stages of training. The following dataset is used in the model:-

The above data set is analyzed which classifies model in machine learning to categorize the images by using pre- trained tensor flow for processing of human sentiments.

(5)

Result and Discussion

The proposed methodology is tested with the different images which predicts the score, its log loss and its human sentiment. For e.g. we choose Image 3 for the test, Image 3 will be inputted in the model then it will be analyzed. By doing so the pre-trained tensor flow model will classify the image with the data and will predict the score for the image and its log loss. It will also describe the sentiment behind the image.

Image 3 for testing the model:

Result:-

The following table describes the result for different images, showing the sentiment of the image, its prediction score and its log loss:-

S No. Images Image Sentiment Prediction Score Per Class Log Loss

1 Image 1 Anger 0.7354348 0.30729343004679144

2 Image 2 Contempt 0.67162246 0.3980589190784853

3 Image 3 Disgust 0.7989082 0.2245092659305949

4 Image 4 Fear 0.71846837 0.3306335994100053

5 Image 5 Joy 0.72446704 0.32231901293088083

6 Image 6 Sad 0.78993654 0.23580266268283095

7 Image 7 Surprise 0.60229003 0.5070161651524836

Log Loss:-

0.3322332936045817

The log loss of data set is shown in the form of graph given below:-

(6)

In this graph log loss value for each image is given for the different human sentiments. Log loss or Binary-class entropy is the negative average of the log of corrected predicted probabilities for each data that is inputted. Formula of log loss by which all the values are calculated:

In this paper we took different images which were classified by the pre-trained tensor flow model with machine learning which in result give the prediction score, its log loss and human sentiment which is shown in the image.

This model can be used in various platforms and sectors for the prediction of human sentiment.

Acknowledgment

The Success of this work would have been uncertain without the help and guidance of our mentor and parents. We would like to express the deepest appreciation to my teacher Mr. Mohammad Shabaz, Department of Computer Science and Engineering, Lovely Professional University, Phagwara (Punjab), India. Who has the attitude and the substance of a genius, he continually and convincingly conveyed a spirit of adventure in regard to research.

References

[1] Keyur Patel, Dev Mehta, Chinmay Mistry, Rajesh Gupta, Sudeep Tanwar, Neeraj Kumar, Mamoun Alazab,

"Facial Sentiment Analysis Using AI Techniques: State-of-the-Art Taxonomies and Challenges", Access IEEE, vol. 8, pp. 90495-90519, 2020.

[2] Deep learning for robust feature generation in audiovisual emotion recognition Yelin Kim, Honglak Lee, and Emily Mower Provost University of Michigan Electrical Engineering and Computer Science, Ann Arbor, Michigan, US, 2013

[3] Khan, Reeshad & Sharif, Omar. (2017). A Literature Review on Emotion Recognition Using Various Methods. Samiullah, Muhammad, Load balancing in cloud computing (2018)

[4] S. Li and W. Deng, "Deep Facial Expression Recognition: A Survey," in IEEE Transactions on Affective Computing, doi: 10.1109/TAFFC.2020.2981446.

[5] B. Liu, Sentiment analysis and opinion mining. San Rafael, CA: Morgan and Claypool Publishers, 2012.

[6] Medhat, W., Hassan A, Korashy H. Sentiment analysis algorithms and applications.

0.7 0.6 0.5 0.4 0.3 0.2 0.1

0

Log Loss Graph

(7)

[7] Araque O, Corcuera-Platas I, Sanchez-Rada JF, Iglesias CA. Enhancing deep learning sentiment analysis with ensemble techniques in social applications. Expert Systems with Applications, 2017 Jul 1;77:236-46.

[8] Akhtar MS, Kumar A, Ekbal A, Bhattacharyya P. A hybrid deep learning architecture for sentiment analysis. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers 2016 (pp. 482-493).

[9] Rezaeinia SM, Rahmani R, Ghodsi A, Veisi H. Sentiment analysis based on improved pre-trained word embeddings. Expert Systems with Applications, 2019 Mar 1;117:139-47.

[10] Huang M, Qian Q, Zhu X. Encoding syntactic knowledge in neural networks for sentiment classification.

ACM Transactions on Information Systems (TOIS), 2017 Jun 9;35(3):26.

[11] Dang, Cach & Moreno García, María & De La Prieta, Fernando. (2020). Sentiment Analysis Based on Deep Learning: A Comparative Study. Electronics. 9. 483. 10.3390/electronics9030483.

[12] M. Shamin Hossain- Department of Software Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia, Ghulam Muhammad - Department of Software Engineering, College of Computer and Information Sciences, King Saud University, Riyadh, Saudi Arabia, “An Emotion Recognition System for Mobile Applications”, IEEE Special Section of emotion-aware mobile computing (2017).

[13] Wanliang Tan, Xinyu Wang, Xinyu Xu – Department of Computer Engineering, Stanford University, USA,

“Sentiment Analysis for Amazon Reviews”. International Conference on Human and AI interaction (2018) [14] D. Yangm Abeer Alsadoon, P.W.C. Prasad, School of Computing and Mathematics, Charles Sturt

University, Sydney, Australia, “An Emotion Recognition Model Based on Facial Recognition in Virtual Learning Environment”. International Conference on Smart Computing and Communications (2017) [15] Sreenidhi MJ, Surabi Sri Dhanya S, Sahithi R, Sharon Femi P. “Human Emotion Recognition System using

Deep Learning Techniques” (2020).