Computers are becoming experts on human nature

Computers are becoming experts on human nature

Assoc. Prof. Ali Albert Salah from Boğaziçi University Computer Engineering department tells his work on computers’ analyzing social and emotional cues from humans.

Assoc. Prof. Ali Albert Salah from Boğaziçi University Computer Engineering department is a scientist, known for his work linking technology and human behavior.  Salah's work focuses on improving the comprehension level of the computers regarding social and emotional signals from humans. His work ranges from estimating the emotional state of children by analyzing their tone of voice and word usage; a robot app to help the elderly with exercising; to AI that can analyze human voice and facial expressions so as to determine if their smile or laughter is genuine or not.

Ranked first in ChaLearn

A team from Boğaziçi and Namık Kemal universities participated as the sole participant from Turkey in a workshop and competition "ChaLearn Workshop on Explainable Computer Vision Multimedia and Job Candidate Screening Coopetition" in Hawaii this year in July where they received an award. They attended with a project that could analyze one-minute long-videos to determine if the person was going to be invited to a job interview or not, and then could explain the reasoning behind its decision to the user. Salah and his teammates attended the same event last year with a similar project, one that could make estimates about the person's personality by analyzing one-minute-long videos, which also earned them the first place. Salah remarked that people have been working on AI to make computers more intelligent for a long time now and these efforts have focused on making sure robots can work with humans in social settings. He explained that it is crucial for social robots and similar systems to understand human emotions.

He added, "Let's say that this one robot is helping us around the house. It has to be able to understand our current emotional state, social signals, what we're saying and how we are saying it to make sure it can tailor its behavior towards our needs. One of our projects in this context was a robot that helped the elderly with exercising. Such a robot has to be able to read the person's facial expressions to see if they are in pain, and if so, end the exercise immediately. We're still working on analyzing facial expressions in Boğaziçi University with Prof. Levent Akın and Ph.D. student Binnur Görer. The robot only has to evaluate the angles at which a joint is situated or the speed of the limbs so as to see if the person is doing the exercise correctly or not. But understanding social clues such as fatigue, weariness, etc. is a lot harder.  Such clues and signals differ from one person to another significantly. The first step to this project is achieved through engineering: we want to build new systems that can help people in a better way. The second step, however, is through cognitive science and psychology. What we're trying to do is to understand human behavior. The systems we developed is helping psychologists analyze long records of human behavior automatically and fast. They can also offer clues and predictions on how some processes in the human brain is taking place if the modeling has matured enough."

Analyzing the laughter of the old and the young

Salah reports that often times, the analysis of computers is more detailed than that of humans. One of the projects that he conducted on human laughter is exhibited in Amsterdam NEMO Science Museum. In this study, computers analyzed the laughing patterns of people ranging in ages from 7 to 76 to figure out how human laughter changes with time. This new system is better at estimating a person's age from recordings than humans.

Salah continues to explain: "'Social laughter' is laughing because the occasion requires it. There is no happiness or any similar emotion behind it. And then there are times when we laugh because we feel like it. We gathered data from about 500 people to distinguish the two laughter patterns. Then we analyzed this data set using computers. The analyses of computers can be more detailed than those of humans. For example, we study the movement of the mouth corners. How much do they move upwards? How many centimeters? How fast? What is its acceleration? Such a detailed analysis cannot be done by a human being. Only if you receive special education on such a matter, then you can distinguish the genuine and fake facial expressions. I conducted this research with the help of a Boğaziçi University Computer Engineering graduate, Dr. Hamdi Dibeklioğlu, who is going to work as an associate professor at Bilkent University this semester.

Computers will be able to recognize children by their voice and face

Salah also is working on a BAP (Scientific Research Project) on multimedia for children: "Children are using the internet and computer systems often in our current time and day. But such systems do not have the ability to understand that the person using it is a child, thus cannot change its behavior to accommodate children. For example, think of a child using any platform on the Internet; the ads might contain adult content, violence or any other content that might not be suitable for children. Many systems need to be tailored to accommodate children. This is not an easy task. There are some basic modules that we need to do. One of them is to estimate the age of the user from their face. A computer or a robot should be able to determine a person's age by looking through the camera. Or by analyzing the voice of the user. The module that allows the computers to predict the user's emotion is also important. We're conducting a research with Assoc. Prof. Sibel Halfon from Bilgi University Psychology department on emotional analysis on the words children use. Children's faces are slightly different from those of adults. As they haven't completed their development yet, their facial morphology is different, therefore their facial expressions are also different. As such, their voices also have distinct features. We’ve conducted another research in collaboration from a research team from St. Petersburg on Russian children. We've gathered a data set to estimate children's ages, sex, and emotional state, then developed machine learning approaches. We're working on this project with our alum Assoc. Prof. Heysem Kaya, who is currently working at Namık Kemal University."

Every data is treasured

Explaining that data is of crucial importance to make sure AI systems are working properly and effectively, Salah pointed out that nowadays, corporations work with the users to gather and store data, and by using the machine learning systems, users are actually donating their information to such companies: "Companies with considerable databases are using this data to train their systems. For instance, to develop a functioning face recognition system, we download and save the faces of actors from movies. This allows us to have millions of images. Today, we have artificial deep neural networks that give rise to incredibly powerful machine learning systems, but what we need is data to feed such systems. Corporations that store data are developing better systems every day, which attracts more and more users, which, in turn, feeds those companies more data. Just like in the board game Monopoly, which is actually a satire of the capitalistic world, data-rich companies get even richer whereas data-poor ones lose the competition. Such companies share their own artificial intelligence research. As they're already losing out, the methodological development in the area will help them more than it helps any other company.

In fact, the notion "Citizen Science", a collection and analysis of data relating to the natural world by members of the general public, serves such purposes. For example, if you're using an online translating service and do not like the result you're getting, you can edit or change it. Those edits are invaluable for the company that's providing the translation service. Millions of people are correcting things every day. Therefore, there emerges a priceless database that cannot have been created by a handful of software developers. This process allows the system to develop as swift as the wind. If you have noticed, online translating services have recently been undoubtedly better thanks to the development of deep learning systems. However, we have yet to see such a development in social systems. To be able to create this cycle, we need to first build robots that will sit and chat with us during dinner."