Beyond Emotion supports caregiving with AI

©Beyond Emotion

Working and looking after relatives in need of care at the same time. This scenario is becoming a reality for more and more people in the face of the growing care crisis and continuously increasing costs for nursing home places. However, this is associated with the lasting concern about how the person one is caring for is doing. Especially when illnesses such as dementia, arthrosis or osteoporosis are added to the mix, many carers are in a permanent stress situation if they do not know how their relatives are doing.

The Beyond Emotion team wants to tackle this problem. The startup has developed a software that recognises and evaluates facial expressions. This should then enable relatives to be notified when the person in need of care is in pain, anxious or confused, in order to provide targeted help.

From HAW to EXIST

It all started in 2010 with several other projects. First, Dr Arne Bernin worked with his colleague Larissa on art projects in which objects were controlled by the facial expressions of their viewers. Then came a few smaller projects and finally the precursor to the Beyond Emotion AI. The EmotionBike. This was a research project at HAW Hamburg, on which the later founders Arne and Sobin Ghose worked. The idea of the project was to provoke emotions, record them with a camera and develop a model for different emotions with matching facial expressions from the results. To do this, participants had to ride a simulated track on an exercise bike and react to what they saw on a screen.

The results of the EmotionBike and Arne's dissertation were then used in the following years to develop Beyond Emotion AI. This can recognise faces via cameras, transfer what it sees into a model and derive the emotions from it.

Currently, different emotional states are recognised. This does not correspond to any particular psychological model. In most cases, these classify into significantly fewer or significantly more emotions. Instead of using these as a guide, the Beyond Emotion team looked at the results of their own research with the EmotionBike and derived the recognisable states of mind from this.

What exactly the AI was to be used for was still unclear at the beginning. The first plan resulted from a request from an event organiser. If a recording of a human being can be automatically analysed for emotions, would it also be possible to track the mood at a concert on a live basis? "With the necessary computing power and enough cameras, no problem", says Arne. But before more could happen in this direction, the year 2020 came and with it the pandemic. So concerts were cancelled and the project fell through.

©Unsplash

Then the search for another application for the AI continued. The challenge was to find a moral application that would have a positive impact. The founding trio ruled out using their emotion-recognition AI as known from China. "We have already written in our founding rules that ethic use is important and have also rejected offers for that reason", Arne explains. Although the trio naturally wants to earn money with their startup hotspot, they also want to do something good.

The suggestion to make everyday care easier with facial recognition then came to the team all on its own. With a trained AI in their pocket and the firm intention to use it, the Beyond Emotion team visited networking events and told others about it. At some point, care advisors approached the team and told them that they could well imagine that it would be a great relief in caregiving to know what emotion the person being cared for was currently experiencing.

With this impulse, Beyond Emotion then contacted family members of dementia patients who were caring for them and interviewed them. The result: the carers were concerned about their loved ones all the time.

"Is the father/mother already awake? Do I have to go over before or after work? Such questions occupy affected persons all the time. This then sometimes leads to people quitting their jobs because they have to concentrate more on caring. They simply don't know when they are needed, when the person they are caring for might have fallen, be scared or confused. So they want to be around all the time."

Hanne Butting, Co-Founder

This gave the team the idea to build a mood notification system, which informs when something is happening. So that the carers know what's going on and can then decide whether they should call or go over. "We are very much in line with HAW's motto: applied science. Applying research to help people", tells Arne.

Knowing this, the team then applied for the EXIST startup funding and got it in September of the same year. That was when things really took off. Since then, Beyond Emotion has brought their idea to market and is now ready for rollout.

A digital photo frame for grandma

In use, the emotion recognition AI is quite discreet. The software runs on a tablet that hangs in a frame on the wall. Like a digital picture frame. Pictures and videos are shown on it, which the family can send from anywhere in the world. Behind the latest holiday pictures of the grandchildren, the real task of the device runs unobtrusively.

We need your consent to load the YouTube Video service

We use a third party service to embed video content that may collect data about your activity. Please review the details and accept the service to watch this video.

In this way, the tablet directly fulfils a double function. On the one hand, observation and, on the other, it is a digital connection for older family members. Grandpa, who may no longer have the desire to learn how to use a smartphone, can still see what the family is doing. If desired, video calls can also be made via the device.

Via the camera, the software searches for faces, recognises them and interprets the facial expressions. If negative emotions such as fear, confusion or bordem are detected, the caregivers receive a notification via app. Then they can call or go by and check on their loved ones.

However, there is no need to fear for one's own data in the process. Beyond Emotion processes the facial recognition locally on the clients' devices and does not send any recordings to servers. The tablets are also selected so that they do not pass on any data to the manufacturers.

Anyone who wants a tablet like this for their loved ones has to buy the device and also take out a subscription for the software. It is not yet clear how much this will cost in the end. When it is released, the product will first go to private households. It is also planned to equip care institutions, but only in the future.

Learn more about Beyond Emotion