Report in Jornal da UNICAMP

In a small studio at Unicamp, a woman dances delicately. Her arms and legs flow with elegance, creating a harmony with the music and colorful images that come alive on the walls. With each gesture of the dancer, the environment transforms: when she raises her arms and shifts her shoulders, the images burst into a vibrant fusion of colors. When she reins in her movements, the figures calm, returning to a serene light-yellow hue.

In this performance, filmed at the University’s Interdisciplinary Sound Communication Center (NICS), the dancer demonstrates how BehCreative works—a digital cave developed by musicologist and postdoctoral researcher Elena Partesotti. The technology, whose name derives from the phrase “behave creatively,” is an extended digital musical instrument (EDMI) combining virtual and augmented reality, where users manipulate images and sounds through interactive tools.

Using a computer, a motion tracker, projectors, and speakers, BehCreative generates visual and auditory responses based on the user’s movements—each tied to different sounds and color patterns. “The idea is that your body becomes a musical instrument. There’s no physical object. You produce sounds and images with your movements—something aligned with the embodied cognition paradigm,” explains Partesotti, referring to the philosophical theory that human cognitive experiences are shaped by the body’s interaction with its environment.

A prof. Gabriela Castellano investiga como o cérebro se reorganiza. Fotografia: Antonio Scarpinetti, Jornal da UNICAMP

Originally created to explore how sound and movement combine in immersive settings, this EDMI aims to stimulate artistic and creative processes within music therapy. Studies have shown that musical interventions provide important tools for promoting patients’ physical and psychological well-being. These tools trigger the release of hormones such as oxytocin, dopamine, norepinephrine, and adrenaline—linked to emotions like love and pleasure—and melatonin, which regulates biological processes such as sleep and wakefulness. As a result, they can reduce anxiety and depression, alleviate pain, and improve quality of life for people with cognitive deficits, including Alzheimer’s disease.

More recent research has also highlighted the positive effects of music therapy on brain neuroplasticity—the central nervous system’s ability to adapt to new environmental stimuli—and on the rehabilitation of stroke survivors. For this reason, one of the music therapist’s goals is to evaluate BehCreative’s ability to rehabilitate motor function and cognitive capacity in these patients, a project conducted under the guidance of Professor Gabriela Castellano from the Gleb Wataghin Institute of Physics (IFGW).

According to Castellano, the musical instrument is still in prototype phase, but researchers hope that, one day, patients will be able to use it in their own homes as a complementary physiotherapy treatment.

“We know that debilitating diseases are becoming more common due to sedentary lifestyles and an aging population. So there are more and more people with neurological sequelae from various health conditions. And nowadays clinics and physiotherapists can’t keep up with the growing demand from these patients,” observes the professor.

Cognitive Rehabilitation

Castellano is one of the principal researchers at Unicamp’s Brazilian Institute of Neuroscience and Neurotechnology (BRAINN), which investigates the mechanisms behind epilepsy and strokes and the damage they cause. Before working with Partesotti, Castellano had already been developing extended-reality applications for stroke therapy in partnership with postdoctoral researcher Alexandre Brandão. “We’ve been evaluating what brain changes result from using these applications, aiming to understand how the brain reorganizes afterward. For that, we use neuroimaging techniques such as functional MRI and electroencephalography,” she explains.

However, those devices don’t use musical elements and, being focused on motor rehabilitation, require patients to make specific, predetermined movements—limiting user creativity and, consequently, potential cognitive response. That’s why the researchers now plan to apply the same neuroimaging techniques to BehCreative users, to see if engaging in a more creative activity leads to cognitive improvement.

Preliminary tests using functional MRI have already shown that healthy individuals exhibit altered connectivity in emotion-related brain regions after using BehCreative, affecting emotional regulation and boosting users’ creative expression. These findings are slated for publication in the Nordic Journal of Music Therapy, one of the world’s leading music therapy journals. But due to COVID-19 restrictions, the initial tests involved only five people, making it difficult to generalize the results. More recently, the study was repeated with 15 healthy users, though that data is still under analysis.

Currently, Partesotti and Castellano are beginning experiments with BehCreative in stroke patients and recruiting new candidates for the research. Over ten days of testing at Unicamp, participants will use the equipment for up to ten minutes, during which they’ll undergo an EEG to record any brain-function changes and receive care from a physiotherapist. “The idea is that people decide how to express themselves using this equipment. We call this creative empowerment—providing tools so they can incorporate what they love into their rehabilitation, aiming for both motor and cognitive improvement,” says the researcher.

April 28th, 2016

by Erik Nardini Medina *

Picture this: a patient with a paralyzed arm (as a result of a stroke) has his image captured by a camera and transmitted to a computer. Then, a software analyzes the patient’s body and “replaces” the member without movement with a healthy arm, virtually created by a technology known as augmented reality. Incredible, right?

It may sound like science fiction, but research on techniques like this one is already happening – including in Brazil. The BRAINN – Brazilian Institute of Neuroscience and Neurotechnology – is one of the reference centers in this area. Institute researchers are studying the use of the latest technology to assist in the recovery of those who have suffered a stroke.

HOW TO TREAT A STROKE?

Despite those scary numbers, there is reason to cheer. Treatments for those who have suffered strokes have evolved considerably in recent years.

Apart from the drugs field – whose administration usually occurs only in the period when the patient is in hospital -, physiotherapy sessions are prescribed, but usually for an indefinite amount of time. This is when patients and physiotherapists can be given support by technologies associated with Virtual Reality (VR) and Augmented Reality (AR).

RESEARCH MADE BY BRAINN

Pioneering studies on these two virtual applications have been conducted under the supervision of BRAINN´s Gabriela Castellano, associate professor of the Institute of Physics “Gleb Wataghin” at the State University of Campinas (Unicamp), and BRAINN´s scientific diffusion coordinator.

The projects, still in their early stages, adopt strategies to encourage motor coordination in the patients, transforming the exercises into more fun and effective activities.

DIFFERENCES BETWEEN VIRTUAL AND AUGMENTED REALITY

We live in the age of Virtual Reality. The technology will be more and more used in educational and medical scenarios.

In the studies conducted by BRAINN, two assistive technologies gain the spotlight: Virtual Reality and Augmented Reality. Do you know the differences between them?

Virtual Reality (VR): this is the one in which the subject is ‘inserted’ into a three-dimensional interface and participates in certain scenarios – in other words, his/her movements are played in a fictional environment.

The Kinetic motion sensor device – the same one used for Xbox games – is employed in the VR activities. “In this case, the person handles objects in the video through gestures. Sometimes, the patient controls an avatar on the screen, which reproduces in the virtual space their movements in real space”, explains Gabriela Castellano.

Augmented Reality (AR): 3D elements can be “inserted in the real world.” To illustrate this concept, just look at what financial institutions are doing nowadays. Banks use AR, for example, to allow users to search for agencies in the city. When someone points the camera phone to a certain position, the application adds visual elements to the screen that mingle with the real images captured by the phone. It is the fictitious presenting itself in the reality.

“For the AR, the focus lies on patients with limited movement in their arms”, said Castellano. “A camera captures the image of the patient and transmits it to a computer. The software then analyzes the patient’s body and ‘replaces’ the numb member for an arm through augmented reality”, explained the professor.

NEUROPLASTICITY

VR has been used for some time now to help wounded soldiers reclaim movements and get back to life.

The projects aim to assess the possible effects of neuroplasticity, which is the ability of the nervous system to adapt at the structural and functional levels when subjected to new experiences. “We want to understand the evolution of the nervous system after using these applications in motor rehabilitation,” says Castellano.

“With the advancement of research techniques, we can compare the results of a rehabilitation supported by AR and VR with the results obtained in ‘conventional’ rehabilitation”, concludes the researcher.

With the help of science and high technology, health teams are already able to handle the aftereffects of strokes with utmost care and efficiency. This partnership will bear even greater fruit in the coming years, when techniques such as Virtual Reality and Augmented Reality are perfected and become routine in medical care. In this way, stroke sequelae may stop debilitating thousands of people every year, both in Brazil and throughout the world.

* Journalist graduated from the Pontifical Catholic University of Campinas (PUC-Campinas), student of Science Journalism specialization at the Advanced Studies in Journalism Laboratory (Labjor / Unicamp). Has a MídiaCiência / FAPESP scholarship.