Originally published on the HC Unicamp website

The physicist Caetano Ternes Coimbra developed an algorithm capable of quantifying movements captured by a smartwatch sensor to assist in analyzing tremors and defining medication dosage therapy. The research is part of a project under the Hub of Artificial Intelligence Applied to Health and Well-being – Viva Bem and is the result of his master’s dissertation defended at the Gleb Wataghin Physics Institute (IFGW) at Unicamp.

Tests for using smartwatches to monitor health and well-being were conducted on patients treated at the Parkinson’s and Movement Disorders clinic of the Neurology Department at the Clinics Hospital (HC) of Unicamp. The research was supervised by Rickson Coelho Mesquita, with co-supervision by Gabriela Castellano. Laura Silveira Moriyama led the HC Unicamp medical team involved in the research.

“Our goal was to measure tremors using a commercially available smartwatch for everyday use in different applications. Based on the existing literature, we already knew that we could measure movement to some extent. However, the challenge was to measure it accurately using the watch’s sensor and all the technology involved, capable of detecting information such as arm movement,” explains Coimbra, who completed his undergraduate studies at the Federal University of Santa Catarina (UFSC), having studied particle physics, “a completely different field.”

Beyond data collection, Coimbra aimed to create a deterministic algorithm to understand under what conditions the tremor occurred. The goal is not to diagnose using a smartwatch, especially since it is a multifactorial disease. The primary contribution of the watch is to provide support for treatment adjustments. The patient takes the watch home, which continuously and automatically measures tremors, recording when they improved or worsened.

Multidisciplinary Research

The coordinator of the Parkinson’s and Movement Disorders service at HC and neurology professor at the Faculty of Medical Sciences (FCM) of the university, Laura Silveira Moriyama, led the medical team involved in the research, stating it represents a significant contribution to treating movement disorders.

“There are various types of neurological problems that affect how a person moves, speaks, swallows, or uses their hands. With the advancement of research, we increasingly see interdisciplinary collaboration from fields such as engineering, physics, and computing. In this project, the researcher collected objective measurements of tremors and patient movement difficulties. As doctors, we observe the tremor visually. He, as a physicist, examines its frequency and amplitude,” said Moriyama.

According to the neurologist, patients often are unsure if their condition has improved after taking medication. “We tried to find ways to measure this objectively. The sensor gives us more objective information.” Coimbra’s protocol involves two types of devices: electromyography (which measures muscle activation responsible for the tremor) and an accelerometer-equipped watch (which measures the acceleration of the body’s movement in that area).

“We used it on the wrist because Parkinson’s tremors significantly affect the arms, but it could be applied to other body parts such as legs, torso, and head. This type of sensor helps us track disease progression.” According to the doctor, the visual representation created by Coimbra also helps patients understand their medical condition better.

“The sensor provides reassurance for both the patient and the doctor. It’s not feasible to send a clinician to the patient’s home, but it’s like they’re taking a piece of our team with them.”

The Algorithm

Coimbra developed a classical algorithm using Fourier analysis to extract the physically relevant characteristics of the tremor (an oscillatory signal, moving back and forth) – its amplitude, frequency, and frequency variation. “The algorithm has the potential to be applied to other symptoms, such as movement slowness, in addition to tremors. Measuring tremors is more than diagnosing a pathology. Everyone has a physiological tremor because many factors can cause tremors, such as anxiety during a job interview or caffeine intake. Thus, measuring tremors is a broad concept and not a new idea,” said the researcher.

Report in Jornal da UNICAMP

In a small studio at Unicamp, a woman dances delicately. Her arms and legs flow with elegance, creating a harmony with the music and colorful images that come alive on the walls. With each gesture of the dancer, the environment transforms: when she raises her arms and shifts her shoulders, the images burst into a vibrant fusion of colors. When she reins in her movements, the figures calm, returning to a serene light-yellow hue.

In this performance, filmed at the University’s Interdisciplinary Sound Communication Center (NICS), the dancer demonstrates how BehCreative works—a digital cave developed by musicologist and postdoctoral researcher Elena Partesotti. The technology, whose name derives from the phrase “behave creatively,” is an extended digital musical instrument (EDMI) combining virtual and augmented reality, where users manipulate images and sounds through interactive tools.

Using a computer, a motion tracker, projectors, and speakers, BehCreative generates visual and auditory responses based on the user’s movements—each tied to different sounds and color patterns. “The idea is that your body becomes a musical instrument. There’s no physical object. You produce sounds and images with your movements—something aligned with the embodied cognition paradigm,” explains Partesotti, referring to the philosophical theory that human cognitive experiences are shaped by the body’s interaction with its environment.

A prof. Gabriela Castellano investiga como o cérebro se reorganiza. Fotografia: Antonio Scarpinetti, Jornal da UNICAMP

Originally created to explore how sound and movement combine in immersive settings, this EDMI aims to stimulate artistic and creative processes within music therapy. Studies have shown that musical interventions provide important tools for promoting patients’ physical and psychological well-being. These tools trigger the release of hormones such as oxytocin, dopamine, norepinephrine, and adrenaline—linked to emotions like love and pleasure—and melatonin, which regulates biological processes such as sleep and wakefulness. As a result, they can reduce anxiety and depression, alleviate pain, and improve quality of life for people with cognitive deficits, including Alzheimer’s disease.

More recent research has also highlighted the positive effects of music therapy on brain neuroplasticity—the central nervous system’s ability to adapt to new environmental stimuli—and on the rehabilitation of stroke survivors. For this reason, one of the music therapist’s goals is to evaluate BehCreative’s ability to rehabilitate motor function and cognitive capacity in these patients, a project conducted under the guidance of Professor Gabriela Castellano from the Gleb Wataghin Institute of Physics (IFGW).

According to Castellano, the musical instrument is still in prototype phase, but researchers hope that, one day, patients will be able to use it in their own homes as a complementary physiotherapy treatment.

“We know that debilitating diseases are becoming more common due to sedentary lifestyles and an aging population. So there are more and more people with neurological sequelae from various health conditions. And nowadays clinics and physiotherapists can’t keep up with the growing demand from these patients,” observes the professor.

Cognitive Rehabilitation

Castellano is one of the principal researchers at Unicamp’s Brazilian Institute of Neuroscience and Neurotechnology (BRAINN), which investigates the mechanisms behind epilepsy and strokes and the damage they cause. Before working with Partesotti, Castellano had already been developing extended-reality applications for stroke therapy in partnership with postdoctoral researcher Alexandre Brandão. “We’ve been evaluating what brain changes result from using these applications, aiming to understand how the brain reorganizes afterward. For that, we use neuroimaging techniques such as functional MRI and electroencephalography,” she explains.

However, those devices don’t use musical elements and, being focused on motor rehabilitation, require patients to make specific, predetermined movements—limiting user creativity and, consequently, potential cognitive response. That’s why the researchers now plan to apply the same neuroimaging techniques to BehCreative users, to see if engaging in a more creative activity leads to cognitive improvement.

Preliminary tests using functional MRI have already shown that healthy individuals exhibit altered connectivity in emotion-related brain regions after using BehCreative, affecting emotional regulation and boosting users’ creative expression. These findings are slated for publication in the Nordic Journal of Music Therapy, one of the world’s leading music therapy journals. But due to COVID-19 restrictions, the initial tests involved only five people, making it difficult to generalize the results. More recently, the study was repeated with 15 healthy users, though that data is still under analysis.

Currently, Partesotti and Castellano are beginning experiments with BehCreative in stroke patients and recruiting new candidates for the research. Over ten days of testing at Unicamp, participants will use the equipment for up to ten minutes, during which they’ll undergo an EEG to record any brain-function changes and receive care from a physiotherapist. “The idea is that people decide how to express themselves using this equipment. We call this creative empowerment—providing tools so they can incorporate what they love into their rehabilitation, aiming for both motor and cognitive improvement,” says the researcher.

The tenth edition of the BRAINN Congress was celebrated in 2024, held from April 3rd to 5th, 2024, in the auditoriums of the Faculty of Medical Sciences (FCM) at the State University of Campinas (UNICAMP). The event was organized by the Brazilian Research Institute on Neuroscience and Neurotechnology (BRAINN), a Research, Innovation, and Dissemination Center (RICD) at FAPESP.

This year, more than 180 researchers, students, and attendees participated in the event, which featured 123 submitted works. These works will be published in the Journal of Epilepsy and Clinical Neurophysiology.

International scientists lectured throughout the 3-day event. Dr. Mariana Bento, an assistant professor at the Schulich School of Engineering, University of Calgary, discussed the use of machine learning to improve brain imaging. Dr. Matthias Van Osch, a professor of experimental cerebrovascular imaging at Leiden University in the Netherlands, presented non-invasive strategies for obtaining brain images. From Brazil, Dr. Daniel Papoti, an adjunct professor of Biomedical Engineering at the Federal University of ABC, lectured on radiofrequency coil technologies for brain imaging.

Also participating in the event were BRAINN researchers Dr. Fernando Cendes, Dr. Gabriela Castellano, Dr. Iscia Lopes-Cendes, and Dr. Lília Freire Li, with lectures on various topics in neuroscience. The presentations explored topics ranging from the panorama of genetic research in epilepsy over the past 30 years (Iscia) to brain changes post-rehabilitation training (Gabriela) and the challenges and controversies in classroom learning (Lília).

Awards

Each edition, the Congress awards students and researchers with the Best Paper Awards, in the categories “Oral Presentations” and “Poster Presentations”. Check out the list of winners of the 10th edition below:

Oral Session Winners

Undergraduate:

• Mariana L. Han (Faculdade de Medicina da USP) – Insights into Post-COVID Anosmia: unraveling neural alterations through 7T MRI and DTI analysis of olfactory pathways
• Vitor A. Oya (UNICAMP) – Seed-based functional connectivity changes in elderly subjects playing serious games

Master’s:

• Rodrigo M. C. Carvalho (Federal University of ABC) – A Novel Method to Investigate Neurodevelopment Using Ising Temperature and Graph Neural Networks

Doctorate:

• Rafael Batista João (UNICAMP) – Comparisons between traditional and brief, and ultra-brief anxiety and depression inventories in people with epilepsy

Post-Doctorate:

• Gabriel Chaves de Melo (UNICAMP) – Timing Investigation of EEG Functional Connectivity Changes During a Motor Imagery Task

Poster Session Winners

Undergraduate:

• Lucas F. Teodosio (Unicamp) – Analysis of the Neural Correlates of Problem Solving in Physics Measured with Functional Near-Infrared
• Thierry K. Alves Silva Souza (Unicamp) – Hypothalamic Changes and Post-COVID Symptoms

Master’s:

• Caetano Ternes Coimbra (UNICAMP) – Exploring the relationship between physiological and inertial characteristics in induced tremor events

Doctorate:

• Ricardo Brioschi (Unicamp) – Persistent seizures modify supra and infratentorial Gray Matter in Idiopathic Generalized Epilepsy

Post-Doctorate:

• Kauê T. N. Duarte (University of Calgary) – Exploratory Prenatal Alcohol Exposure classification in infants using brain MR: 3D Convolutional Neural Networks assisted by Data Augmentation

April 28th, 2016

by Erik Nardini Medina *

Picture this: a patient with a paralyzed arm (as a result of a stroke) has his image captured by a camera and transmitted to a computer. Then, a software analyzes the patient’s body and “replaces” the member without movement with a healthy arm, virtually created by a technology known as augmented reality. Incredible, right?

It may sound like science fiction, but research on techniques like this one is already happening – including in Brazil. The BRAINN – Brazilian Institute of Neuroscience and Neurotechnology – is one of the reference centers in this area. Institute researchers are studying the use of the latest technology to assist in the recovery of those who have suffered a stroke.

HOW TO TREAT A STROKE?

Despite those scary numbers, there is reason to cheer. Treatments for those who have suffered strokes have evolved considerably in recent years.

Apart from the drugs field – whose administration usually occurs only in the period when the patient is in hospital -, physiotherapy sessions are prescribed, but usually for an indefinite amount of time. This is when patients and physiotherapists can be given support by technologies associated with Virtual Reality (VR) and Augmented Reality (AR).

RESEARCH MADE BY BRAINN

Pioneering studies on these two virtual applications have been conducted under the supervision of BRAINN´s Gabriela Castellano, associate professor of the Institute of Physics “Gleb Wataghin” at the State University of Campinas (Unicamp), and BRAINN´s scientific diffusion coordinator.

The projects, still in their early stages, adopt strategies to encourage motor coordination in the patients, transforming the exercises into more fun and effective activities.

DIFFERENCES BETWEEN VIRTUAL AND AUGMENTED REALITY

We live in the age of Virtual Reality. The technology will be more and more used in educational and medical scenarios.

In the studies conducted by BRAINN, two assistive technologies gain the spotlight: Virtual Reality and Augmented Reality. Do you know the differences between them?

Virtual Reality (VR): this is the one in which the subject is ‘inserted’ into a three-dimensional interface and participates in certain scenarios – in other words, his/her movements are played in a fictional environment.

The Kinetic motion sensor device – the same one used for Xbox games – is employed in the VR activities. “In this case, the person handles objects in the video through gestures. Sometimes, the patient controls an avatar on the screen, which reproduces in the virtual space their movements in real space”, explains Gabriela Castellano.

Augmented Reality (AR): 3D elements can be “inserted in the real world.” To illustrate this concept, just look at what financial institutions are doing nowadays. Banks use AR, for example, to allow users to search for agencies in the city. When someone points the camera phone to a certain position, the application adds visual elements to the screen that mingle with the real images captured by the phone. It is the fictitious presenting itself in the reality.

“For the AR, the focus lies on patients with limited movement in their arms”, said Castellano. “A camera captures the image of the patient and transmits it to a computer. The software then analyzes the patient’s body and ‘replaces’ the numb member for an arm through augmented reality”, explained the professor.

NEUROPLASTICITY

VR has been used for some time now to help wounded soldiers reclaim movements and get back to life.

The projects aim to assess the possible effects of neuroplasticity, which is the ability of the nervous system to adapt at the structural and functional levels when subjected to new experiences. “We want to understand the evolution of the nervous system after using these applications in motor rehabilitation,” says Castellano.

“With the advancement of research techniques, we can compare the results of a rehabilitation supported by AR and VR with the results obtained in ‘conventional’ rehabilitation”, concludes the researcher.

With the help of science and high technology, health teams are already able to handle the aftereffects of strokes with utmost care and efficiency. This partnership will bear even greater fruit in the coming years, when techniques such as Virtual Reality and Augmented Reality are perfected and become routine in medical care. In this way, stroke sequelae may stop debilitating thousands of people every year, both in Brazil and throughout the world.

* Journalist graduated from the Pontifical Catholic University of Campinas (PUC-Campinas), student of Science Journalism specialization at the Advanced Studies in Journalism Laboratory (Labjor / Unicamp). Has a MídiaCiência / FAPESP scholarship.