Dementia smart glasses turn “moonshot” into reality

For the best part of a decade, Animorph Co-operative, a London-based software development organisation, had harboured a dream of improving memory for people with dementia through a device that builds cross-sensory associations. 

This experimental idea, named CrossSense, was a labour of love pursued in the odd snatched hour around the bread-and-butter work of other projects. “Whenever we had time, we would do some research, […] discuss it, and allocate a bit of time – as much as we could afford – to prototype things in-house,” says Szczepan Orlins, software architect and director at Animorph. Other work consumed most of their time: Animorph brings together diverse disciplines – software design, programming, engineering, neuroscience, animation, conceptual art, even origami design – in pursuit of software to “enhance human potential” and “address unmet needs,” such as relapse prevention for psychosis. 

Turning a “moonshot” into reality 

This all changed in June 2023, when the team was named a semi-finalist on the Longitude Prize on Dementia. “Over the years, we had prototyped a few versions [of CrossSense], both on mobile phone and on glasses, but nowhere near the substance we were able to commit thanks to becoming semi-finalists,” says Szczepan. “When this stage unfolded, we turbocharged it and fully committed to it – now we have pretty much shut down all other operations, in order to not distract us from what’s important.” 

Last month, Animorph made it to the Finalist phase of the Longitude Prize on Dementia – an “exhilarating” turn of events for the team. “Our idea, while definitely powerful, felt like a little bit of a moonshot,” Szczepan says. Research and testing have shown “great promise […] but still we felt we are just a small team in North London that has been hoping to build this for the best part of a decade and now it’s happening, it’s exhilarating.”

“Addressing a burning need”

CrossSense combines wearable tech – lightweight smart glasses – and an augmented reality app to help people living with dementia to remember loved ones, objects, and daily tasks. The glasses connect with an energy efficient box – an edge server that stores data and processes requests, doing “the heavy lifting” for the system. This box is encrypted and functions offline “in a secure, independent network,” to protect people’s privacy – no data is sent to the Cloud. Using an AI agent named ‘Wispy’, the user can assign text and audio tags to objects in their physical environment. These objects could be personal such as photographs and mementos, or practical (toasters, kettles etc.). 

With practical objects, Wispy helps the user carry out domestic tasks safely. For example, “if you look at a toaster, it would tell you what you can do with it but also to watch out when you take out the toast,” Szczepan says. “So it’s quite common sense but also, because of the lack of any tool of this kind, it addresses a burning need.” The system also provides reminders and guidance for scheduled tasks such as taking medication.

The idea is that users wear the glasses for up to two hours (time limited by battery life) at times when they struggle more with memory. “We want to be helpful to people when they need it and have that autonomy and independence, but also to not be dependent on it entirely,” says Szczepan.

Drawing on world class research 

CrossSense also aims to be a “cognitive aid” to “slow down the impact of dementia.” In this, it applies cutting-edge research on synaesthesia – a perceptual phenomenon where the stimulation of one sense leads to involuntary experiences in another, usually in a way that is highly consistent over time.  A synaesthete might always experience a certain musical note as a specific colour, for example.      

CrossSense aims to help people acquire associations that mimic synaesthesia, by exposing them to personalised associations over time. For example, people could use the system to attach a colourful aura to a loved one whom they associate with that colour. Repeatedly seeing this colour when you look at the person could “help you build another association that will reinforce your memory, which can help with one of the pathways towards retrieving the information about who they are,” Szczepan explains.  

“You’re effectively creating new routes in the brain, and there’s evidence […] that this improves memory and can improve cognition more broadly, including IQ or pattern recognition,” he says. By creating a “training method for acquiring those associations […] and building cognitive reserve,” the team hopes that CrossSense will help “ward off” some dementia symptoms.  

Lessons from cutting-edge research and co-design

Over the past 17 months, support from the Longitude Prize on Dementia has enabled Animorph to develop CrossSense – making it “really robust” – alongside world class researchers, such as Professor Julia Simner at the University of Sussex’s MULTISENSE Lab. A key component has been co-design – working directly with people with lived experience of dementia to gather feedback and “find the intersections” between research and co-design. The team tested CrossSense with people living with dementia in their homes, resulting in “very encouraging” feedback on user experience, how people interact with it, and the features they want. 

“It’s become much less about our idea and much more about what people and science propose,” Szczepan says. “We were given a lot of feature requests and they formed a roadmap that we are now reviewing.” The Prize’s Discovery phase also revealed the importance of making CrossSense less structured around a “linear process of interacting with objects” and more supportive of how people interact with their environments in non-linear, unpredictable ways. 

“The user is in charge”

Users said they liked how Wispy allows them to add labels and messages – “interactive contextual post-it notes” – in a “very natural way,” Szczepan says. “The interesting thing about the accessibility of this device is that people […] put it on, […] start speaking to the agent, they see things, things pop up – it’s actually very natural, and they don’t need to learn any interface […] or how to tap or swipe something.” This intuitive interface gels well with an ageing population, including those often sidelined or “belittled” by mainstream technology. “We are really committed to supporting people so they’re in charge,” Szczepan says.   

The solution is also designed to be flexible and avoid the frustrations triggered by some mainstream systems: “if I want to stop the interaction then I just say ‘stop’ or ‘let’s move on’ […] – or you just walk away.”

Finetuning features 

In the Finalist phase, testing and development will focus on user acceptability, how to improve functions and interactivity, and improvements to the hardware, including upgrading the processor box and making the glasses lighter. The plan is to develop additional types of labelling beyond text and audio, such as coloured labels, shapes, and ribbons – “anything you would like to engage with that will help you build another association that will reinforce your memory.” 

“Expanding the testing tenfold” 

In early 2025,  the team will expand testing to include 15 to 30 people living with dementia and 15 to 30 carers. Each user will complete a baseline questionnaire, trial the glasses for a day, and participate in a a qualitative semi-structured interview. Animorph will combine these findings with data collected from the app. The plan is to test the device with a broad range of people, selected via Animorph’s partnerships with institutions at the University of Sussex, the NIHR HealthTech Research Centres at King’s College London and Cambridge, the South London and Maudsley NHS Foundation Trust (SLaM), and Leading Lives, a care support network in Suffolk. 

Guidance from the Longitude Prize on Dementia will be key: “We are definitely interested in advice on how we run the co-design sessions and on user-centred design,” says Szczepan. “With this specific population, it’s sensitive, so we want to keep on learning on this front.”

“How do we build a company that is investable?” 

 Another major focus for the Finalist phase is commercialisation. As a co-operative, this is a complex area for Animorph: any investor would need to align with their strong ethical frameworks and social value policies. Certain issues would be deal breakers. For example, “if someone in the future tries to override the commitment to privacy and wants to sell the data to advertisers, then that’s problematic,” Szczepan explains. 

They will also work to keep costs down, to make CrossSense as accessible as possible. Ideas include making the system rentable or allowing instalment payments. And, “there will be no hidden costs that you might have with some Cloud systems.” Their plans also include exploring markets in Europe, the United States, and Japan – a country “ahead of the curve when it comes to AI and population ageing challenges.” 

From a kernel of an idea in a North London software lab to targeting global markets, the CrossSense journey so far is testament to the power of pursuing dreams and turning “moonshots” into reality.