Intelligent Supportive Environments for Older Adults (COACH project)
Keywords: Cognitive device, cognitive orthosis, smart homes, assisted cognition, context-aware design, ADL prompting, ADL guidance.
Overview of Research
Dementia reduces a person's ability to perform activities of daily living (ADL) because of related difficulties in remembering the proper sequence of events that must occur and how to use the required tools. The current solution is to have a caregiver supervise activities and continually provide prompts as to the next step in the activity. Family caregivers can find assisting their loved ones to be particularly upsetting and embarrassing as it necessitates invasion of privacy and often role reversal. Research suggests that dependence on a caregiver might be improved using a supportive environment, or cognitive orthosis, that provides needed reminders and monitors progress.
The COACH (Cognitive Orthosis for Assisting with aCtivites in the Home) is a prototype of an intelligent supportive environment being developed to assist people with dementia complete ADLs with less dependence on a caregiver, representing one of the first clinically tested supportive devices to use artificial intelligence techniques. The device operates with a personal (desktop) computer and a single video camera to unobtrusively track a user during an ADL and provides pre-recorded (visual or video) prompts when necessary. To date, three successive prototypes of the COACH system have completed clinical trials based around the ADL of handwashing with subjects who had moderate-to-severe dementia. In the first two cases, these trials showed that the number of handwashing steps subjects were able to complete without assistance from the caregiver increased noticeably when the device was present. Statistically significant individual changes ranged from approximately 10 to 45 percent. The third trial looked to explore the capabilities of the COACH in a real-world setting and to identify any changes required to further advance the system in future settings within the homes of older adults with dementia. The COACH was installed in a long-term deployment at the Toronto Memory Program – a multidisciplinary, community based, medical facility specializing in the diagnosis and treatment of Alzheimer’s disease and related disorders. This trial was unique to the first two in that the system was not individually calibrated for each specific user, revealing weaknesses in the generalizability of the system to multiple unknown users.
System Overview
COACH uses artificial intelligence (AI) algorithms and a single video camera to monitor progress, determine context, and provide pre-recorded prompts when necessary. For example, if the person forgets to turn off the water, the device prompts him/her to do so. The device has the capability of adjusting its parameters and cueing strategies to meet the changing needs and preferences of each user.
The only hardware visible to the user can be as little as a single video camera mounted over the sink, or may even include a video monitor to provide audiovisual cues (Figure 1). The COACH system can be thought of as three systems working together: 1) the tracking system, 2) the planning system, and 3) the prompting system, as depicted in Figure 2. The tracking system uses a video camera to identify the position of the user's hands, as well as interactions with task objects (i.e. the soap, towel, tap, water, and sink). The latest version of COACH uses edge skin colour segmentation and flocking routines to track hand and object positions Figure 3. There are several acceptable pathways to complete the handwashing task, as represented in Figure 4. The planning system is responsible for determining which step in the handwashing the user is completing and what the appropriate response (i.e. action) should be. The planning system is currently based on a partially observable Markov decision process (POMDP) model and is able to estimate unobservable states, such as the user's responsiveness and level of dementia. The planning system calculates the most probable state of the world, which is used with a pre-computed policy to determine what action COACH should take. Actions available to COACH are to do nothing (i.e. continue observing the user), select a prompt, or call the caregiver to intervene. The prompting system initiates the selected action.
If the user is prompted, the amount of detail
that is provided in the prompt varies in specificity to match the
user's abilities. For example, if the user is being asked to turn
on the tap, a low specificity prompt would be audio only (e.g. "[Subject
name], you are washing your hands. Turn on the tap.") while
a high specificity would have a video demonstrating the step plus
more detailed audio (e.g. "[Subject name], use the silver lever
in front of you to turn the tap on"). If necessary,
the device will repeat a cue or increase the level of specificity
of cuing to promote the correct response from the user. If the user
does not respond to any of the cues, the device calls for a caregiver
to provide assistance.
Figure 1: Photo of the washroom setup at the Toronto Memory Program
Figure 2: Outline of the systems of the COACH system
Figure 3: A captured video frame with raw image (top left), hand tracking and region (top right), belief state (bottom left) and skin image (bottom right) [Click to enlarge Fig. 3].
Figure 4: Flow diagram of steps involved in handwashing.
Future Work
Past clinical trials have guided our development of the prototype for the next set of clinical trials, which are to be home-based (i.e. within-home trials) set to start in late 2012:
- Tracking: The current colour segmentation/ flocking method will be combined with a stereovision camera to track the position of both hands, the position of task objects (e.g. soap and towel), and the interaction between the them in 3D.
- Planning: The POMDP has been refined using data from the previous set of clinical trials. This will allow the new system to more accurately determine the steps the user is completing, whether that sequence is acceptable, and whether a prompt should be issued.
- Prompting: The system will continue to play verbal or video (i.e. verbal and visual prompt over a LCD screen) prompts to the user when an error is made.
Funding Sources
Research Team
Alex Mihailidis, University of Toronto
Jesse Hoey, University of Waterloo
Stephen Czarnuch, University of Toronto
Marek Grzes, University of Waterloo
Babak Taati, University of Toronto
Jennifer Boger, University of Toronto
Craig Boutilier, University of Toronto
Geoff Fernie, Toronto Rehabilitation Institute
Pascal Poupart, University of Waterloo
