US10896756

US010896756B2
(12) United
Cook et al .
States Patent ( 10) Patent No .: US 10,896,756 B2
(45) Date of Patent : Jan. 19 , 2021
( 54 ) ENVIRONMENTAL SENSOR- BASED ( 56 ) References Cited
COGNITIVE ASSESSMENT
U.S. PATENT DOCUMENTS
( 71 ) Applicants : Diane J. Cook , Pullman , WA (US ) ; 2005/0113703 A1 * 5/2005 Farringdon A61B 5/0428
Maureen E. Edgecombe , Pullman , WA 600/509
(US ) ; Prafulla N. Dawadi , Foster City, 2013/0238538 A1 * 9/2013 Cook G05B 15/02
CA (US ) 706/20
( 72 ) Inventors : Diane J. Cook , Pullman , WA (US ) ; OTHER PUBLICATIONS
Maureen E. Edgecombe , Pullman , WA
(US ) ; Prafulla N. Dawadi , Foster City, Schmitter- Edgecombe, M. et al . , “ Assessment of Functional Change
CA (US) and Cognitive Correlates in the Progression from Healthy Cognitive
Aging to Dementia ,” Neuropsychology, vol . 28 , No. 6 , pp . 881-893 ,
( 73 ) Assignee : Washington State University , Pullman , Nov. 2014 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/
WA (US ) 24933485 .
( * ) Notice: Subject to any disclaimer, the term of this (Continued )
patent is extended or adjusted under 35
U.S.C. 154 ( b ) by 1134 days. Primary Examiner Taelor Kim
( 21 ) Appl. No .: 15 /135,390 (74 ) Attorney, Agent, or Firm — Ellen M. Bierman ; Lowe
Graham Jones PLLC
(22) Filed : Apr. 21 , 2016
( 57 ) ABSTRACT
( 65 ) Prior Publication Data
US 2016/0314255 A1 Oct. 27 , 2016 Methods, systems , and techniques for facilitating cognitive
assessment are provided. Example embodiments provide a
Related U.S. Application Data Cognitive Assessment Facilitator System CAFS , which
( 60 ) Provisional application No. 62 / 150,794, filed on Apr. facilitates the gathering and prediction of cognitive assess
21, 2015 . ment of individuals using machine learning and sensors
placed in the home of a resident. These predictive assess
(51 ) Int. Ci . ments can then be used by a clinician to further diagnose
G06F 16/25 ( 2019.01 ) and / or provide health intervention . In one embodiment, the
GO6F 16/906 ( 2019.01 ) CAFS comprises a sensor input module, a machine learning
( Continued ) engine ( or algorithm as part of another component ), a CAAB
( 52) U.S. CI . tool , and activity curve change engine (activity tools ) , and a
CPC G16H 50/20 (2018.01 ) ; G06F 16/254 reporting module 308. These components cooperate to pro
cess and transform smart home based sensor data into
( 2019.01 ) ; GO6F 16/906 ( 2019.01 ) ; G06F activity performance features and statistical activity features
19/00 (2013.01 ) ; which are then processing through a machine learning
( Continued ) engine to predict clinical cognitive assessment values .
( 58 ) Field of Classification Search
CPC G06F 19/00 ; G06N 20/00 ; G06N 20/10 ;
GO6N 20/20 ; G16H 50/20 19 Claims , 21 Drawing Sheets
See application file for complete search history . ( 12 of 21 Drawing Sheet (s ) Filed in Color)
CAA process
Receive sensor eve data rav

Adipurioin 4
Ruta a lively recognizer 29'clo

suorancally labai the raw 402
cu tata Heseno
sonts
5x95 cavity teltunar

features for the activity tuled
SenSof Ouipu ! date
Extract statstical any saturaz

Fair, the activity performance 205
esturss
Fnsitive through ? ) frein the
: Asachine leam sigorities by
USITE : Etatistical acavity 19 atillas
& Pedict Cºv, clicioai co :
Rotte trained inachoo
leechny igorithms on the
extracted sit.scal activity
performando foecures to prodict
cognitive erdor sobility Sccres
Output YeoAgnitive over

Roby scores
US 10,896,756 B2
Page 2
( 51 ) Int . Ci . Chen , C. et al ., “ CASASviz : Web -based Visualization of Behavior

G16H 50/20 ( 2018.01 ) Patterns in Smart Environments, ” in 2011 IEEE International Con
GOON 20/00 (2019.01 ) ference on Pervasive Computing and Communications Workshops
G06F 19/00 ( 2018.01 ) ( PERCOM Workshops ), IEEE , Mar. 2011 , pp . 301-303 , retrieved at
GOON 20/10
http://eecs.wsu.edu/~cook/pubs/percomllp2.pdf.
( 2019.01 ) Cook , D. et al., “ Activity Discovery and Activity Recognition: A
GO6N 20/20 ( 2019.01) New Partnership ,” IEEE Trans. Syst . Man Cybern. B 43 ( 3 ) ( 2013 )
(52 ) U.S. Ci . 820-828 , retrieved at https://eecs.wsu.edu/~cook/pubs/smc 12.pdf.
CPC G06N 20/00 (2019.01 ) ; G06N 20/10 Cook , D. et al . , “ CASAS: A Smart Home in a Box ,” Computer, vol .
(2019.01 ) ; GO6N 20/20 (2019.01 ) 46 , No. 7 , pp . 62-69 , Jul. 2013 , retrieved at https://eecs.wsu.edu/
cook /pubs/computer12.pdf. Dakos, V. et al ., Methods for Detect
( 56 ) References Cited
ing Early Warnings of Critical Transitions.
Dakos, V. et al . , “ Methods for Detecting Early Warnings of Critical
Transitions in Time Series Illustrated Using Simulated Ecological
OTHER PUBLICATIONS Data , ” PLoS ONE , vol . 7 , No. 7 , p . 241010 , Jan. 2012 , retrieved at
http://journals.plos.org/plosone/article?id=10.1371/journal.pone.
0041010 .
Schmitter - Edgecombe, M. et al . , " Cognitive Correlates of Func Dawadi, P. et al . , “ Automated Assessment of Cognitive Health
tional Performance in Older Adults : Comparison of Self-Report, Using Smart Home Technologies , ” Technol. Health Care 21 (4 )
Direct Observation , and Performance - Based Measures,” J. of the ( 2013 ) 323-343 , retrieved at https://eecs.wsu.edu/~cook/pubs/thc13.
International Neuropsychological Society JINS , vol . 17 , No. 5 , pp . pdf.
853-864 , 2011 , retrieved at https://eecs.wsu.edu/~cook/pubs/jins11. Dawadi, P. et al . , “ Automated Cognitive Health Assessment Using
pdf. Smart Home Monitoring of Complex Tasks,” IEEE Transactions on
Schmitter - Edgecombe, M. et al . , “ Development and Psychometric Systems , Man, and Cybernetics: Systems , vol . 43 , No. 6 , pp .
Properties of the Instrumental Activities of Daily Living : Compen 1302-1313 , 2013 , retrieved at https://eecs.wsu.edu/~cook/pubs/
sation Scale , ” Archives of Clinical Neuropsychology , J. of the thms 12.pdf.
National Academy of Neuropsychologists, vol . 29 , No. 8 , pp . Deschenes , C. et al . , “ Current Treatments for Sleep Disturbances in
776-792 , Dec. 2014 , retrieved at https://www.ncbi.nlm.nih.gov/ Individuals With Dementia , ” Curr Psychiatry Rep ., vol. 11 , No. 1 ,
pubmed / 25344901. pp . 20-26 , Feb. 2009 , retrieved at https://www.ncbi.nlm.nih.gov/
Schmutte , T. et al . , “ The Relation Between Cognitive Functioning pmc /articles /PMC2649672 /.
and Self -Reported Sleep Complaints in Nondemented Older Adults: Dodge, H. et al . , “ In -home walking speeds and variability trajec
Results From the Bronx Aging Study,” Behav. Sleep Med . 5 ( 1 ) tories associated with mild cognitive impairment, ” Neurology, vol .
( 2007 ) 39-56 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/ 78 , No. 24 , pp . 1946-1952 , Jun . 2012 , retrieved at https://www .
17313323 . ncbi.nlm.nih.gov/pubmed/22689734 .
Sriperumbudur, B et al ., “ The effect of kernelchoice of RKHS based Driscoll , H. et al . , “ Sleeping Well, Aging Well : A Descriptive and
statistical tests, ” in : Representations and Inference on Probability Cross - Sectional Study of Sleep in ‘ Successful agers’ 75 and Older , "
Distributions Workshop , NIPS , Vancouver, B.C , Canada, 2007 . Am . J. Geriatr. Psychiatry 16 ( 1 ) ( 2008 ) 74-82 .
Sugiyama, M. et al . , “ Least- Squares Two -Sample Test,” Neural Eeles , E. , “ Sleep and its management in dementia , ” Rev. Clin .
Networks, 24 ( 7 ) ( 2011 ) 735-751 . Geront . 16 ( 01 ) ( 2007 ) 59-70 .
Sun , F. et al . , “ Nonparametric Discovery of Human Routines from Farias, S. et al . , “ MCI is Associated With Deficits in Everyday
Sensor Data ,” in : 2014 IEEE International Conference on Pervasive Functioning, " Alzheimer Dis Assoc Disord . , vol . 20 , No. 4 , pp .
Computing and Communications, PerCom , IEEE , 2014 , pp . 11-19 . 217-223 , 2006 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/
Suzuki, T. et al . , “ Influence of Outdoor Activity and Indoor Activity 17132965 .
on Cognition Decline : Use of an Infrared Sensor to Measure Farrahi, K. et al . , “ Discovering Routines from Large - Scale Human
Activity ,” Telemedicine journal and e -health , journal of the Ameri Locations using Probabilistic Topic Models,” ACM Trans. Intell.
can Telemedicine Association, vol . 16 , No. 6 , pp . 686-690 , 2010 , Syst . Technol . 2 ( 1 ) ( 2011 ) 1-27 , retrieved at http://www.idiap.ch/
retrieved at https://www.ncbi.nlm.nih.gov/pubmed/20575611 . ~ gatica /publications/FarrahiGatica -tist11.pdf.
Wang, S. et al . , “ Activity Density Map Visualization and Dissimi Farrahi, K. et al., “ What Did You Do Today ? Discovering Daily
larity Comparison for Eldercare Monitoring, ” IEEE Transactions on Routines from Large - Scale Mobile Data , " in : Proceeding of the 16th
Information Technology in Biomedicine, vol . 16 , No. 4 , pp . 607 ACM International Conference on Multimedia , MM '08 , ACM
614 , Jul. 2012 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/ Press, New York , New York , USA , 2008 , pp . 849-852 , retrieved at
22547460 . http://www.idiap.ch/-gatica/publications/FarrahiGatica-mm08.pdf.
Zheng , J. et al . , “ Effective Routine Behavior Pattern Discovery from Galambos, C. et al . , “ Management of Dementia and Depression
Sparse Mobile Phone Data via Collaborative Filtering," in : 2013 Utilizing In -Home Passive Sensor Data,” Gerontechnology , vol . 11 ,
IEEE International Conference on Pervasive Computing and Com No. 3 , pp . 457-468 , 2013 , retrieved at https ://www.ncbi.nlm.nih .
munications, PerCom , IEEE , 2013 , pp . 29-37 . gov /pubmed / 24049513.
Albert, M. et al . , “ The diagnosis of mild cognitive impairment due Gretton , A. et al . , “ A Kernel Two- Sample Test , ” J. Mach . Learn .
to Alzheimer's disease: Recommendations from the National Insti Res . 13 ( 1 ) ( 2012 ) 723-773 .
tute on Aging - Alzheimer's Association workgroups on diagnostic Gross, A. et al . , “ Cognitive Predictors of Everyday Functioning in
guidelines for Alzheimer's disease ,” Alzheimer's & dementia , the Older Adults: Results From the ACTIVE Cognitive Intervention
journal of the Alzheimer's Association , vol . 7 , No. 3 , pp . 270-279 , Trial,” J. Gerontol. B Psychol. Sci . Soc . Sci . 66 ( 5 ) ( 2011 ) 557-566 ,
May 2011 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/ retrieved at https://www.ncbi.nlm.nih.gov/pubmed/21558167.
21514249 . Hayes, T. et al., “ Unobtrusive assessment of activity patterns
Artero , S. et al . , “ Disability and mild cognitive impairment: A associated with mild cognitive impairment,” Alzheimer's Dement.
longitudinal population -based study,” Int . J. Geriatr. Psychiatry 16 4 ( 6 ) ( 2008 ) 395-405 , retrieved at https://www.ncbi.nlm.nih.gov/
( 11 ) ( 2001 ) 1092-1097 . pubmed / 19012864.
Benjamini, Y. et al . , “ Controlling the False Discovery Rate: A Hodges, M. et al ., “ Automatic Assessment of Cognitive Impairment
Practical and Powerful Approach to Multiple Testing," J. R. Stat. Through Electronic Observation of Object Usage , ” in : P. Floréen , A.
Soc . Ser. B Stat. Methodol. 57 ( 1 ) ( 1995 ) 289-300 . Krüger, M. Spasojevic ( Eds . ) , Proc . International Conference on
Chaytor, N. et al . , “ Improving the ecological validity of executive Pervasive Computing, in : Lecture Notes in Computer Science, vol .
functioning assessment,” Archives of Clinical Neuropsychology, 6030 , Springer Berlin Heidelberg , Berlin , Heidelberg, 2010 , pp .
vol . 21 , No. 3 , pp . 217-227 , Apr. 2006 , retrieved at https://www .
ncbi.nlm.nih.gov/pubmed/16554143.
192-209 , retrieved at http://www.mwnewman.people.si.umich.edu/
pubs/pervasivel0 -hodges.pdf.
US 10,896,756 B2
Page 3
( 56 ) References Cited Ouchi , Y. et al., “ Impaired instrumental activities of daily living

affect conversion from mild cognitive impairment to dementia : the
OTHER PUBLICATIONS Osaki - Tajiri Project, ” Psychogeriatrics, vol . 12 , No. 1 , pp . 34-42 ,
Mar. 2012 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/
Hope , T. et al . , “ Predictors of Institutionalization for People with 22416827
Dementia Living at Home with a Carer,” Int. J. Geriat. Psychiatry, Paavilainen, P. et al . , “ Circadian activity rhythm in demented and
vol . 13 , No. 10 , pp . 682-690 , Oct. 1998 , retrieved at https : //www. non - demented nursing home residents measured by telemetric
ncbi.nlm.nih.gov/pubmed/9818303. actigraphy, " J. Sleep Res ., vol . 14 , No. 1 , pp . 61-68 , Mar. 2005 ,
Hotelling , H. , “ The Generalization of Student's Ratio,” The Annals retrieved at https://www.ncbi.nlm.nih.gov/pubmed/15743335.
of Mathematical Statistics, vol . 2 , No. 3 , pp . 360-378 , Aug. 1931 . Paavilainen , P. et al . , “ Telemetric activity monitoring as an indicator
Huynh, T. et al . , “ Discovery of Activity Patterns using Topic of long - term changes in health and well -being of older people , "
Models," in : Proceedings of the 10th International Conference on Gerontechnology , vol . 4 , No. 2 , pp . 77-85 , 2005 .
Ubiquitous Computing, UbiComp '08 , ACM Press , New York , New
York , USA , 2008 , pp . 10-19 , retrieved at http : //citeseerx.ist.psu . Pedrosa , H. et al . , “ Functional Evaluation Distinguishes MCI Patients
edu /viewdoc/download ? doi= 10.1.1.232.1932 & rep = rep1 & type = From Healthy Elderly People — The ADCS/ MCI/ ADL Scale , ” J.
pdf. Nutr. Health Aging 14 ( 8 ) ( 2010 ) 703-709 , retrieved at https : // www .
Jelicic , M. et al ., “ Subjective sleep problems in later life as ncbi.nlm.nih.gov/pubmed/20922349.
predictors of cognitive decline. Report from the Maastricht Ageing Pérès , K. et al . , “ Restriction in complex activities of daily living in
Study (MAAS ) ,” Int J Geriatr Psychiatry, vol . 17 , No. 1 , pp . 73-77 , MCI , Impact on outcome , ” Neurology, vol . 67 , No. 3 , pp . 461-466 ,
Jan. 2002 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/ Aug. 2006 , retrieved at https://www.ncbi.nlm.nih.gov/pubmed/
11802234 . 16894108 .
Kanis, M. et al . , “ Sensor Monitoring in the Home : Giving Voice to Podsiadlo, D. et al ., “ The Timed “ Up & Go " : A Test of Basic
Elderly People, ” in Pervasive Computing Technologies for Health Functional Mobility for Frail Elderly Persons.” JAGS , vol . 39 , No.
care ( PervasiveHealth ), 2013 7th International Conference on , Ven 2 , pp . 142-148,1991 , retrieved at https://www.ncbi.nlm.nih.gov/
ice , Italy, 2013 , pp . 97-100 . pubmed / 1991946.
Kanis, M. et al . , Sensor Monitoring in the Home : Giving Voice to Rashidi, P. et al . , “ Keeping the Resident in the Loop : Adapting the
Elderly People ,: in : 2013 7th International Conference on Pervasive Smart Home to the User,” IEEE Trans. Syst . Man Cybern . A 39 ( 5 )
Computing Technologies for Healthcare, PervasiveHealth , Venice, ( 2009 ) 949-959 , retrieved at https://eecs.wsu.edu/~cook/pubs/smca09.
Italy, 2013 , pp . 97-100 . pdf.
Krishnan , N. et al . , “ Activity Recognition on Streaming Sensor Rashidi, P. et al . , “ Mining Sensor Streams for Discovering Human
Data,” Pervasive and mobile computing, vol . 10 , pp . 138-154 , Feb. Activity Patterns Over Time," in : 2010 IEEE International Confer
2014 , retrieved at https://eecs.wsu.edu/~cook/pubs/pmc12b.pdf. ence on Data Mining , IEEE , 2010 , pp . 431-440 , retrieved at
LeBellego , G. et al . , “ A Model for the Measurement of Patient https://eecs.wsu.edu/~cook/pubs/icdm10.pdf.
Activity in a Hospital Suite,” IEEE Transactions on Information Riboni, D. et al . , “ Fine -grained Recognition of Abnormal Behaviors
Technology in Biomedicine , vol . 10 , No. 1 , pp . 92-99 , Jan. 2006 . for Early Detection of Mild Cognitive Impairment, ” Proc . PerCom
Luck, T. et al., Impact of impairment in instrumental activities of (2015 ) 149-154 .
daily living and mild cognitive impairment on time to incident Robben , S. et al . , “ How is Grandma Doing ? Predicting Functional
dementia : results of the Leipzig Longitudinal Study of the Aged , Health Status from Binary Ambient Sensor Data , " in : 2012 AAAI
Psychol. Med . 41 ( 5 ) ( 2011 ) 1087-1097 , retrieved at https : // www . Fall Symposium Series , Washington D.C , 2012 , pp . 26-31 , retrieved
ncbi.nlm.nih.gov/pubmed/20667169. at https://www.aaai.org/ocs/index.php/FSS/FSS12/paper/view/
Martin, J. et al . , “ Sleep Quality in Residents of Assisted Living 5605 .
Facilities: Effect on Quality of Life, Functional Status, and Depres Robben , S. et al . , “ Identifying and Visualizing Relevant Deviations
sion , ” J Am Geriatr Soc . , vol . 58 , No. 5 , pp . 829-836 , May 2010 , in Longitudinal Sensor Patterns for Care Professionals, ” in : 7th
retrieved at https://www.ncbi.nlm.nih.gov/pubmed/20722819. International Conference on Pervasive Computing Technologies for
McAlister, C. et al . , “ Naturalistic Assessment of Executive Function Healthcare, PervasiveHealth , Venice, Italy, 2013 , pp . 416-419 .
and Everyday Multitasking in Healthy Older Adults ,” Neuropsychol Robben, S. et al . , “ Longitudinal Ambient Sensor Monitoring for
ogy , development, and cognition. Section B , Aging , neuropsychol Functional Health Assessments: A Case Study,” in Proceedings of
ogy and cognition, vol . 20 , No. 6 , pp . 735-756 , Jan. 2013 , retrieved the 2014 ACM International Joint Conference on Pervasive and
at https://www.ncbi.nlm.nih.gov/pubmed/23557096 . Ubiquitous Computing Adjunct Publication — UbiComp ’14 Adjunct,
Noury, N. et al . , “ Building an Index of Activity of Inhabitants From New York , New York , USA , ACM Press, Sep. 2014 , pp . 1209-1216 .
Their Activity on the Residential Electrical Power Line , ” IEEE Scherder, E. et al., " Gait in ageing and associated dementias; its
transactions on information technology in biomedicine , a publica relationship with cognition ,” Neuroscience and Biobehavioral Reviews,
tion of the IEEE Engineering in Medicine and Biology Society, vol . vol . 31 , No. 4 , pp . 485-497 , Jan. 2007 , retrieved at https://www .
ncbi.nlm.nih.gov/pubmed/17306372 .
15 , No. 5 , pp . 758-766 , Sep. 2011 , retrieved at https: //www.ncbi.
nlm.nih.gov/pubmed/21896382. Schmitter-Edgecombe, M. et al, “ Naturalistic Assessment of Every
Ohayon, M. et al . , “ Meta- Analysis of Quantitative Sleep Parameters day Functioning in Individuals with Mild Cognitive Impairment:
From Childhood to Old Age in Healthy Individuals : Developing The Day -Out Task , ” Neuropsychology 26 ( 5 ) ( 2012 ) 631-641 ,
Normative Sleep Values Across the Human Lifespan ,” Sleep , vol . retrieved at https://www.ncbi.nlm.nih.gov/pubmed/22846035.
27 , No. 7 , pp . 1255-1273 , Nov. 2004 , retrieved at https ://www.ncbi. Stopping elderly accidents , deaths & injuries, Center for Disease
nlm.nih.gov/pubmed/15586779. Control and Prevention , 2017 , page available online at https : // www .
Ojala, M. et al . , “ Permutation Tests for Studying Classifier Perfor cdc.gov/steadi/pdf/TUG_Test-print.pdf.
mance," J. of Machine Learning Research, vol . 11 , pp . 1833-1863 ,
Mar. 2010 . * cited by examiner
U.S. Patent Jan. 19 , 2021 Sheet 1 of 21 US 10,896,756 B2
03
1
.
Fig
koments Diagnosi 205 206
tfCoAlrBs amcetinvsudrienscgplriednctailngase mnt fortCAcoutirvltsey apctivertnys

ACsogenimtveFSacyilsteomn
2
.
Fig
oza
wat(eichdntoelhdgy Sensor
207
CAcutriv teyECnhgaine MReopdrutlieng

306
3
.
Fig
MLaecrhnieg/AElngorinthem CognitveAse mnt SAtacistvcaly Extraco
SAFCsuoyngectimvnael SIenpsuortEngine Recognizer ActivtyPerfomanc EFxetarucos

CAAB process
Receive sensor event data in raw

Output form
Run activity recognizer logic to

automatically label the raw
output data from the sensor
Extract activity performance

features from the activity labeled
sensor output data
Extract statistical activity features

from the activity performance
features
(First time through ? ) Train the
.
machine learning algorithms by
using the statistical activity features
.
to predict known clinical scores
leaming algorithms on the
extracted statistical activity
performance features to predict
cognitive and /or mobility scores
Qutput predicted cognitive and/or

Fig . 4
ONCM09:004201o09.-405-k251 OFEatM09:00420109-.250-519 ONRM09:00420e209-l.5a0-21x5 ONCM09:004202o09.-285-k215 ONCM09:004202o09-.258-k125

502
5
.
Fig
ONM09:00420109.-405-251 OFFM09:00420109-.250-519 ONM09:00420209-.50-215 ONM09:00420209-.258-125 ONM4209:000309.-215-765

>
Sle p OActhiverty
250
,thetime(m288).Inthisgat5miniwerecdadataAnsrltceiaurbvlgptiaolhenysd
,andantbed:sadmcAnerigxolhufnadsvermilgaptvtnoesyd
hsofamx=tuwasdc.T"ogeortnmiahvupgrenahltsgdy
0ratiAM1enp2trde:s0vnaxtls
***
) normalized( interval that

in activity in spent time .Avg
6
.
Fig
thefstistoc.Theinad-bpeohriuaemtrspnovaugeitcmtpsdoyn
;
}
oforecissandthetNarer,.Inoaliaetcrmugpclasoeintghpaiesdcl fsthetbycisp,.Inairbeghmnstpfaiosercnmtapidlgce byiscs,a.Fdthepfotw)setof(uanebihclartmouintslraglaemhiydc 7

.
Fig
.dinacofsntiohugrmhnfbvtuigceanorsyt
.
Ona .
855
To ls Network (eSy.stgei.ns
Other Progams Data Repositre 2806
OCotmhpuetr MReadiblae 2805

1870 CAPIAFS
2804
Perf.Activity
Devices 8
.
Fig
SFACasoycgienltmavoer Smart-home1/1 803

(
SCoymsptuemr Memory
SIenpsourt RCA B ecognizer Perf.Activity
Display
9
.
Fig
linerinx-axis.Thehtoagewiththeyseorcpisaoezpxrenitcasl camandthesrlvage.ecpeirntoainsrceaentls
DofR(left)andTUGracislBitreAngsbiuNhmcteaStonl
incroyitnineg ctioKnmiseostoimiston
.Uumaurm
?
)b(topandRcTheaporefMtdiwlcSuaetoinEmlds 10
.
Fig
andtrdw)sTUG(lReicrBfhemongAirfvqnhNeuatdSsl
aSVRtotareuswirsnazediodnsw
??
3
)(lRandaSVRpbtopcCoreBfMtdAiwlcfSuaeNtointESlmds
and-lrfdutheSVRwastw)sTUG(aecirhnbodtgeulvnhmrengtsdy theSVRtotwasufsetofs.ThecaAR-leorsamtbipveultirecdnasl 11
.
Fig
M
JOJE
thedincb)ofSVMw(AUCandG-MpClieoasrief tamciyznheogd 12
.
Fig
atdintotwocwass.TheR)(leftandTUGrilcBifaorAgsertNhieznStsd
thetacuSsfhroemnhVspitlouedMrtgs
{ {
OK
AUG
(AUCandG-Mean)oftheSVMpCleftTUGrRcwleasiBhfogArimfchNlaytnictSeog
dtfutheSVMiswscare.arelnhirotdvaeunrcmitlgsdy twoisathecanddStotsetwasufTheloermnVsaptilMeumcratosdnzel
13
.
Fig
,clas es
14
.
Fig
.
***
PM12:0
n.dhsaofmt**uwascgoertnaimuvgrenhatglsdy iat5mintcwdaAgs.liecr2ubmlgrv8atieonsyd
(
,
.
)
>
-
s.TademtcofAnigoxfuhradvgempithvlstneyd
Melax
X0 0
) normalized(interval that in activity in spent time .Avg

15
.
Fig
min.Iiis5ofthetimedThesizekluacngtsedriaevgnixctgelyd
andofabmpimaptheHcsetriwbvuaetniocstxye
16
.
Fig
13
sitfofasadkl.pcA-nuavisectrzmawvinreogcsal
0.6
distance pairwise Average
17
.
Fig
motion
www .
Š
man
sivwofaDgntirsezgbdauevto.iswln
3
.........
mm
size window aggregation of Distribution

18A
.
Fig
18B
.
Fig
820
0.33 0.32
9
10
of6mwonasaPbyrctwoTheoleuihngCctoadrnlAieuhotgRswemd
19
.
Fig
of30dswanagiarnegyztdaoisehwn
60
20
.
Fig
wonasaPbyrcoftToleuihngCwaedrlnAiuogetRhpwsmd
M AM M MA
withanaof6msize30days.wgoirnegtdahoiswn
mengenai
US 10,896,756 B2
1 2
ENVIRONMENTAL SENSOR-BASED the relationship between different high-level features repre
COGNITIVE ASSESSMENT senting the location and transition patterns of an individual's
indoor mobility behavior with the Assessment of Motor and
CROSS - REFERENCE TO RELATED Process Skills (AMPS ) scores . Similarly, Suzuki and Murase
APPLICATIONS 5 [7 ] compared indoor activities and outings with Mini-Mental
State Examination (MMSE) scores . Dodge et al . used latent
This application claims the benefit of U.S. Patent Appli trajectory modeling techniques to explore the relationship
cation No. 62 / 150,794 , entitled “ MODELING PATTERNS between gait parameters and cognition [ 8 ] . Similarly, LeB
OF ACTIVITIES USING ACTIVITY CURVES,” filed Apr. ellego et al. [ 9 ] investigated the relationship between indi
21 , 2015 , which is incorporated herein by reference in its 10 cators such as mobility and agitation with patient health
entirety. status in a hospital setting.
In other work, researchers such as Galambos et al . [ 10 ]
STATEMENT REGARDING FEDERALLY developed techniques to visualize long -term monitoring of
SPONSORED RESEARCH OR DEVELOPMENT sensor data including activity level and time spent away
15 from home [ 10] , [ 11 ] . Similarly, other researchers have
This work was supported in part by grants from the developed techniques to visualize activity and behavioral
National Institutes of Health (RO1EB015853 and patterns by monitoring them with smart home sensors [ 12 ] ,
RO1EB009675 ) and by a grant from the National Science [ 13 ] , and by monitoring consumption of electricity usage
Foundation ( 1064628) . [ 14 ]
20 In earlier work , we demonstrated a correlation between
TECHNICAL FIELD smart home sensor - based performance measurements of
simple and complex ADLs and validated performance mea
The present disclosure relates to methods, techniques, and sures derived from direct observation of participants com
systems for using environmental sensor- based data to pro- pleting the ADLs in a smart home laboratory [ 15 ] .
vide insights into cognitive assessment and, in particular, to 25 In addition, researchers have studied the problem of
methods , techniques, and systems for using sensor-based automated discovery and recognition of daily activity rou
data to determine and model activities of daily living, to tines using the data collected from wearable sensors [ 39 ] ,
predict cognitive assessment, and / or to detect change in [ 40 ] , GPS signals [ 41 ] and mobile phones [ 42 ] using algo
functional cognitive abilities over time . rithms such as topic modeling [ 39 ] and collaborative filter
30 ing [ 43 ] . In these approaches, raw sensor data are converted
BACKGROUND to a bag -of-words representation which contains the histo
gram of activity label occurrences or histogram of location /
A decrease in cognitive and /or physical health may proximity information . Data from wearable sensors can be
severely impact an individual's ability to function indepen- used to discover daily routines such as having lunch and
dently. Several clinical studies support a relationship 35 brushing teeth [ 39 ] . Similarly, data from mobile phones can
between daily behavior and cognitive and physical health be used to recognize routines such as commuting to office
[ 1 ] . Everyday activities like cooking and eating are essential and working
Activities of Daily Living (ADLs ) that are required to Researchers also have proposed visualization techniques
maintain independence and quality of life . For example , to visualize daily activity patterns. For example, Galambos
some everyday behavior parameters affected by cognitive 40 et al . [ 44 ] , [ 45 ] developed methods to visualize activity
health include areas such as computer usage (keyboard and level , time spent away from home , deviations in activities of
mouse usage , typing speed, computer game performance ), daily living , and behavioral patterns. Similarly, other
mobility ( ability to move around, climb stairs , stride length ), researchers have developed techniques to visualize devia
gait ( gait velocity, balance ), and everyday activities ( ability tions in activity routines and behavioral patterns using smart
to initiate and complete ADL such as bathing, toileting, 45 home sensors [ 46 ] , [ 47 ] . These methods provide a tool to
eating, etc. ) . Decline in the ability to independently perform understand sensor -monitoring data and to study daily activ
ADLs has been associated with placement in long -term care ity routines. However, these approaches rely on manual
facilities, shorter time to conversion to dementia , and a inspection of the data in order to make any higher -level
lower quality of life for both the functionally -impaired 50 conclusions regarding daily routines .
individuals and their caregivers [ 2 ] . The two - sample test discussed below is a widely used
The relationship between inhome sensor -based measure- statistical analysis tool to compare between two sample
ments of everyday abilities and corresponding clinical mea- populations . Classical two - sample tests such as the t - test are
surements has been explored using statistical tools and used to compare the means of two populations having the
visualization techniques. Researchers have correlated sensor same or different variances . However, the t - test is a para
measurements of sleep patterns, gait , and mobility with 55 metric test that is limited to comparing between two Gauss
standard clinical measurements and self - report data . In one ian distributions. Other examples of non -parametric classi
such work , Paavilainen et al . [ 4 ] monitored the circadian cal versions of two -sample tests are the Wald - Wolfowitz
rhythm of activities of older adults living in nursing homes runs test , the Anderson -Darling test and the Kolmogorov
using the IST Vivago WristCare system . In this study, they Smirnov test [48 ] .
compared the changes in activity rhythms with clinical 60 Recently, Maximum Mean Discrepancy ( MMD ) was pro
observations of subject health status. In a separate study, posed as another non -parametric two - sample test technique
these researchers [ 5 ] studied the relationship between [ 49 ] . MMD compares the means of two distributions in a
changes in the sleep pattern of demented and non - demented universal reproducing kernel Hilbert space and has superior
individuals over a 10 -day period . performance to several of the classic two-sample tests .
Several other researchers have considered the relationship 65 However, the superior performance of MMD relies on a
between sensor-based activity performance and clinical valid choice of a kernel and kernel width , and recommen
health assessment. For example , Robben et al . [ 6 ] studied dations have been made in the literature for obtaining
US 10,896,756 B2
3 4
optimal performance with MMD [ 50 ] . Similarly, the Least FIG . 2 is an example block diagram of the data flow in the
Squares Sample Test (LSTT ) technique has been proposed in smart home based sensor environment to predict assessment
the literature to make use of permutation to perform two- and change using an example Cognitive Assessment Facili
sample tests [ 51 ] . In the LSTT based two -sample test , tator System .
divergence is estimated using the density ratio estimation 5 FIG . 3 is an example block diagram of components of an
technique and the permutation -based technique is used to example Cognitive Assessment Facilitator System running
one or more CAAB tools and / or activity curve tools for
test the significance of the estimated divergence . Such predicting
permutation -based tests are preferable because they are activity pattern cognitive assessment and / or for determining
changes.
data - centric approaches that make inferences directly from 10 FIG . 4 is an example
data . block diagram of an overview of an
Also , researchers have developed functional assessment example CAAB tool process of an example Cognitive
algorithms based on different parameters of everyday abili Assessment Facilitator System .
FIG . 5 is an example block diagram of activity labeling
ties . These researchers have studied correlations between automatically performed by an activity recognition logic of
everyday abilities and corresponding standard clinical mea- 15 an example CAFS.
surements . Researchers have correlated sensor measure FIG . 6 is an example aggregated activity curve used by
ments of sleep patterns, gait , activity rhythms, indoor activi activity curve tools in an example Cognitive Assessment
ties and outings, and mobility with standard clinical Facilitator System .
measurements such as MMSE and self-report data . For FIG . 7 demonstrates the steps involved in detecting
example , Paavilainen et al . [ 54 ] compared the changes in 20 changes in an activity curve using a permutation -based
circadian rhythm of activities of older adults living in method
nursing homes with clinical observations of the health status FIG . 8 is an example block diagram of an example
of subjects. In other work , Robben et al . [ 55 ] 6 ] , [ 57 computing system that may be used to practice embodiments
studied the relationship between different high - level features of an example Cognitive Assessment Facilitator System .
representing the location and transition patterns of an indi- 25 FIG . 9 plots the distribution of TUG and RBANS clinical
vidual's indoor mobility behavior, namely the frequency, assessment scores against the ages of the participants.
duration and times being carried out , with the Assessment of FIG . 10 illustrates the correlation coefficients and RMSE
Motor and Process Skills ( AMPS ) scores [ 58 ] . Other actual ( root mean square error) values between predicted and
researchers have studied the relationship between walking cognitive assessment results using different trend
speed and the amount of in - home activity among healthy 30 removal techniques.
older adults and older adults with Mild Cognitive Impair values whenillustrates
FIG . 11
machine
the correlation coefficients and RMSE
learning engine was trained using
ment ( MCI ) [ 59 ] . These researchers found out that coeffi randomly labeled activities versus activity recognition algo
cient of variation in the median walking speed was higher in
the MCI group as compared with the healthy group . How 35 rithm labeled activities .
FIG . 12 illustrates variations in performance of learning
ever , none of these works considered parameters reflecting algorithms when trained with class labels that were dis
the performance of activities of daily living. cretized at different thresholds.
Other researchers have developed functional assessment FIG . 13 illustrates variations in performance of learning
algorithms based on performance of an individual in a fixed algorithms when trained with randomly labeled activities .
set of activities . They have correlated the performance in 40 FIG . 14 is an example aggregated activity curve that
these activities with the direct observation of participants model eight different activities of daily living .
completing the ADLs . In one such work , Dawadi et al . [ 60 ] , FIG . 15 is a heat map representation of the pairwise
[ 61 ] proposed learning algorithms to obtain activity perfor distance matrix between activity distributions computed
mance measures of simple and complex ADLs from sensor from the aggregated activity curve of FIG . 14 .
data and correlated them with validated performance mea- 45 FIG . 16 illustrates how activity distribution distances
sures derived from direct observation of participants com- within an activity curve change as a function of time
pleting the ADLs . They also studied the relationship interval.
between sensor - detected activity performance measures and FIG . 17 illustrates the variations in the length of an
overall cognitive health . In another work, Hodges et al . [ 62 ] aggregate window at different interval sizes calculated using
correlated sensor events gathered during a coffee-making 50 available sensor data .
task with an individual's neuropsychological score . Simi- FIGS . 18A and 18B illustrate the correlations between
larly, in an another research effort by Riboniet al . [ 63 ] activity change scores and clinical scores (of TUG and
researchers developed a Fine - grained Abnormal BEhavior RBANS ) .
Recognition ( FABER ) algorithm to detect abnormal behav- FIG . 19 illustrates different continuous change score for
ior using a statistical- symbolic technique. These researchers 55 two different residents.
hypothesized that such abnormal activity routines may indi- FIG . 20 illustrates different continuous sleep change score
cate the onset of early symptoms of cognitive decline. for the two different residents shown in FIG . 19 .
BRIEF DESCRIPTION OF THE DRAWINGS DETAILED DESCRIPTION
60
The patent or application file contains at least one drawing Smart home technologies offer potential benefits for
executed in color. Copies of this patent or patent application assisting clinicians by automating health monitoring and
publication with color drawings will be provided by the wellbeing assessment. The embodiments described herein
Office upon request and payment of the necessary fee. provide enhanced computer- and network - based methods,
FIG . 1 is an example block diagram of a smart home 65 techniques , and systems and tools for monitoring Activities
sensor environment for providing continuous data regarding of Daily Living (ADLs) using automatically collected (con
behavioral activities using an example CAFS . tinuous) sensor - based data and , based upon such data , pro
US 10,896,756 B2
5 6
viding technology assisted cognitive assessments of indi- Example Cognitive Assessment Facilitator Systems also
viduals. The sensor-based data can be used to predict provide a set of Activity Curve (AC ) tools that facilitate the
cognitive and physical health assessments that correlate to modeling and analyzing of activity -based behavioral pat
certain standard clinical assessments as well as to measure terns to detect changes to the activity curves of individuals
changes over time in an individual's cognitive health by 5 over a period of time . An “ activity curve ” represents an
modeling activity patterns . Using such tools , it is possible to abstraction of an individuals normal daily routine based on
help , for example, the aging population by prompting indi automatically recognized activities . Changes to activity
viduals to perform needed activities such as taking medica curves can be used to identify and quantify or even predict
tion on time or to eat, perform home automation , monitor the changes in cognitive and /or physical health so that improved
health of an individual or even to provide preventive mea 10 healthcare such as activity aware interventions can be
administered . In overview , the Activity Curve tools define
sures such as alerts to medical personnel. and use a set of algorithms to compare activity curves
Example embodiments provide a computer based Cogni generated using the smart -home sensor data collected as
tive Assessment Facilitator System ( “ CAFS ” ) which com above in conjunction with statistical evaluation and machine
prises tools that utilize statistical and machine learning 15 learning techniques. Changes to the activity curves are
techniques in conjunction with data received in real time analyzed to identify the possibility of changes to an indi
from sensors placed in an individual's residence (home , vidual's cognitive or physical health and to provide a
apartment, and the like) to predict cognitive health and functional health assessment. Experiments have been con
changes in a person's activities over time . These sensors are ducted to evaluate the performance of the Activity Curve
sometimes referred to hereinafter as “ smart home sensors . " 20 tools and to validate the results using data from 18 real
The smart home sensors provide the capability of providing world smart homes with older adult residents . These experi
measurements, such as measuring activities in real- time (or ments and their results are presented in Appendix B , incor
near real - time) , which can be stored and later analyzed or porated herein by reference in its entirety.
analyzed in real -time using streaming capabilities designed Smart Home - Based Sensor Environment
for capturing large amounts of data. In an example embodi- 25 FIG . 1 is an example block diagram of a smart home
ment, the sensors are placed on the ceilings , walls , ingress sensor environment for providing continuous data regarding
and egresses such as doorways and windows, and / or asso- behavioral activities using an example CAFS . Floor plan
ciated with appliances, so that identification, duration and 101 shows an example smart home environment as envi
location of an activity can be measured . As the ability to sioned by the smart home laboratory used to develop and test
provide sensors associated with appliances increases , for 30 CAAB . The example residence shown by floor plan 101 is
example via Internet of Things (IoT ) technologies or the a single resident apartment, with at least one bedroom , a
like , the smart home sensors may improve in their capacities kitchen , a dining area , and at least one bathroom , although
to measure activities of daily living beyond those discussed many other variations of layout and rooms are contemplated .
herein . Here, the terms “ residence ” or “ home” or “ smart home "
Example Cognitive Assessment Facilitator Systems pro- 35 refers to any potential layout that could be enhanced with
vide a Clinical Assessment Using Activity Behavior tool technology sensors including a single or multi- residence
( CAAB ) to predict the cognitive and mobility scores of home , apartment, condominium , townhome, or the like . The
smart home residents by monitoring a set of basic and homes are equipped with combination motion /light sensors
instrumental activities of daily living . CAAB may be imple- throughout the environment, for example on the ceilings
mented in a networked environment that receives data from 40 e.g. , sensor 103 , and combination door/ temperature sensors
the sensors , performs various measurements and uses vari- on the cabinets and doors , e.g. sensor 102. The location of
ous tools to quantify and convert the data and predicts each sensor is indicated with the corresponding motion (M) ,
cognitive assessments as output. In overview , CAAB first light ( LS ) , door (D ) , or temperature (T ) sensor number. For
labels the raw data with activity labels and processes the example, sensor D002 T102 corresponds to a door /tempera
activity - labeled sensor dataset to extract activity perfor- 45 ture sensor at location 102. Other types and locations of
mance features. CAAB then extracts statistical activity fea- sensors are contemplated . The sensors are meant to unob
tures from the activity performance features to train machine trusively and continuously monitor the daily activities of the
learning algorithms that can predict the cognitive and mobil- residents without awareness or interference by the residents .
ity scores . Various machine learning algorithms can be The residents perform their normal activities in their smart
incorporated . 50 residences, unobstructed by the smart home instrumentation
Experiments were conducted to evaluate the performance Overview of a Cognitive Assessment Facilitator System
of CAAB and to validate the results using data from 18 (CAFS )
real - world smart homes with older adult residents. In the FIG . 2 is an example block diagram of the data flow in the
prediction -based experiments, a statistically significant cor- smart home based sensor environment to predict assessment
relation ( r = 0 :72 ) between CAAB predicted and clinician- 55 and change using an example Cognitive Assessment Facili
provided cognitive assessment scores was obtained and a tator System . Technology assisted assessment environment
statistically significant correlation ( r = 0 : 45 ) between CAAB 200 comprises a residences 202 with smart sensor technol
predicted and clinician -provided mobility scores was ogy for measuring activities such as the Activities of Daily
obtained . Further, for the classification -based experiments, Living of resident201. The data is collected on a continuous
CAAB showed an accuracy of 72 % while classifying cog- 60 basis and stored somewhere over network 203. In some
nitive assessment scores and 76 % while classifying mobility embodiments it is stored in cloud based server environ
scores . These prediction and classification results suggest ments, such as cloud storage 207 , others it is served on a
that it is feasible to predict standard clinical scores using streaming basis to other computer systems such as server
smart home sensor data and learning -based data analysis as computer system 204 that hosts the CAAB and AC tools . In
performed by CAAB . These experiments and their results 65 other environments the data is collected elsewhere ( e.g. ,
are presented in Appendix A , incorporated herein by refer- onto another computer system or network storage device and
ence in its entirety. pulled as needed by computer system 204 ) . Other architec
US 10,896,756 B2
7 8
tures for getting the data to computer system 204 are niques. The embodiments described also can be practiced
similarly incorporated. Once delivered , the raw sensor data without some of the specific details described herein , or with
is processed by the CAAB tools 205 on computer system other specific details, such as changes with respect to the
204 to be annotated by CAAB with activity labels . There- ordering of the logic , different logic , etc. Thus, the scope of
after, the CAAB tools 205 employ machine learning tools to 5 the techniques and/ or functions described are not limited by
predict clinical assessments which can be then forward to a the particular order, selection , or decomposition of aspects
human clinician 208 for further health purposes such as described with reference to any particular routine , module,
preemptive intervention , consultation with the resident, fur component , and the like .
ther interpretation or diagnosis 209 , and the like. In some Clinical Assessment Using Activity Behavior Tool (CAAB )
deployments, Activity Curve tools 206 are employed to 10 The Clinical Assessment using Activity Behavior tool ( s )
provide further analysis and computation on the labeled data (CAAB ) ( such as CAAB tools 205 ) models a smart home
in order to detect change in activity patterns and produce a resident's daily behavior and predicts the corresponding
functional health assessment. This functional health assess standard clinical assessment scores . In overview , CAAB
ment can also be forwarded to a clinician 208 to further act
upon . 15 extracts and uses statistical features that describe character
FIG . 3 is an example block diagram of components of an istics of a resident's daily activity performance to train
example Cognitive Assessment Facilitator System running machine learning algorithms that can then be used on the
one or more CAAB tools and / or activity curve tools for resident's further daily activity performance to predict the
predicting cognitive assessment and / or for determining clinical assessment scores .
activity pattern changes, respectively . In one example 20 By way of introduction , the notations displayed in Table
embodiment, CAFS comprises one or more functional com- 1 below are defined and used as follows for describing
ponents /modules that work together to predict cognitive functions in CAAB :
assessment and / or to provide a functional assessment uti
lizing activity curve change analysis. For example, a Cog TABLE I
nitive Assessment Facilitator System 300 may comprise a 25 n Number of Activities
sensor input module 301 , a machine learning engine (or T Total number of data collection days
algorithm as part of another component) 305 , a CAAB tool Set of n activities being modeled
310 , and activity curve change engine ( activity tools ) 307 , Pi Activity performance feature vector for activity i modeled over
and a reporting module 308. As described in FIG . 2 , the data collection period T
Activity performance feature d for activity i activity on day t
CAFS receives sensor input monitoring a resident via sensor 30 Pij , d, t Time point at which clinical measurements are made
input engine 301. The CAAB then takes the input, trans S; Clinical assessment score measured at time point ?
forms the ADL input into labeled activities via activity W Sliding window size
recognizer 302 , extracts performance features from this
input using activity performance features extractor 303 , FIG . 4 is an overview block diagram of the process for
extracts statistical features from these activity performance 35 providing
features using statistical performance features extractor 304 , tool of anaexample
clinical assessment score by an example CAAB
Cognitive Assessment Facilitator Sys
and then primes and then uses the machine learning engine tem . The blocks shown
305 via the cognitive assessment predictor 306 to predict processed by one or morein the logic of FIG . 4 may be
components of a computing
clinic assessments which are forwarded to a clinician (or
otherwise used ) via reporting module 308. In some 40 system or environment which communicate with each other
examples, the CAFS includes an activity change engine 307 over one or more networks. Smart home sensors produce a
which uses the activity recognize 302 ( or another one ) and continuous sequence of time- stamped sensor readings , or
extracts and compares change curve data to provide a sensor events . These sensors continuously generate raw
functional cognitive assessment of the resident ( have the sensor events while residents perform their routine activities
characteristics of the activities changed over time). This 45 of daily living . In block 401 , the CAAB collects continuous
functional assessment is then forward to a clinician (or raw output data from a set of sensors in the residence . In
otherwise used) via reporting module 308 . block 402 , CAAB employs activity recognition logic or
Although the techniques of smart sensor technology and algorithms ( as embodied by software , hardware, firmware or
the Cognitive Assessment Facilitator are generally appli- a combination or system element such as an “ activity
cable to any type of measurement of cognitive assessment, 50 recognizer ” ) to automatically annotate each of these sensor
the phrase cognitive assessment is used generally to imply events with a corresponding activity label . As explained
any type of assessment of cognitive abilities including the further below , the events may be streamed in , stored and
activities of daily living discussed herein . Also , although forwarded, delivered in bursts, pulled from storage by the
certain terms are used primarily herein , other terms could be CAAB , etc. In some example CAAB tools , the activity
used interchangeably to yield equivalent embodiments and 55 labeling is performed in real -time (or near real - time) . In
examples. In addition, terms may have alternate spellings some example CAAB tools , the activity recognition logic
which may or may not be explicitly mentioned , and all such labels the raw output data from the sensor events using a
variations of terms are intended to be included . sliding window as explained further below.
Example embodiments described herein provide applica- Activity recognition algorithms map a sequence of raw
tions , tools , data structures and other support to implement 60 sensor events onto an activity label Aj , where the label is
a Cognitive Facilitator Assessment System to be used for drawn from the predefined set of activities A= { A1 ,
predicting a clinical cognitive assessment and / or reporting A2 , ... , An }. Our activity recognition algorithm generates
functional changes in activities . Other embodiments of the a label that corresponds to the last event in the sequence (i.e. ,
described techniques may be used for other purposes. In the the label indicates the activity that was performed when the
following description, numerous specific details are set 65 last event was generated ). Activities from set A can be
forth , such as data formats and code sequences , etc. , in order recognized even when the resident interweaves them or
to provide a thorough understanding of the described tech- multiple residents perform activities in parallel.
US 10,896,756 B2
9 10
In block 403, CAAB extracts activity performance fea- Activity recognition algorithms label activities based on
tures from activity labeled smart home sensor data and readings ( or events) that are collected from smart environ
utilizes these features to predict standard clinical assessment ment sensors such as the smart home sensors shown in FIG .
scores . Therefore, there are two steps involved in CAAB : 1. As described earlier, the challenge of activity recognition
Modeling the ADL performance from the activity - labeled 5 is to map a sequence of sensor events onto a value from a set
smart home sensor data ; and of predefined or predetermined activity labels . These activi
Predicting the cognitive and mobility scores using a ties may consist of simple ambulatory motion , such as
machine learning algorithm . walking and sitting, or complex basic or instrumental activi
Activity Modeling: ties of daily living, depending upon what type of underlying
The tool extracts a d - dimensional activity performance sensor
10
The
technologies and learning algorithms are used .
activity recognition algorithm used by CAAB (AR)
. Pid > to model the daily activity [ 22 ] , recognizes
feature vector P;=<P1,1;, 19! ... activities of daily living, such as cooking,
performan ance of an activity A? . Observation Pi,det provides a eating, and sleeping using streaming sensor data from envi
value for feature d of activity A, observed on day t ( 1stsT) . ronmental sensors such as the motion sensors and door
The set of all observations in P , is used to model the 15 sensors described with reference to FIG . 1. These motion
performance of A, during an entire data collection period and door sensors are discrete - event sensors with binary
between day 1 and day T. For example, the activity " sleep " states ( On / Off, Open / Closed ). In an example CAAB , human
may involve features such as duration and number of sensor annotators label one month of sensor data from each smart
events detected while sleeping . home with predefined activity labels to provide the ground
Additionally, during the same data collection period, 20 truth activity labels for training and evaluating the algo
standard clinical tests are administered for the resident every rithm . The inter - annotator reliability ( Cohen's Kappa ) val
m time units , resulting in clinical assessment scores S? , ues of the labeled activities in the sensor data ranged from
S2 , ... , Se (p = T / m ). The clinical tests are administered 0:70 to 0:92 , which is considered moderate to substantial
biannually ( m = 180 days ). Therefore , the clinical measure- reliability . The trained model was then used to generate
ments are very sparse as compared to the sensor observa- 25 activity labels for all of the unlabeled sensor data.
tions . The baseline clinical measurement, S? , is collected AR identifies activity labels in real time as sensor event
after an initial 180 days of smart home monitoring . sequences are observed . Thus, the algorithm can be
Clinical Assessment/Clinical Assessment Scores Predic- employed on continuously streamed data . This is accom
tion: plished by moving a sliding window over the data and using
CAAB's goal is to accurately predict clinical assessment 30 the sensor events within the window to provide a context for
scores at time k, or Sk, using activity performance data Pi labeling the most recent event in the window. The window
between time points j and k , j < k CAAB relies on an activity size is dynamically calculated based on the current sensor.
recognition (AR ) algorithm to generate labeled data for the Each event within the window is weighted based on its time
performance feature vector that is an integral component of offset and mutual information value relative to the last event
activity modeling. The method for activity recognition is 35 in the window . This allows the events to be discarded that
explained briefly below and explored in detail elsewhere are likely due to other activities being performed in an
[ 16 ] . interwoven or parallel manner. A feature vector is calculated
In block 404 , the CAAB extracts statistical activity fea- using accumulated sensor events in a window from the
tures ( certain statistical metrics based upon the extracted labeled sensor data collected over a month . The feature
activity performance features ), which are used to run a 40 vector contains information such as time of the first and last
machine learning algorithm . In block 405 , if this is the first sensor events , temporal span of the window , and influences
time processing the activity data from a resident, the CAAB of all other sensors on the sensor generating the most recent
trains the machine learning algorithm (e.g. , tool , system , event based on mutual information . Currently, the AR used
function, etc. ) using the extracted statistical activity features by an example CAAB embodiment recognizes the activities
versus the actual clinical assessment measurement S? . Then 45 monitored with 95 % accuracy based on 3 - fold cross vali
in block 406 , the CAAB runs the trained machine learning dation . An example of activity -labeled sensor data is pre
algorithm on the extracted activity performance features and sented in FIG . 2 of [ 22 ] . More details on this and other
in block 407 outputs a predicted cognitive and / or mobility approaches to activity recognition can be found in the
score . As shown in FIG . 2 , this score can be provided, for literature [ 16 ] . The CAAB tools can incorporate activity
example, to a clinician for further diagnosis or intervention . 50 recognition algorithms in addition to that described here .
A. Activity Recognition Algorithm B. Modeling Performances of Activities and Mobility Per
FIG . 5 is an example block diagram of activity labeling formances
automatically performed by an activity recognition logic of The first step of the CAAB tool once the data is annotated
an example OAFS . Each sensor event is represented by four is to model the performance of the activities in set A ( steps
fields: date, time , sensor identifier, and sensor value . The 55 403 of FIG . 4 ) . CAAB models activity performance by
raw sensor data does not contain activity labels. An activity extracting relevant features from the activity - labeled sensor
recognition (AR ) algorithm ( running, for example, as part of data . For each activity A ,EA , the CAAB can represent such
CAAB or external to CAAB ) labels collected raw sensor performance features using the d - dimensional activity per
data ( individual sensor events) with the corresponding formance feature vector P ;= < Pi,1, P1,2 , ... , Pi,d > .
activities . Data 501 shows sensor data collected prior to 60 Depending upon the nature of the sensor data and the
automatic labeling. Data 502 shows sensor data after the AR performance window we want to monitor, the CAAB can
algorithm has automatically labeled the data . CAAB utilizes aggregate activity performance P; for activity A, over a day,
sensor data collected from the smart homes without altering week , or other time period . In one example CAAB, activity
the resident's routine and environment. Therefore, the AR performance features were aggregated over a day period (the
approach offers an ecologically valid method to characterize 65 time unit is one day ). For example , if the sleep activity
the ADL parameters and assess the cognitive and physical performance P.1 , is calculated as the time spent sleeping in
health of a smart home resident [ 3 ] . the bedroom on day t , the observation P1,1, ,t + 1 occurs one day
US 10,896,756 B2
11 12
after observation Pi,1, For each individual, the CAAB cal- of times sensor events are triggered on bed to toilet activity
culates activity performance features for the entire data because the data shows that the number of sensor events
collection period T for all activities in the activity set A
generated when performing the bed to toilet activity is often
( 1stsT ). very low . Because of the known importance of sleep and its
Activity performance is modeled by an example CAAB 5 relationship with physical and cognitive health , CAAB
using two (d=2 ) specific activity performance features: a conducts a separate analysis of sleep and bed to toilet
time-based feature and a sensor-based feature { Pi , 1 , P1,2 } . parameters from the other activities that are analyzed as a
Feature Pi , 1 represents the duration of activity A, and P 1,2 group [25 ] , [26] .
represents the number of sensor events generated during 2 ) Mobility : Mobility is the ability of an individual to
activity A;. These two features are generalizable to other 10 move around the individual's home environment and the
activities , are easily interpretable, and can model how the community. Mobility impairments limit an individual's abil
residents perform their daily activities [ 15 ] . In addition to ity to maintain independence and quality of life and are
capturing activity performance, CAAB also represents and common predictors of institutionalization among older
monitors a person's overall mobility. Mobility refers to adults [ 27 ] . Evidence supports a close connection between
movement generated while performing varied activities (as 15 executive brain function and walking speed [ 28 ] . Therefore,
opposed to representing a single activity of its own) and is CAAB separately models mobility as an everyday behav
therefore represented using two different types of features: ioral feature. CAAB models the mobility of a smart home
the number of sensor events triggered throughout the home resident based on the number of sensor events they trigger
and the total distance that is covered by movement through- and the total distance they cover in a day while in the home
out the course of a single day ( Table 2 below ) . 20 (estimated based on known distances between motion sen
sors placed in the home) . As listed in Table 2 , the value for
TABLE 2 the distance -based mobility feature is calculated as the total
Group Activity Performance Features
distance covered by a resident in one day ( the example
Variable
aggregation time period ) while inside the home. Similarly,
Mobility Mobility Total distance traveled, # Total sensor 25 the value for the sensor -based mobility feature is calculated
Sleep Sleep
events as the number of sensor events that a resident triggers over
Sleep duration , # Sleep sensor
Bed toilet transition Bed toilet transition duration
events the course of one day while moving around in the home .
ADL Cook Cook Duration 3 ) Activities of Daily Living : Basic activities of daily
Eat Eat Duration living ( e.g. , eating , grooming) and the more complex instru
Relax Relax Duration 30 mental activities of daily living (IADLs ; e.g. , cooking ,
Personal Hygiene Personal Hygiene Duration managing finances) are fundamental to independent living.
Leave Home Leave Home Duration
Data indicate that increased difficulties in everyday activity
completion (e.g. , greater task inefficiencies, longer activity
C. Selection of ADLs completion times ) occur with older age [ 29 ] , [ 30 ] . Clinical
In an example CAAB , a subset of automatically - labeled 35 studies have also demonstrated that individuals diagnosed
resident daily activities is modeled . These activities are with MCI experience greater difficulties (e.g. , increased
sleep, bed to toilet ( a common type of sleep interruption ), omission errors) completing everyday activities when com
cook , eat , relax, and personal hygiene. The CAAB also pared with healthy controls [ 31 ] , [ 32 ] . Therefore , clinicians
captures and model a resident's total mobility in the home . argue the importance of understanding the course of func
1 ) Sleep : The effects of aging include changes in sleep 40 tional change given the potential implications for developing
patterns that may influence cognitive and functional status. methods for both prevention and early intervention [ 30 ] . The
For example, individuals over the age of 75 have been found example CAAB considers five activities of daily living (in
to experience greater fragmentation in nighttime sleep ( e.g. , addition to sleep ): cook, eat , personal hygiene, leave home,
[ 23 ] ) , which concurrently causes decreased total sleep time and relax. Of note the “ relax ” activity represents a combi
and sleep efficiency. Sleep problems in older adults can 45 nation of watching TV, reading , and napping that typically
affect cognitive abilities [ 24 ] and have been associated with takes place in a single location other than the bedroom where
decreased functional status and quality of life . Moreover, the resident spends time doing these activities , such as a
individuals with dementia often experience significant dis- favorite chair. The example CAAB focuses on these activi
ruption of the sleep wake cycle . Thus, the effects of sleep on ties because they are activities of daily living that are
the health of older adults are important clinical construct that 50 important for characterizing daily routines and assessing
both clinicians and caregivers are interested in understand- functional independence. For each of these activities , CAAB
ing [ 25 ] . calculates the total activity duration . Our data shows the
Using activity recognition, the CAAB recognizes sensor number of sensor events generated when performing these
events that correspond to sleep ( in the bedroom , as opposed activities is often very low . Thus, for these activities , fea
to naps taken outside the bedroom ) and bed - to -toilet activi- 55 tures were excluded that calculate number of times sensor
ties . The CAAB then extracts the time spent and number of events are triggered . As listed in Table 2 , CAAB calculates
sensor events features that correspond to these two activi- the value for the time -based ADL feature as the total number
ties . As listed in Table 2 , four features model a smart home of minutes spent in an activity on a particular day.
resident's sleep activity. The value for the time -based sleep D. Activity Feature Extraction
feature is calculated as the total number of minutes spent in 60 The second step of the CAAB tool after sensor event
sleep on a particular day and the value for the sensor-based annotation (e.g. , step 404 of FIG . 4 ) is to extract statistical
sleep feature is calculated as the number of sensor events features from the activity performance vector generated in
that are triggered over the course of one day while the the prior step ( e.g. , step 403 of FIG . 4 ) . CAAB extracts
resident slept . Similarly, the time -based bed to toilet feature features from the time series - based representation of activity
is calculated as the total number of minutes spent in bed to 65 performance and uses these to train a machine - learning
toilet activity on a particular day . An example CAAB algorithm (e.g. , step 405 of FIG . 4 ) . Namely, CAAB extracts
excludes the sensor based feature that calculates the number four standard time series features and one new change
US 10,896,756 B2
13 14
feature . These five features will be referred to as “ statistical
activity features.” Table 3 lists an example set of statistical Algorithm 1
activity features. In Table 3 , u is the mean of the activity ALGORITHM 1 Extracting statistical activity features
performance features p of size n . 1: Input : Activity performance features
5 2: Output: Statistical activity features
TABLE 3 3: Initialize : Feature vector
4: // T? and T , are two consecutive clinical testing time points
1 Variance Variance is the Var (p ) = Ex= 1 " (p ; – u )? 5: Given : T1 ; T2
6: Given : skip size 1
=
measure of spread . 7: while Tj < ( T2 - W ) do

10 8: for each activity performance feature do :
2 Auto- Auto 9: Place a window of size W at T? ;
correlation correlation (AC )
is the similarity
between
observations that
AC – lagl (p ) Mi ?
i= 1
( P ; - u ) (Pi + 1 – u )
–
(p ; - 42
10:
11 :
Remove missing observations and detrend based on
the observations that fall into this window ;
Calculate the variance, autocorrelation, skewness,
kurtosis and change features (Algorithm 2 ) using the
n=1
are displaced in 15 observations in the window ;
time . Calculate 12 : Append these values to the feature vector ;
autocorrelation at 13 : T2 = T1 + skip size ;
lag 1 . 14 : end foreach
15 : end while
16 : return average ( Feature matrix )
3 Skewness Skewness
measures the
degree of
asymmetry in the skewness ( p ) = =
?
1n
i= 1
( p; -1)* 20
In addition to these four different time series features,
CAAB uses a fifth feature, a change- based feature , to
distribution of 2 ( P; -2) º |
values . characterize the amount of change in an individual's activity
25
performance . Algorithm 2 details the steps in calculating this
4 Kurtosis Kurtosis measures new feature . In order to compute this feature, CAAB uses a
the amount of
peakedness of the kurtosis( p )
distribution toward
? i= 1
(p: -4) sliding window of size w days and divides the activity
performance feature observations that fall in W into two
the mean . C. ((P9;:---42 30
different groups. The first group contains feature observa
tions that fall in the first half of Wand second group contains
5 Change Change See Algorithm 1 feature observations that fall in the other half. CAAB then
characterizes the compares between these two groups of feature observations
amount of change using a change detection algorithm . For example, in one
in an individual's example CAAB the Hotelling - T test algorithm [ 34 ] is used .
activity perfor 35 However, different CAAB implementations can also apply
mance over time . other change detection algorithms. CAAB then slides the
window by one day ( skip size= 1 ) and re -computes the
change feature . CAAB calculates the final change value as
1 ) Statistical activity features : To calculate the first four the average over all windows. Similar to the other four
features, CAAB runs a sliding window (e.g. , window size , 40 statistical activity features computed , CAAB computes the
W= 30 days ) over each of the activity performance features value of the change feature for each of the activity perfor
listed in Table 2 and calculates the statistical activity features mance features listed in Table 2 .
( e.g. , variance, autocorrelation, skewness, and kurtosis )
using the observations from data that falls within the sliding Algorithm 2
window . The sliding window starts at one clinical assess- 45 ALGORITHM 2 Calculation of change feature
ment time point and ends at the next assessment time point,
thus capturing all of the behavior data that occurred between 1:
2:
Input : Activity performance features
Initialize : CH = [ ]
two subsequent assessments. For example, CAAB can cal 3: // T , and T2 are two consecutive clinical testing time points
culate the variance, autocorrelation, skewness , and kurtosis 4: Given : T? ; T2
of the duration feature for each activity based on duration 50 5: Given : skip size = 1
observations that fall inside each W sized data window . 6: while T , < ( T2 - W ) do
CAAB repeats the process and calculates these four statis 7: for each activity performance feature do :
8: Place a window of size W at Ti ;
tical activity features for all other activity performance 9: Remove missing values that fall into this window ;
features for all of the activities in set A. Before calculating 10 : Put first half of W in the group A and second half in
the group B ;
these features , CAAB first removes the time series trend 55 11 : // Returns True or False ;
from the sliding window observations in order to remove the 12 : change = Hotelling T -test ( A , B ) ;
effect of non - stationary components (e.g. periodic compo 13 :
13 :
append ( CH , change )
T1 = T1 + skip size ;
nents ) in the time series [ 33 ] . For this step , CAAB fits a 14 : end foreach ;
Gaussian or a linear trend to the data within the sliding 60 15 : end while ;
window . CAAB then detrends the data by subtracting the 16 : return average (CH ).
fitted trend from the data . CAAB slides the window by one
day ( skip size = 1 ) and re - computes all of the statistical Of note , the change feature is different from the variance
activity features. For each feature , CAAB slides a window feature that CAAB calculates earlier. While variance mea
through the sensor home data and computes the final feature 65 sures the variability of samples around its mean , the change
values as an average over all of the windows . Algorithm 1 feature empirically calculates the “ chance ” of observing a
can be utilized to compute these steps . change when two sample groups each of size n from the
US 10,896,756 B2
15 16
given activity performance features are compared with each Modeling and Evaluating Patterns of Activities Using Activ
other. Here , a higher amount of detected change indicates a ity Curve Tools
greater chance of detecting changes in the activity perfor- Many pervasive computing applications ( also known as
mance feature . ubiquitous computing applications) such as home automa
E. Clinical Assessment 5 tion , activity aware interventions, and health assessment
In the final steps of the CAAB tool ( e.g. , steps 405-407 of require analyzing and understanding activity -based behav
FIG . 4 ) , CAAB predicts the clinical assessment scores of the ioral patterns. The performance of such applications
residents being monitored using the smart home sensors depends on the ability to correctly learn a model of general
using the activity performance features computed from the 10 daily activity behavior from a large amount of data and be
able to predict when such daily behavior is likely to continue
activity labeled sensor data . CAAB first aligns the sensor or change. These big data -based approaches to activity
based data collection date with the clinical assessment -based
data collection date before extracting statistical activity modeling activity -
can then in turn be used to provide effective
aware services such as improved healthcare.
features (in step 404 of FIG . 4 ) . After extracting features and Activity recognition lies at the heart of any pervasive
aligning
learning the data , CAAB
algorithm then trainsthea supervised
and predicts machine 15 computing approach to modeling behavioral routines. As
clinical assessment described above , an activity recognition algorithm maps a
scores ( step 405 of FIG . 4 ) . To accomplish this goal , CAAB sensor reading or sequence of readings to a corresponding
extracts statistical activity features from the activity perfor activity label . In order to answer general questions related to
mance features that lie between any given two consecutive daily activity patterns, such information needs to be trans
clinical testing points, t1 and t2 . Similarly, it obtains the 20 formed to a higher - level representation . For example, ques
clinical score S2 (or S1 ) at time point t2 (or t1 ) . It considers tions such as how average daily activity patterns have
the pair, statistical activity features and clinical score S2 , as changed over a year, or generally what hours did a particular
a point in the dataset and repeats the process for each of individual sleep last month are difficult to answer using raw
resident that it is monitoring and for every pair of the output from activity recognition algorithms. However, many
consecutive clinical testing points. Algorithm 3 summarizes 25 pervasive computing applications such as home automation
the steps involved to prepare the dataset to train the super- and health assessment require answering such questions .
vised machine learning algorithm . Obtaining higher -level representations or models of
activities has several additional advantages . Higher -level
representations can abstract variations in day - to - day activity
Algorithm 3 30 routines. For example, wake -up times in the morning may be
ALGORITHM 3 Training Set Creation slightly different each day even if the overall routine is fairly
1: Input: Activity performance features for all residents stable. Additionally, such representations simplify the task
2: Output: Training set to train the learning algorithm of modeling an individual's daily routine and at the same
3: Initialize: Empty training set TrSet time make visualization and interpretation of daily activity
4:
5:
for each resident do :
for each consecutive clinical testing point T , and T2 do :
35 routines easy. Collecting big datasets over long periods of
6: F = CAAB ( statistical activity performance time allows us to abstract activity models over such daily
features between T? and T2); variations. Such representations aid with the process of
7: S = clinical score (T1 , T2) and T2 ; identifying long - term changes in a behavioral routine .
8:
9:
Append (F, S , TrSet );
end for each ;
For example, consider the following description high
10 : end for each . 40 lighting aspects of an individual's routine at two different
points in time :
Month of March 2012 : Sleep at 10:00 PM , get up at 6:00
The CAAB is then able to predict the clinical assessment AM , eat breakfast at 7:00 AM , eat lunch at 12:00 PM ,
scores ( see step 406 of FIG . 4 ) using the machine learning go out for a walk at 4:00 PM , and dine at 8:00 PM .
algorithm (tool , system , etc. ) . As explained above, CAAB 45 Month of September 2013 : Sleep at 8:00 PM , wake up
trains a learning algorithm to learn a relationship between frequently during the night, get up at 10:00 AM , no
statistical activity features and the clinical assessment scores breakfast, eat lunch at 11 AM , no going out for a walk ,
using the dataset that is initially constructed . ( Block 405 of and dine at 7:00 PM .
FIG . 4 shown in dashed lines as it is only performed initially Note that each of these sample activity - based descriptions
or when the machine learning tool is updated ). Then in the 50 is aggregated over a one -month period and therefore
final step ( block 406 of FIG . 4 ) for each resident, at each describes a general routine that is maintained over a pro
time point (except the first one ) , CAAB predicts the clinical longed period of time . Based on these descriptions we also
assessment scores using a machine learning algorithm such note changes in the routine from the first observation to the
as an SVM ( support vector machine ) . The machine learning second . From this example, we can infer that by September
algorithm may be part of the CAAB , a separate computing 55 2013 the observed individual was experiencing disturbances
system , a call , for example, to an algorithm provided by a in sleep , was skipping meals , and stopped exercising. Deter
software library of functions, or the like . mining if the overall daily activity patterns has changed may
Of note the CAAB predicts clinical assessment scores be difficult based only on the raw sensor data or even based
based on the relationship that the learning algorithm models on event -by -event labels from an activity recognition algo
between the clinical assessment scores and behavior fea- 60 rithm . Such questions can be more easily answered by
tures. This approach is followed because there are very few comparing two higher - level representations of these activity
clinical observations for a resident. Furthermore , the CAAB patterns.
computes activity performance features by temporally fol- Example embodiments provide an Activity Curve tool ,
lowing an individual over a period and computes statistical such as the Activity Curve Change Engine 307 of FIG . 3 to
activity features by comparing past observations with cur- 65 model an individual's generalized daily activity routines .
rent observations. In this way, CAAB uses an individual as The activity curve modeling algorithm ( for example, as
their own baseline for predictive assessment. implemented by code logic ) employed by the Activity Curve
US 10,896,756 B2
17 18
tool uses activity - labeled sensor events to learn a higher- breakfast at 7:15 AM one day while we might wake up at
level representation of the individual's regular routine. 7:30 AM and eat breakfast at 8:00 AM the next day. In order
These activity labels are automatically -recognized using an to generalize the activity curve model over such day - to -day
activity recognition algorithm . variations in activity routines, we define the notion of an
The Activity Curve tools use a Permutation -based Change 5 aggregated activity curve that is calculated over an aggre
detection in Activity Routine ( PCAR) algorithm as imple gation window of x days.
mented for example by code logic to compare activity Definition 1 .
curves between different time points in order to detect Given a time interval t , an activity distribution models the
changes in an activity routine . The tools and algorithms have daily routine based on the predefined set of activities A as a
been validated using longitudinal smart home sensor data 10 probability distribution over activities in A. The probability
collected by monitoring everyday behavior of residents over distribution can be estimated
two years. The activity curve modeling algorithm and the normalized time an individualfrom sample data based on the
spends on a predefined set of
PCAR algorithm can be used to perform important pervasive n activities during time intervals t as observed during one or
computing tasks such as automated assessment of an indi more days .
vidual's functional health . 15
The activity curve modeling algorithm implemented by An activity distribution for time interval t is a n - element
the Activity Curve tool provides a generic activity curve set Di= {de,1, d.,2 , ... , d.,n } whose length is equal to that of
model that can be calculated both from the output of an the activity set A. The i -th element in an activity distribution,
activity recognition algorithm as well as using other algo- die represents the probability of performing activity A¡
rithms from prior studies . The activity curve model facili- 20 during time interval t .
tates answ swering more complex questions related to activity To model a person's overall daily activity routine, the AC
routines such as whether changes in an activity routine have tool uses m activity distributions corresponding to each of
occurred or not . In an example embodiment, activity curve the m time intervals . The AC tool can then construct an
tool uses the output from an activity recognition algorithm activity curve by collecting activity distributions that model
to calculate the activity curves . 25 daily activity patterns at all different times of the day.
The PCAR algorithm detects significant changes in activ- Definition 2 .
ity curves based on a permutation -based two -sample test
using symmetric Kullback -Leibler divergence as a distance butionsactivity
An curve C is the compilation of activity distri
D , ordered by time interval t .
metric . The Activity Curve (AC ) tool for detecting changes The length of an activity curve is m . The model that
in activity routines has some similarities with activity track- 30 compiles activity distributions is referred to as an “ activity
ing algorithms that have been previously proposed [ 52 ] , curve,” because
[ 53 ] . However, these previously proposed activity -tracking activity A , for alliftime
we consider the activity distribution of
intervals 1 , 2 , ... , m , these activity
algorithms discover and track changes in sensor sequence distributions form a curve that represents the “ fraction of a
patterns for the purpose of adapting home automation strat
egies. In contrast, the Activity Curve tool tracks changes in 35 time ” that an individual is likely to perform activity A ; over
successive time intervals .
the distribution of automatically - recognized activity pat The AC tool calculates an aggregated activity distribution
terns.
Activity Curve Ô, for time interval t by aggregating activity distributions
An activity curve is a model that represents an individu- Dk, ( 1sksx ) over an aggregation window of x days . If D1,69
al’s generalized activity routine . The AC tool models activ- 40 D2,19 • . . D. are activity distributions for the t - th time
ity routines for a day -long period but the time period can be interval aggregated over a window of x days and follow
changed as needed . The activity curve uses automatically normal distributions, then we can define an aggregated
recognized activity labels to express daily behavioral char- activity distribution as follows.
acteristics based on the timing of recognized activities . Definition 3 .
Typically, a continuous sequence of time - stamped sensor 45 An aggregated activity distribution Ô , at time interval t is
events is available . the maximum likelihood estimate of the mean that is
In summary , the AC tool first uses an activity recognition obtained from activity distributions Dkt ( 1 sksx) that fall
algorithm to annotate each of these sensor events with an within an aggregation window of size x .
activity label . Activity recognition algorithms map a We can write the aggregated activity distribution , at
sequence of sensor events { e , e2 , ... , en } onto the 50 time interval t as shown in Eq . ( 1 ) :
corresponding activity label A? , where the label is drawn
from the predefined set of activity classes A = { A , A2 ,
An }. (1 )
We note that prevalence of common activities differs by ÔDA-???? . k =1
the time of day. For example, the “ sleep ” activity dominates 55
the prevalent distribution of activities at midnight and the
" cook breakfast” and “ eat breakfast activities dominate the Definition 4 .
early morning hours. To capture such differences in activity An aggregated activity curve is the compilation of aggre
patterns throughout the day, the AC tool segments the gated activity distributions obtained over an aggregation
day - long observation period into m equal-size consecutive 60 window of size x .
windows, or time intervals, and define probability distribu- If E = {C1, C2 , C3 , ... , Cx } is a set of activity curves over
tions over activities , or activity distributions, for each of an aggregation window of size = x days, we can represent an
these time intervals ( see FIG . 1 for an example ). An activityaggregated activity curve over { as C ?. The aggregated
curve is a compilation of these activity distributions for the activity curve C2 compiles the aggregated activity distribu
entire day - long period . 65 tions , Dr. FIG . 6 illustrates an example of an aggregated
In general, activity routines tend to vary from one day to activity curve that models three different activities : sleep ,
the next. For example, we may wake up at 6:30 AM and eat bed toilet transition, and other.
US 10,896,756 B2
19 20
Activity Distribution Distance particular time interval. A minimum aggregate window size
We calculate the distance between two activity distribu- Xmin is chosen such that no smaller window would ensure the
tions using the Kullback -Leibler ( KL) divergence measure . stability criterion .
We assume that the activity distributions model the same To determine the ideal aggregation window size , the
activity set A for the same time interval size and aggregation 5 algorithm starts with a window of size x =2 and considers the
window size . The KL divergence between two activity corresponding aggregated activity curve C ?, aggregated
distributions from the set of individual activity curves > = { C1 , Cx }.
D = { d1,1,01,2 , ...die ... , d1n } and D2= {d2,1, 02,29 - . We estimate the distance between C and Cx+ 1 ?. If the
d2.j ... , d2n } is defined as shown in Eq . ( 2 ) : distance is greater than a predefined threshold T, we increase
10 the window size . Therefore, if SDK (C $||C ++representa 25 < T and
SDK (Cx + 1S ||Cx + 2 ? ) < T, then x is selected as the
n ( 2) tive aggregation window size , otherwise the size of the
DkL ( D1 | D2 ) = \ du,jlog3dii
dzi aggregation window is increased by one and the process
i= 1
15
repeated. This process is shown in Algorithm 4 .
We note that the standard KL distance metric is a non Algorithm 4
symmetric measure of the differences between two prob ALGORITHM 4 Aggregation Size ( E , T )
ability distributions D , and D2 . Therefore , we use a sym
metric version of the Kullback - Leibler divergence between 20
1: Il Calculate the minimum size of an aggregation window
activity distributions D , and D2 , which is defined as shown 2: E = { C1 , C2 , ... , Cn } for each of the N days in the input data.
3: // Return the minimum aggregation window size .
in Eq . (3 ) . Throughout the remainder of the paper, our 4: initialize x 2 ;
discussion of KL divergence will refer to this symmetric 5:
6:
repeat :
Create C?, aggregated activity curve for window size x ;
version of the KL divergence measure . 7: Create Cx+ 1 , aggregated activity curve for window size x + 1 ;
25 8: Create Cx+ 2 , aggregated activity curve for window size x + 2 ;
SDKL (D1||D2 ) =Dkz (Di ||D2 ) + (D2||D ) (3) 9: Compute di distance between two aggregated activity
curves SDK (CE11 CX 5);
Before defining the distance between two activity curves 10 : Compute d2 = distance between two aggregated activity
C and C2 of length m , we first need to align the activity curves SDK (Cx+ 1 | Cx+25);
distributions in the activity curves (as described later ). As a 12 : If d? > T and dz > T , then x = x + 1 ;
30 13 : else return x ;
result of the alignment step , we obtain a vector of alignment 14 : until x < N ;
pairs I= (p , q ) of length 1= 1 that aligns an activity distri
bution at time interval p ( 1spsm) of activity curve C1 with Activity Curve Alignment
activity distribution at time interval q ( 1sqsm) of activity In order to compute similarity (or distance ) between two
curve C2 . We calculate the total distance , SDK ( C ||C2),
between two activity curves, C , and C2 , as the sum of 35 activity curves, we need to compare each of the activity
distributions that belong to these two activity curves . How
distances between each aligned activity distribution for the
two activity curves , as shown in Eq . (4 ) . ever , we first need to determine which pairs of distributions
to compare by considering alternative distribution alignment
techniques. Activity curve alignment can be performed
( 4) 40 based on aligning the same time of day between two curves .
SD«z«C ||C)= ? SDKL (D1,p||D2,g)
a= 1
Alternatively, we can try to maximally align the activity
occurrences between two curves before performing such a
such that la = ( p , q) comparison . Here we provide details for these two align
ment techniques.
45 Time Interval -Based Activity Curve Alignment
where Dip and D1 , g are the activity distributions that belong The time interval -based activity curve alignment tech
to activity curves C and C2 at time intervals p and q, nique presumes that distributions between two curves should
respectively. be aligned based on time of day and thus aligns activity
Determining the Size of an Aggregation Window distributions between two activity curves if the time inter
Daily activity routines are performed differently from one 50 vals are the same . In essence , this method does not make any
day to the next. As a result , the daily activity curve that extra effort to align activities that occur at different times in
models these activity routines will vary from one day to the the distribution , but simply compares the activity distribu
next. The activity curve tools want to calculate an aggre- tions based on time of day alone . If C , and C2 are two
gated activity curve that generalizes over minor day - to - day activity curves of length m , the time interval-based activity
variations while still capturing the typical routine behavior. 55 distribution alignment method aligns the corresponding
When determining the appropriate size of an aggregation activity distributions using a vector of alignment pairs , T = ( r,
window , the goal is to find the smallest possible number of r ). This technique aligns an activity distribution at time
days that is considered stable . The tool determine that an interval r ( 1srsm) of activity curve C , with activity distri
aggregated activity curve is stable if the shape of the curve bution at time interval r ( 1 srsm ) of activity curve C2 .
remains mostly unchanged when more days are added to the 60 Dynamic Time Warping -Based Activity Curve Alignment
aggregation window . By keeping the aggregation window A person's routine may be relatively stable , even though
small , the model can be more sensitive to significant changes there are minute changes in the time an activity occurs or the
in routine behavior. If the window is too small it will not be duration of a particular activity. For example, an individual
general enough to encompass normal variations in daily may sleep at 10:00 PM one day, an hour earlier at 9:00 PM
routines. Algorithm 1 is used to determine the minimum 65 the next day, an hour later at 11:00 PM a few days later, and
length of an aggregate window that is required to calculate eventually go back to sleeping at 10:00 PM . Aligning
a stable, representative aggregated activity curve for a activity distributions using dynamic time warping allows the
US 10,896,756 B2
21 22
tool to maximally align common activities before comparing Permutation -Based Two - Sample Test
two activity curves . Such an alignment accommodates activ- PCAR identifies significant changes in an activity routine
ity time changes that are shifted temporally backward ( for using a two -sample permutation test [ 64 ] . The permutation
example, an hour earlier ), forward ( for example, an hour based technique provides a data - driven approach to calculate
later ), expanded (longer duration ), compressed ( shorter 5 an empirical distribution of a test statistic . The empirical
duration ), or not changed at all from one day to another. The distribution of a test statistic is obtained by calculating the
tool optimizes activity alignment using Dynamic Time test statistic after randomly shuffling ( rearranging) the data
Warping (DTW ) to align distributions between two activity a specified number of times . The permutation -based test is
curves . Dynamic time warping finds an optimal alignment or exact if the joint distributions of rearranged samples are the
warping path between activity curves . This optimal warping 10 same as the joint distribution of the original samples. In
path has minimal total cost among all possible warping other words, the samples are exchangeable when the null
paths. The tool uses the symmetric KL distance metric that hypothesis is true. This type of test allows us to determine
we previously mentioned to compute this warping path . The the significant difference between two aggregated activity
warping path has the following three main properties : curves . We use a permutation -based test to perform a two
Boundary property: The first and last elements (activity 15 sample homogeneity test . In a two -sample homogeneity test ,
distributions ) from the two activity curves are always we test the null hypothesis that the two samples come from
aligned with each other. the same probability distribution versus the alternate hypoth
Monotonicity property : Paths are not allowed to move esis that they come from different probability distributions .
backwards. Changes in Activity Distributions
Step size property : No activity distributions are omitted 20 We use the permutation -based two - sample test to deter
from the curve alignment. mine whether there is a significant change among a set of
We also note that due to the monotonicity property , DTW activity distributions at a particular time interval. We for
does not allow backward alignments. However, as we have mulate the null hypothesis that the set of activity distribu
seen in practice , activity distributions can be shifted tem- tions comprising two activity curves are identical versus the
porally backward and /or temporally forward . Therefore, the 25 alternative hypothesis that the set of activity distributions is
standard approach is modified to perform two independent significantly different between the two aggregated activity
iterations of DTW : curves . We test the hypothesis of a significant change
In forward dynamic time warping, we start from the first between two aggregated activity distributions, Ô 1.4 and D2.
activity distribution and move forward in time toward Calculate the test statistic : Calculate the test statistic
the last activity distribution to find an optimal align- 30 Dist = SDK ( D2 ) between the two aggregated
ment between activity curves that are similar in the activity distributions.
forward time direction . Permutation : Randomly shuffle individual activity distri
In backward dynamic time warping, we start from the last butions between the two aggregated distributions and
activity distribution and move backward in time toward recalculate the aggregated distributions 1.c and Ô 2.c
the first activity distribution to find an optimal align- 35 Calculate the KL divergence between the new aggre
ment between activity curves that are similar in the gated activity distributions Dist = SDXL DD2( .)).
backward time direction . Repeat the process a specified number of times to
If C? and C2 are two activity curves of length m , the obtain an empirical distribution of KL divergence ( the
DTW -based activity distribution alignment outputs two test statistic ), Dist,
alignment vectors , Iforward (u , v) of length 1toward? and 40 Significance testing: To test if a significant difference
I backward (r, s) of length lbackward, respectively. The forward
DTW aligns an activity distribution from curve C , at time
exists between Du, and D2 ,, calculate the p -value by
calculating the number of times the test statistic from
interval u ( 1 susm) with an activity distribution from curve the permuted sample is equal to or greater than the
C2 at time interval v ( 1svsm) . Similarly, the backward DTW original test statistic Dist ,, in the empirical distribution
aligns an activity distribution from curve C , at time interval 45 Dist . If a small p - value is obtained , reject the null
r ( 1srsm) with an activity distribution from curve Cz at time hypothesis in favor of alternative hypothesis. This is
interval s ( 1 sssm) . The DTW method outputs whichever shown in Eq. (5 ).
vector, Tforward or T backwards that results in the maximal
alignment between the two distributions and thus minimizes
the difference . The activity curve tool will utilize these two 50 (5 )
different alignment techniques in the PCAR algorithm to Pperm ##Dist, > Dist:
permutations
detect changes between two aggregated activity curves and
calculate change scores.
PCAR where Dist, is the empirical distribution of the test statistic
Based on this notion of an activity curve , we now intro- 55 at the t - th time interval. We reject the null hypothesis
duce our Permutation - based Change Detection in Activity that no changes have occurred at a significance level of
Routine ( PCAR ) algorithm . This algorithm , which is used a =0.01 .
by the activity curve tool , identifies and quantifies changes Changes in Activity Curves
in an activity routine. PCAR operates on the assumption that We now extend the technique of detecting significant
daily activities are scheduled according to a routine and are 60 changes between activity distributions to quantify the dif
not scheduled randomly. For example , we regularly “ wake ference in activity routine observed from two separate
up ” , “ bathe” and “ have breakfast " in the morning and “ dine” aggregation windows, W , and W2 , each of size x days. To do
and “ relax ” in the evening . In contrast, we rarely dine in the this, PCAR counts the total number of significant differences
middle of night. Such regularities are useful, for example, to between the individual activity distributions within window
determine if there are significant changes in lifestyle behav- 65 W and the distributions within window W2 to output a
ior that might indicate changes in cognitive or physical change score that quantifies the significant changes observed
health . among the activity curves . FIG . 7 demonstrates the three
US 10,896,756 B2
23 24
main steps involved in detecting changes in an activity curve -continued
using the permutation -based method .
PCAR calculates a sum , S , over changes that are detected Algorithm 6
between activity curve distributions for each individual time ALGORITHM 6 PCAR (E1 , E2N )
interval. In order to identify the time intervals at which 5 10 : S = S + 1;
changes in the activity distributions comprising the aggre- 11 : end for;
12 : return S ;
gated activity curves are deemed significant, PCAR per
forms the following steps :
Permutation : Calculate the empirical distributions of the We note that we compare at least m activity distributions
test statistic ( KL divergence) by permuting and com- 10 during this process where m is the number of aggregated
paring the individual activity distributions within the activity distributions in an activity curve. To control the
two aggregation windows W , and W , at each time False Discovery Rate ( FDR) at level a * ( a * =0.01 ) , we apply
interval using the method summarized in Algorithm 5 . the Benjamini-Hochberg (BH ) method [ 65 ] . The BH method
Alignment: Calculate the two aggregated activity curves first orders the p -values, P1 , P2 , , Pks Pm , in ascending
C? and C2 using the activity distributions aggregated 15 order and for a given value of a * , the BH method finds the
for each time interval over windows W and W2 . Align largest k such that
curves C and C , using one of the alignment techniques
described in the previous section to generate the align
ment vector I = ( u , v ) . 20
a*
Calculate the test statistic : For each alignment pair u , p(k) skx
V ) -T , calculate the test statistic Disty , between the
aggregated activity distribution at time interval u of The BH algorithm rejects the null hypothesis corresponding
activity curve C , and the aggregated activity distribu to Pi if isk . If a significant change is detected between
tion at time interval v of C2 .
Significance testing : To test if there is a significant change 25 aligned activity distributions, PCAR increments its change
between activity distributions at time intervals u and v , score , S , by one . PCAR generates two different change
calculate the p -value based on the number of times the scores based on the alignment techniques that are employed:
test statistic from the permuted sample is equal to or either the same index alignment or the DTW -based align
greater than the original test statistic . The steps are 30 ment .
Example Computing System
summarized in Algorithm 6 .
FIG . 8 is an example block diagram of an example
computing system that may be used to practice embodiments
Algorithm 5 of a Cognitive Assessment Facilitator System described
ALGORITHM 5 Empirical Distribution (E1, E2 · N ) herein . Note that one or more general purpose virtual or
1 : // Build empirical distribution of the test statistic . 35 physical computing systems suitably instructed or a special
2 : 21 , 2 - two sets of activity curves purpose computing system may be used to implement an
3 : N , - number of permutations
4 : Initialize Dist , as N , x m matrix;
OAFS . Further, the OAFS may be implemented in software ,
// m is # activity distributions in the activity curves hardware, firmware, or in some combination to achieve the
5 initialize i 0 ; capabilities described herein .
6 : while i < N do : 40 The computing system 800 may comprise one or more
7:
8:
Shuffle the activity curves;
Generate aggregated activity curves CE1 and CE2 by
server and / or client computing systems and may span dis
aggregating the distributions in 21 , 22 ; tributed locations. In addition , each block shown may rep
:
9 Using the time interval-based alignment technique, align the resent one or more such blocks as appropriate to a specific
two aggregated activity curves to obtain an alignment vector T ; embodiment or may be combined with other blocks . More
10 : for all alignment pairs (u , u) in I do : 45 over , the various blocks of the Cognitive Assessment Facili
11 Find a distance SDK ( D 1,24
distributions in two activity curves ;
| D2u) between uth activity tator System 810 may physically reside on one or more
12 : Insert SDKL ( D 1,44 | D2 , u ) to empirical distribution machines , which use standard (e.g. , TCP/ IP ) or proprietary
Dist at location [i, u) ; interprocess communication mechanisms to communicate
13 : end for; with each other.
14 : i - i+ 1 ; 50 In the embodiment shown, computer system 800 com
15 : end while ; prises a computer memory ( “memory ” ) 801 , a display 802 ,
16 : return Dist one or more Central Processing Units (“ CPU ” ) 803 , Input/
Output devices 804 (e.g. , keyboard , mouse , CRT or LCD
display, etc. ) , other computer - readable media 805 , and one
55 or more network connections 806. The OAFS 810 is shown
Algorithm 6 residing in memory 801. In other embodiments, some por
ALGORITHM 6 PCAR (21 , 22 Np) tion of the contents, some of, or all of the components of the
1: E1 , E2 - two sets of activity curves OAFS 810 may be stored on and /or transmitted over the
2:
3:
// Return a change score S
C1 - Aggregate ActivityCurves (21) other computer -readable media 805. The components of the
4: C2 - Aggregate ActivityCurves (22) 60 Cognitive Assessment Facilitator System 810 preferably
5: 1 - AlignCurves ( C1 , C2 ) execute on one or more CPUs 803 and manage the genera
6: for all alignment pairs (u , v ) in I do : tion and use of predictive cognitive assessments and func
7: Calculate SDK ( D1,2 | D2 , v) between activity distribution tional assessments, as described herein . Other code or pro
D1, EC and D2 , E Cz ; grams 830 and potentially other data repositories, such as
8: Perform significance testing of estimated distance by querying
Dist; 65 data repository 806 , also reside in the memory 801 , and
9: if change is significant: preferably execute on one or more CPUs 803. Of note , one
or more of the components in FIG . 8 may not be present in
US 10,896,756 B2
25 26
any specific implementation. For example , some embodi- be distributed, pooled or otherwise grouped, such as for load
ments embedded in other software may not provide means balancing , reliability or security reasons . A variety of dis
for user input or display. tributed computing techniques are appropriate for imple
In a typical embodiment, the OAFS 810 includes one or menting the components of the illustrated embodiments in a
more smart -home sensor input logic 811 , one or more 5 distributed manner including but not limited to TCP / IP
CAAB logic components 812 , and one or more activity sockets, RPC , RMI , HTTP, Web Services (XML -RPC , JAX
curve change logic components (or engines) 813 , and one or RPC , SOAP, etc.) and the like . Other variations are possible .
more machine learning engines 814 that operate as described Also , other functionality could be provided by each com
herein . In at least some embodiments, the machine learning ponent
logic is provided external to the OAFS such as machine 10 amongst/module , or existing functionality could be distributed
learning tools 855 and is available, potentially , over one or achieve the functions of /an
the components modules in different ways, yet still
OAFS .
more networks 850. Other and / or different modules may be
implemented. In addition , the OAFS may interact via a components of the OAFS 810 may ,besome
Furthermore , in some embodiments or all of the
implemented or
network 850 with application or client code 860 that uses
results computed by the OAFS 810, and /or one or more 15 provided in other manners, such as at least partially in
firmware and / or hardware , including , but not limited to one
third -party information provider systems 865 , such as smart or
home sensor data providers.Also , of note, the OAFS data standard more application - specific integrated circuits ( ASICs ) ,
repository 816 , which may include sensor CAAB , and integrated circuits, controllers executing appropri
change curve data, may be provided external to the OAFS as ate instructions, and including microcontrollers and / or
well , for example in a knowledge base accessible over one 20 GAs
or more networks 850 .
embedded controllers, field -programmable gate arrays (FP
), complex programmable logic devices ( CPLDs ) , and
In an example embodiment, components /modules of the the like . Some or all of the system components and / or data
OAFS 810 are implemented using standard programming structures may also be stored as contents (e.g. , executable
techniques. For example, the OAFS 810 may be imple- or other machine -readable software instructions or struc
mented as a “ native ” executable running on the CPU 103 , 25 tured data ) on a computer -readable medium ( e.g. , a hard
along with one or more static or dynamic libraries . In other disk ; memory ; network ; other computer - readable medium ;
embodiments, the OAFS 810 may be implemented as or other portable media article to be read by an appropriate
instructions processed by a virtual machine . In general, a drive
range of programming languages known in the art may be flash ormemory via an appropriate connection, such as a DVD or
device ) to enable the computer - readable
employed for implementing such example embodiments , 30 medium to execute
including representative implementations of various pro to perform at least some or otherwise use or provide the contents
gramming language paradigms, including but not limited to , or all of the components and of the described techniques . Some
/ or data structures may be stored
object -oriented, functional, procedural, scripting, and on tangible, non - transitory storage
declarative . mediums. Some or all of
The embodiments described above may also use well- 35 stored as data signals (e.g. , by beingstructures
the system components and data may also be
known or proprietary, synchronous or asynchronous client carrier wave or included as part of an analogas orpartdigital
encoded of a
server computing techniques. Also , the various components
may be implemented using more monolithic programming propagated signal) on a variety of computer- readable trans
techniques, for example , as an executable running on a mission mediums , which are then transmitted, including
single CPU computer system , or alternatively decomposed 40 across wireless -based and wired /cable -based mediums , and
using a variety of structuring techniques known in the art, may take a variety of forms ( e.g. , as part of a single or
including but not limited to , multiprogramming, multi- multiplexed analog signal, or as multiple discrete digital
threading, client- server, or peer - to -peer, running on one or packets or frames . Such computer program products may
more computer systems each having one or more CPUs. also take other forms in other embodiments . Accordingly,
Some embodiments may execute concurrently and asyn- 45 embodiments of this disclosure may be practiced with other
chronously and communicate using message passing tech- computer system configurations.
niques . Equivalent synchronous embodiments are also sup- All of the above U.S. patents, U.S. patent application
ported . publications, U.S. patent applications, foreign patents, for
In addition, programming interfaces to the data stored as eign patent applications and non -patent publications referred
part of the OAFS 810 (e.g. , in the data repositories 816) can 50 to in this specification and /or listed in the Application Data
be available by standard mechanisms such as through C , Sheet, including but not limited to U.S. Provisional Patent
C ++ , C # , and Java APIs ; libraries for accessing files, data
bases , or other data repositories ; through scripting languages Application
TERNS OF
No. 62 / 150,794 , entitled “MODELING PAT
ACTIVITIES USING ACTIVITY CURVES,"
such as XML ; or through Web servers, FTP servers , or other filed Apr. 21 , 2015 , are incorporated herein by reference in
types of servers providing access to stored data. The data 55 its entirety.
repositories may be implemented as one or more database From the foregoing it will be appreciated that, although
systems , file systems, or any other technique for storing such
information, or any combination of the above, including specific embodiments have been described herein for pur
implementations using distributed computing techniques. poses of illustration , various modifications may be made
Also the example OAFS 810 may be implemented in a 60 without deviating from the spirit and scope of the invention .
distributed environment comprising multiple, even hetero For example , the methods and systems for performing
geneous, computer systems and networks. Different con- facilitating cognitive assessment discussed herein are appli
figurations and locations of programs and data are contem- cable to other architectures other than a client- server archi
plated for use with techniques of described herein . In tecture . Also , the methods and systems discussed herein are
addition , the [ server and /or client] may be physical or virtual 65 applicable to differing protocols, communication media (op
computing systems and may reside on the same physical tical , wireless , cable , etc.) and devices ( such as wireless
system . Also , one or more of the modules may themselves handsets, electronic organizers, personal digital assistants,
US 10,896,756 B2
27 28
portable email machines , game machines , pagers, navigation Technologies for Healthcare ( PervasiveHealth ), 2013 7th
devices such as GPS receivers , embedded systems, etc.). International Conference on , Venice, Italy, 2013 , pp .
97-100 .
REFERENCES [ 14 ] N. Noury, M. Berenguer, H. Teyssier, M.-J. Bouzid , and
5 M. Giordani, “ Building an index of activity of inhabitants
[ 1 ] M. Schmitter - Edgecombe, C. Parsey, and R. Lamb , from their activity on the residential electrical power
“ Development and sychometric properties of the instru line . ” IEEE transactions on information technology in
mental activities of daily living : compensation cale." biomedicine : a publication of the IEEE Engineering in
Archives of clinical neuropsychology: Journal of the Medicine and Biology Society , vol . 15 , no . 5 , pp . 758-66 ,
National Academy of Neuropsychologists, vol . 29 , no . 8 , 10 September 2011
pp . 776-92 , December 2014 . [ 15 ] P. Dawadi, D. Cook , and M. Schmitter -Edgecombe,
[ 2 ] Y. Ouchi , K. Akanuma, M. Meguro , M. Kasai , H. Ishii , “ Automated cognitive health assessment using smart
and K. Meguro, “ Impaired instrumental activities of daily home smart monitoring of complex tasks. ” IEEE Trans
living affect conversion from mild cognitive impairment actions on Systems , Man , and Cybernetics: Systems , vol .
to dementia : the Osaki- Tajiri Project. ” Psychogeriatrics, 15 43 , no . 6 , pp . 1302-1313 , 2013 .
vol . 12 , no . 1 , pp . 34-42 , March 2012 . [ 16 ] D. J. Cook and N. C. Krishnan , Activity Learning:
[ 3 ] N. Chaytor, M. Schmitter - Edgecombe, and R. Burr, Discovering, Recognizing , and Predicting Human Behav
“ Improving the ecological validity of executive function ior from Sensor Data . New York : Wiley, 2015 .
[ 17 ] D. J. Cook , A. S. Crandall, B. L. Thomas , and N. C.
ing assessment . ” Archives of clinical neuropsychology, 20 Krishnan , “ CASAS : A Smart Home in a Box .” Computer,
vol . 21 , no . 3 , pp . 217-27 , April 2006 . vol . 46 , no . 7 , pp . 62-69 , July 2013 .
[ 4 ] P. Paavilainen, I. Korhonen , J. Lötjönen, L. Cluitmans, [ 18 ] A. P. Association , Diagnostic and statistical manual of
M. Jylhä, A. Sârelâ , and M. Partinen , " Circadian activity mental disorders: DSM -IV - TR ., 4th ed . , ser. Diagnostic
rhythm in demented and non -demented nursing -home and statistical manual of mental disorders. Washington ,
residents measured by telemetric actigraphy. ” Journal of 25 D.C .: American Psychiatric Association , 2000 , vol . 4th ,
sleep research , vol . 14 , no . 1 , pp . 61-68 , March 2005 . no . 2.
[ 5 ] P. Paavilainen , I. Korhonen , and M. Partinen , “ Telem- [ 19 ] M. S. Albert, S. T. DeKosky, D. Dickson , B. Dubois, H.
etric activity monitoring as an indicator of long -term H. Feldman , N. C. Fox , A. Gamst , D. M. Holtzman, W. J.
changes in health and well -being of older people.” Geron- Jagust, R. C. Petersen, P. J. Snyder, M. C. Carrillo , B.
technology , vol . 4 , no . 2 , pp . 77-85 , 2005 . 30 Thies , and C. H. Phelps, “ The diagnosis of mild cognitive
[ 6 ] S. Robben , M. Pol , and B. Kröse, “ Longitudinal ambient impairment due to Alzheimer's disease : recommendations
sensor monitoring for functional health assessments .” in from the National Institute on Aging - Alzheimer's Asso
Proceedings of the 2014 ACM International Joint Con- ciation workgroups on diagnostic guidelines for Alzheim
ference on Pervasive and Ubiquitous Computing Adjunct er's disease.” Alzheimer's & dementia : the journal of the
Publication — UbiComp ’14 Adjunct. New York , N.Y., 35 Alzheimer's Association, vol . 7 , no . 3 , pp . 270-9, May
USA : ACM Press , September 2014 , pp . 1209-1216 . 2011 .
[ 7 ] T. Suzuki and S. Murase , “ Influence of outdoor activity [ 20 ] C. Randolph , Repeatable Battery for the Assessment of
and indoor activity on cognition decline: use of an infra- Neuropsychological Status Update. San Antonio , Tex .:
red sensor to measure activity . ” Telemedicine journal and Psychological Corporation ., 1998 .
e -health : journal of the American Telemedicine Associa- 40 [ 21 ] D. Podsiadlo and S. Richardson , “ The timed “ Up &
tion , vol . 16 , no . 6 , pp . 686-690 , 2010 . Go ” : a test of basic functional mobility for frail elderly
[ 8 ] H. H. Dodge, N. C. Mattek , D. Austin , T. L. Hayes , and persons . ” Journal of the American Geriatrics Society , vol .
J. A. Kaye , “ Inhome walking speeds and variability 39 , no . 2 , pp . 142-148 , 1991 .
trajectories associated with mild cognitive impairment." [ 22 ] N. C. Krishnan and D. J. Cook , " Activity Recognition
Neurology, vol . 78 , no . 24 , pp . 1946-1952 , June 2012 . 45 on Streaming Sensor Data.” Pervasive and mobile com
[ 9 ] G. LeBellego , N. Noury , G. Virone, M. Mousseau , and puting, vol . 10 , pp . 138-154 , February 2014 .
J. Demongeot, “ A Model for the Measurement of Patient [ 23 ] M. M. Ohayon, M. A. Carskadon, C. Guilleminault, and
Activity in a Hospital Suite .” IEEE Transactions on M. V. Vitiello, “ Meta - analysis of quantitative sleep
Information Technology in Biomedicine, vol . 10 , no . 1 , parameters from childhood to old age in healthy individu
pp . 92-99 , January 2006 . 50 als : developing normative sleep values across the human
[ 10 ] C. Galambos, M. Skubic , S. Wang, and M. Rantz , lifespan . ” Sleep , vol. 27 , no . 7 , pp . 1255-1273 , November
“ Management of dementia and depression utilizing in- 2004 .
home passive sensor data . ” Gerontechnology, vol . 11 , no . [ 24 ] M. Jelicic , H. Bosma , R. W. H. M. Ponds , M. P. J. Van
3 , pp . 457-468 , 2013 . Boxtel , P. J. Houx , and J. Jolles , “ Subjective sleep prob
[ 11 ] Wang , M. Skubic , and Y. Zhu, “ Activity density map 55 lems in later life as predictors of cognitive decline. Report
visualization and dissimilarity comparison for eldercare from the Maastricht Ageing Study (MAAS ).” Interna
monitoring.” IEEE Transactions on Information Technol- tional journal of geriatric psychiatry, vol . 17 , no . 1 , pp .
ogy in Biomedicine , vol . 16 , no . 4 , pp . 607-614 , July 73-77 , January 2002 .
2012 . [ 25 ] C. L. Deschenes and S. M. McCurry, “ Current treat
[ 12 ] C. Chen and P. Dawadi , " CASASviz : Web -based visu- 60 ments for sleep disturbances in individuals with demen
alization of behavior patterns in smart environments.” in tia . ” Current psychiatry reports, vol . 11 , no . 1 , pp . 20-26 ,
2011 IEEE International Conference on Pervasive Com- February 2009.
puting and Communications Workshops (PERCOM [ 26 ] J. L. Martin , L. Fiorentino , S. Jouldjian , K. R. Joseph
Workshops ). IEEE , March 2011 , pp . 301-303 . son, and C. A. Alessi , “ Sleep quality in residents of
[ 13 ] M. Kanis, S. Robben , J. Hagen, A. Bimmerman , N. 65 assisted living facilities: effect on quality of life , func
Wagelaar, and B. Kröse, “ Sensor Monitoring in the Home : tional status, and depression .” Journal of the American
Giving Voice to Elderly People .” in Pervasive Computing Geriatrics Society, vol . 58 , no .5 , pp . 829-36 , May 2010 .
US 10,896,756 B2
29 30
[ 27 ] T. Hope, J. Keene , K. Gedling, C. G. Fairburn , and R. [ 41 ] K. Farrahi, D. Gatica - Perez , Discovering routines from
Jacoby, “ Predictors of institutionalization for people with large - scale human locations using probabilistic topic
dementia living at home with a carer. ” International models , ACM Trans. Intell . Syst . Technol. 2 ( 1 ) (2011 )
journal of geriatric psychiatry, vol . 13 , no . 10 , pp . 682- 1-27 .
690 , October 1998 . 5 [ 42 ] K. Farrahi, D. Gatica -Perez , What did you do today ?
[ 28 ] E. Scherder, L. Eggermont, D. Swaab , M. van discovering daily routines from large - scale mobile data ,
Heuvelen , Y. Kamsma , M. de Greef, R. van Wijck, and T. in : Proceeding of the 16th ACM International Conference
Mulder, “ Gait in ageing and associated dementias; its on Multimedia , MM '08 , ACM Press , New York, N.Y.,
relationship with cognition .” Neuroscience and biobehav- USA, 2008 , pp . 849-852 .
ioral reviews, vol . 31 , no . 4 , pp . 485-97 , January 2007 . 10 [ 43 ] J. Zheng, S. Liu , L. M. Ni , Effective routine behavior
[ 29 ] C. McAlister and M. Schmitter - Edgecombe, “ Natural- pattern discovery from sparse mobile phone data via
istic assessment of executive function and everyday mul- collaborative filtering, in : 2013 IEEE International Con
titasking in healthy older adults . " Neuropsychology, ference on Pervasive Computing and Communications,
development, and cognition . Section B , Aging , neuropsy- PerCom , IEEE , 2013 , pp . 29-37 .
chology and cognition, vol . 20 , no . 6 , pp . 735-56 , January 15 [ 44 ] C. Galambos, M. Skubic, S. Wang, M. Rantz, Manage
2013 . ment of dementia and depression utilizing in -home pas
[ 30 ] M. Schmitter -Edgecombe, C. Parsey, and D. J. Cook , sive sensor data , Gerontechnology 11 (3 ) (2013 ) 457-468 .
“ Cognitive correlates of functional performance in older [ 45 ] S. Wang, M. Skubic , Y. Zhu, Activity density map
adults : comparison of self report, direct observation , and visualization and dissimilarity comparison for eldercare
performance -based measures . ” Journal of the Interna- 20 monitoring, IEEE Trans. Inf. Technol. Biomed . 16 ( 4 )
tional Neuropsychological Society JINS , vol . 17 , no . 5 , (2012) 607-614 .
pp . 853-864 , 2011 . [ 46 ] C. Chen , P. Dawadi, CASASviz : Web -based visualiza
[ 31 ] S. T. Farias, D. Mungas, B. R. Reed , D. Harvey, D. tion of behavior patterns in smart environments, in : 2011
Cahn -Weiner, and C. Decarli, “MCI is associated with IEEE International Conference on Pervasive Computing
deficits in everyday functioning . ” Alzheimer disease and 25 and Communications Workshops, PERCOM Workshops,
associated disorders, vol . 20 , no . 4 , pp . 217-223 , 2006 . IEEE, 2011 , pp . 301-303 .
[ 32 ] M. Schmitter - Edgecombe and C. M. Parsey, “ Assess- [ 47 ] M. Kanis, S. Robben, J. Hagen , A. Bimmerman , N.
ment of functional change and cognitive correlates in the Wagelaar, B. Kröse , Sensor monitoring in the home:
progression from healthy cognitive aging to dementia. ” Giving voice to elderly people , in : 2013 7th International
Neuropsychology, vol . 28 , no . 6 , pp . 881-893 , November 30 Conference on Pervasive Computing Technologies for
2014 . Healthcare, PervasiveHealth , Venice , Italy, 2013 , pp .
[ 33 ] V. Dakos, S. R. Carpenter, W. A. Brock , A. M. Ellison , 97-100 .
V. tal , A. R. Ives , S. Kéfi, V. Livina, D. A. Se E. [ 48 ] D. J. Sheskin , Handbook of Parametric and Nonpara
H. van Nes , and M. Scheffer, “ Methods for detecting early metric Statistical Procedures, third ed . , Chapman & Hall /
warnings of critical transitions in time series illustrated 35 CRC , New York , 2007 .
using simulated ecological data .” PloS one , vol . 7 , no . 7 , [ 49 ] A. Gretton , K. M. Borgwardt, M. J. Rasch , B.
p . e41010 , January 2012 . Schölkopf, A. Smola , A kernel two - sample test , J. Mach .
[ 34 ] H. Hotelling , “ The Generalization of Student's Ratio . ” Learn . Res . 13 ( 1 ) ( 2012 ) 723-773 .
The Annals of Mathematical Statistics , vol . 2 , no . 3 , pp . [ 50 ] B. K. Sriperumbudur, A. Gretton , F. K. , B. Schölkopf,
360-378 , August 1931 . 40 The effect of kernel choice of RKHS based statistical
[ 35 ] I. H. Witten and E. Frank , Data Mining : Practical tests , in : Representations and Inference on Probability
Machine Learning Tools and Techniques, Second Edition Distributions Workshop, NIPS , Vancouver, B. C , Canada ,
(Morgan Kaufmann Series in Data Management Sys- 2007 .
tems ) . Morgan Kaufmann Publishers Inc. , June 2005 . [ 51 ] M. Sugiyama, T. Suzuki , Y. Itoh , T. Kanamori, M.
[ 36 ] K. Pérés, V. Chrysostome, C. Fabrigoule , J. M. 45 Kimura , Least -squares two - sample test , Neural Netw . 24
Orgogozo, J. F. Dartigues, and P. Barberger -Gateau, ( 7 ) ( 2011 ) 735-751 .
“ Restriction in complex activities of daily living in MCI : [ 52 ] P. Rashidi, D. Cook , Keeping the resident in the loop :
impact on outcome.” Neurology, vol . 67 , no . 3 , pp . Adapting the smart home to the user, IEEE Trans. Syst .
461-466 , August 2006 . Man Cybern . A 39 ( 5 ) (2009 ) 949-959 .
[ 37 ] “ Stopping elderly accidents, deaths & injuries. Center 50 [ 53 ] P. Rashidi, D. J. Cook , Mining sensor streams for
for Disease Control and Prevention . ” [Online ]. Available : discovering human activity patterns over time , in : 2010
http://www.cdc.gov/homeandrecreationalsafety/pdf/ IEEE International Conference on Data Mining, IEEE ,
steadi /timed up and go test.pdf. 2010, pp . 431-440 .
[ 38 ] M. Ojala and G. C. Garriga, “ Permutation Tests for [ 54 ] P. Paavilainen, I. Korhonen , J. Lötjönen , L. Cluitmans ,
Studying Classifier Performance,” The Journal of 55 M. Jylhä, A. Särelä , M. Partinen, Circadian activity
Machine Learning Research, vol . 11 , pp . 1833-1863 , rhythm in demented and non -demented nursing -home
March 2010 . residents measured by telemetric actigraphy, J. Sleep Res .
[ 39 ] T. Huynh, M. Fritz , B. Schiele , Discovery of activity 14 ( 1 ) ( 2005 ) 61-68 .
patterns using topic models , in : Proceedings of the 10th [ 55 ] S. Robben , G. Englebienne, M. Pol , B. Kröse, How is
International Conference on Ubiquitous Computing, Ubi- 60 grandma doing ? predicting functional health status from
Comp '08 , ACM Press , New York, N.Y., USA , 2008 , pp . binary ambient sensor data , in : 2012 AAAI Fall Sympo
10-19 . sium Series, Washington D.C. , 2012 , pp . 26-31 .
[ 40 ] F.-T. Sun, H.-T. Cheng, C. Kuo , M. Griss, Nonpara- [ 56 ] S. Robben , M. Boot , M. Kanis, B. Kr, Identifying and
metric discovery of human routines from sensor data , in : visualizing relevant deviations in longitudinal sensor pat
2014 IEEE International Conference on Pervasive Com- 65 terns for care professionals, in : 7th International Confer
puting and Communications, PerCom , IEEE , 2014 , pp . ence on Pervasive Computing Technologies for Health
11-19 . care , PervasiveHealth , Venice, Italy, 2013 , pp . 416-419 .
US 10,896,756 B2
31 32
[ 57 ] S. Robben , M. Pol , B. Kröse, Longitudinal ambient [ 73 ] M. S. Albert, S. T. DeKosky, D. Dickson , B. Dubois , H.
sensor monitoring for functional health assessments, in : H. Feldman , N. C. Fox , A. Gamst , D. M. Holtzman, W. J.
Proceedings of the 2014 ACM International Joint Con- Jagust, R. C. Petersen, P. J. Snyder, M. C. Carrillo , B.
ference on Pervasive and Ubiquitous Computing Adjunct Thies , C. H. Phelps, The diagnosis of mild cognitive
Publication, UbiComp ’14 Adjunct, ACM Press , New 5 impairment due to Alzheimer's disease : recommendations
York , N.Y., USA , 2014 , pp . 1209-1216 . from the National Institute on Aging - Alzheimer's Asso
[ 58 ] A. G. Fisher, B. Jones , Assessment of Motor and ciation workgroups on diagnostic guidelines for Alzheim
Process Skills . User Manual, seventh ed . , Three Star er's disease , Alzheimer's Dement. 7 (3 ) (2011 ) 270-279 .
Press, Fort Collins , Colo . , 2012 . [ 74 ] C. L. Deschenes , S. M. McCurry, Current treatments for
[ 59 ] T. L. Hayes , F. Abendroth , A. Adami , M. Pavel, T. A. 10 sleep disturbances in individuals with dementia, Curr.
Zitzelberger, J. A. Kaye , Unobtrusive assessment of activ
ity patterns associated with mild cognitive impairment, [ 75 ] M. JelicicRep
Psychiatry
,
. 11 ( 1 ) ( 2009 ) 20-26 .
H. Bosma , R. W. H. M. Ponds , M. P. J. Van
Alzheimer's Dement. 4 ( 6 ) ( 2008 ) 395-405 . Boxtel , P. J. Houx , J. Jolles , Subjective sleep problems in
[ 60 ] P. Dawadi , D. Cook , M. Schmitter -Edgecombe, Auto
mated cognitive health assessment using smart home 15 later life as predictors of cognitive decline . Report from
smart monitoring of complex tasks , IEEE Trans. Syst . Sci . the Maastricht ageing study (MAAS ), Int. J. Geriatr.
Cybern. 43 ( 6 ) ( 2013 ) 1302-1313 . Psychiatry 17 ( 1 ) ( 2002 ) 73-77 .
[ 61 ] P. Dawadi, D. J. Cook, M. Schmitter - Edgecombe, C. [ 76 ] T. Schmutte , S. Harris, R. Levin , R. Zweig , M. Katz , R.
Parsey, Automated assessment of cognitive health using Lipton , The relation between cognitive functioning and
smart home technologies, Technol. Health Care 21 (4 ) 20 self - reported sleep complaints in nondemented older
(2013 ) 323-343 . adults: results from the Bronx aging study, Behav. Sleep
[ 62 ] M. R. Hodges , N. L. Kirsch , M. W. Newman , M. E. Med . 5 ( 1 ) (2007 ) 39-56 .
Pollack , Automatic assessment of cognitive impairment [ 77 ] H. C. Driscoll, L. Serody, S. Patrick , J. Maurer, S.
through electronic observation of object usage , in : P. Bensasi , P. R. Houck , S. Mazumdar, E. A. Nofzinger, B.
Floréen , A. Krüger, M. Spasojevic ( Eds . ) , Proc . Interna- 25 Bell , R. D. Nebes , M. D. Miller, C. F. Reynolds, Sleeping
tional Conference on Pervasive Computing, in : Lecture well , aging well : a descriptive and cross - sectional study
Notes in Computer Science , vol . 6030 , Springer Berlin of sleep in “ successful agers” 75 and older, Am . J. Geriatr.
Heidelberg , Berlin, Heidelberg, 2010, pp . 192-209 . Psychiatry 16 ( 1 ) (2008 ) 74-82 .
[ 63 ] D. Riboni , C. Bettini, G. Civitarese, Z. H. Janjua, R.
Helaoui , Finegrained recognition of abnormal behaviors 30 [ 78son
] J. L. Martin , L. Fiorentino , S. Jouldjian, K. R. Joseph
, C. A. Alessi , Sleep quality in residents of assisted
for early detection of mild cognitive impairment, Proc . living facilities : effect on quality of life, functional status,
PerCom (2015 ) 149-154 . and depression , J. Am . Geriatr. Soc . 58 ( 5 ) ( 2010) 829
[ 64 ] B. Ef R. Tibs An Introduction to the Boot
836 .
strap, Chapman & Hall , New York , 1994 .
[ 65 ] Y. Benjamini, Y. Hochberg, Controlling the false dis- 35 [79Clin
] E.. Eeles, Sleep and its management in dementia, Rev.
Geront. 16 ( 01 ) (2007 ) 59-70 .
covery rate : A practical and powerful approach to multiple [ 80 ] M. Schmitter - Edgecombe, C. Parsey, D. J. Cook, Cog
testing , J. R. Stat . Soc . Ser. B Stat. Methodol. 57 ( 1 )
( 1995 ) 289-300 . nitive correlates of functional performance in older adults :
[ 66 ] T. Luck , M. Luppa, M. C. Angermeyer, A. Villringer, comparison of self- report, direct observation , and perfor
H.-H. König , S. G. Riedel-Heller, Impact of impairment 40 mance -based measures , J. Int. Neuropsychol. Soc . 17 ( 5 )
in instrumental activities of daily living and mild cogni- (2011 ) 853-864 .
tive impairment on time to incident dementia: results of [ 81 ] S. Artero, J. Touchon , K. Ritchie, Disability and mild
the Leipzig longitudinal study of the aged , Psychol. Med . cognitive impairment: a longitudinal populationbased
41 ( 5 ) ( 2011 ) 1087-1097 . study, Int. J. Geriatr. Psychiatry 16 ( 11 ) ( 2001) 1092
[ 67 ] D. Marson , K. Hebert, Geriatric neuropsychology 45 1097 .
assessment and intervention, in : Geriatric Neuropsychol- [ 82 ] S. T. Farias , D. Mungas, B. R. Reed , D. Harvey, D.
ogy Assessment and Intervention, The Guilford Press, Cahn -Weiner, C. Decarli, MCI is associated with deficits
New York , USA , 2006 , pp . 158-189 . Ch . Functional . in everyday functioning, Alzheimer Dis . Assoc . Disord .
[ 68 ] Y. Ouchi , K. Akanuma, M. Meguro, M. Kasai , H. Ishii , 20 ( 4 ) ( 2006 ) 217-223 .
K. Meguro, 50 [ 83 ] H. Pedrosa, A. De Sa , M. Guerreiro , J. Maroco , M. R.
Impaired instrumental activities of daily living affect con- Simoes , D. Galasko , A. de Mendonca, Functional evalu
version from mild cognitive impairment to dementia: the ation distinguishes MCI patients from healthy elderly
Osaki- Tajiri project, Psychogeriatrics 12 ( 1 ) (2012 ) peoplethe ADCS /MCI/ ADL scale , J. Nutr. Health
34-42 . Aging 14 ( 8 ) ( 2010 ) 703-709 .
[ 69 ] D. J. Cook , N. C. Krishnan, P. Rashidi , Activity 55 [ 84 ] M. Schmitter - Edgecombe, C. M. Parsey, Assessment of
discovery and activity recognition: a new partnership , functional change and cognitive correlates in the progres
IEEE Trans. Syst . Man Cybern . B 43 (3 ) (2013 ) 820-828 . sion from healthy cognitive aging to dementia , Neuro
[ 70 ] N. C. Krishnan, D. J. Cook , Activity recognition on psychology 28 ( 6 ) ( 2014 ) 881-893 .
streaming sensor data, pervasive Mobile Comput . 10 [ 85 ] A. L. Gross , G. W. Rebok , F. W. Unverzagt, S. L. Willis,
(2014) 138-154 . 60 J. Brandt, Cognitive predictors of everyday functioning in
[ 71 ] D. J. Cook , A. S. Crandall, B. L. Thomas , N. C. older adults: Results from the ACTIVE cognitive inter
Krishnan , CASAS : A smart home in a box , Computer 46 vention trial, J. Gerontol. B Psychol. Sci . Soc . Sci . 66 ( 5 )
(7) (2013 ) 62-69 . (2011 ) 557-566 .
[ 72 ] A. P. Association, Diagnostic and Statistical Manual of [ 86 ] M. Schmitter - Edgecombe, C. McAlister, A. Weakley,
Mental Disorders : DSM -IV - TR , fourth ed . , in : Diagnostic 65 Naturalistic assessment of everyday functioning in indi
and Statistical Manual of Mental Disorders , vol . 4 , Ameri- viduals with mild cognitive impairment: the day - out task ,
can Psychiatric Association , Washington , DC , 2000 . Neuropsychology 26 ( 5 ) (2012 ) 631-641 .
US 10,896,756 B2
33 34
[ 87 ] D. Podsiadlo , S. Richardson, The timed “ Up & Go ” : a tests . FIG . 9 plots the distribution of these two scores against
test of basic functional mobility for frail elderly persons , the ages of the participants. Chart 901 measures RBANS
J. Am . Geriatr. Soc . 39 ( 2 ) ( 1991 ) 142-148 . scores against age . Chart 902 measures TUG scores against
[ 88 ] C. Randolph , Repeatable Battery for the Assessment of age .
Neuropsychological Status Update , Psychological Corpo- 5 Prediction
ration , San Antonio , Tex ., 1998 . We performed the following four different prediction
APPENDIX A EXPERIMENTAL VALIDATION
based experiments to evaluate the performance of CAAB
OF CAAB approach and its components: 1 ) We first evaluate the overall
10
CAAB performance in predicting clinical assessment scores .
CASAS Smart Home Test Bed Here , we trained CAAB using the complete set of available
The CAAB approach was tested in CASAS smart homes features. We compared results from several representative
( see http://casas.wsu.edu ), with test single residence apart supervised learning algorithms. 2 ) We then investigated the
ments [ 17 ] with smart home sensors laid out as in FIG . 1 . importance of different activity feature subsets by observing
Each apartment included at least one bedroom , a kitchen , a 15 the resulting performance of CAAB in predicting the clinical
dining area, and at least one bathroom . The homes were assessment scores. 3 ) Next , we investigated the influence of
equipped with combination motion / light sensors on the parameter values andonanalyzing
parameter choices performance by varying CAAB
the impact on prediction
ceilings and combination door / temperature sensors on cabi
nets and doors . These sensors in the smart home test beds performance. 4 ) In the final experiment, we compared
unobtrusively and continuously monitored the daily activi- 20 CAAB performance utilizing AR - labeled activities with a
ties of its residents. The CASAS middleware collected these baseline method that utilized random activity labels .
sensor events and stored the data on a database server. We evaluated all of the above experiments using linear
Dataset correlation coefficient (r ) and m an squared error ( RMSE ).
The CASAS middleware collected sensor data while All performance values were generated using leave - one -out
monitoring the daily behavior of 18 smart home senior 25 cross validation . The data for each participant was used for
residents for approximately 2 years . We used the AR activity training or held out for testing , but was not used for both to
recognition algorithm to automatically label the sensor avoid biasing the model . We used the following methods to
events with the corresponding activity labels . By running compute our performance measures .
CAAB on the activity - labeled ) sensor data , the activity
performance features were computed and statistical activity 30 Correlation
between
coefficient( r ): The correlation coefficient
two continuous variables X and Y is given as :
features were extracted from them . CAAB then created a
training set by combining the statistical activity features and
the corresponding clinical assessment scores (RBANS and COV ( x, y )
TUG ) to train a machine learning algorithm . rx, y =
Tx Oy
Residents 35
Residents included 18 community -dwelling seniors ( 5
females, 13 males ) from a retirement community . All par where Ox and 0 , are the standard deviations of X and Y
ticipants were 73 years of age or older (M= 84 : 71 , SD = 5 : 24 , and coy (X , Y) is the covariance between X and Y In
range 73-92 ) and have a mean education level of 17:52 years our experiments, we evaluate the correlation between
( SD =2 : 15 , range 12-20) . At baseline 51 , participants were 40 the learned behavior model and clinical assessment
classified as either cognitively healthy ( N = 7 ) , at risk for scores . We end up interpreting the experimental results
cognitive difficulties ( N = 6 ) or experiencing cognitively dif based on the absolute value of the correlation coeffi
ficulties ( N = 5 ). One participant in the cognitively compro- cient because our learning algorithm finds a nonlinear
mised group met the Diagnostic and Statistical Manual of relationship between statistical activity features and the
Mental Disorders (DSM -IV - TR ) criteria for dementia [ 18 ] , 45 clinical assessment scores.
while the other four individuals met criteria for mild cog Root Mean Squared Error ( RMSE ) : If ? is a size -n vector
nitive impairment (MCI ) as outlined by the National Insti of predictions and y is the vector of true values , the
tute on Aging - Alzheimer's Association workgroup [ 19 ] . RMSE of the predictor is :
Participants in the risk group had data suggestive of lowered
performance on one or more cognitive tests ( relative to an 50
estimate of premorbid abilities ) , along with sensory and / or
mobility difficulties.
Clinical Tests
RMSE =
i= 1 -
(?: - y )^
Clinicians biannually administered standardized clinical ,
cognitive , and motor tests to the residents . The tests included 55
the Timed Up and Go mobility measure (TUG) as well as the 1 ) Overall CAAB prediction performance: To validate the
Repeatable Battery for the Assessment of Neuropsychologi- overall performance of CAAB performance , we computed
cal Status measure of cognitive status ( RBANS ). The Timed correlations between the CAAB -predicted clinical assess
Up and Go ( TUG) measure [ 21 ] is a test that measures basic ment scores and the provided clinical assessment scores
mobility skills . Participants are tasked with rising from a 60 using the complete set of activity features and three different
chair, walking 10 feet, turning around, walking back to the supervised learning algorithms:
chair, and sitting down . The TUG measure represents the Support Vector Regression ( SVR) : Support vector regres
time required for participants to complete the task at a sion uses a support vector machine algorithm to make
comfortable pace . The RBANS [ 20 ] is a global measure of numeric predictions. The learning model can be
cognitive status identifiers and characterizes cognitive 65 expressed in term of support vectors and kernel func
decline in older adults . A clinical dataset was created using tions can be used to learn a non - linear function . SVR
TUG and RBANS scores obtained from biannual clinical uses the epsilon insensitive loss function that ignores
US 10,896,756 B2
35 36
errors that are smaller than threshold > 0 . We used a this experiment, we created five different training sets, each
linear kernel to generate all our prediction -based per- of which contains a subset of the statistical activity features.
formance results [ 35 ] . For example, the first training set contained all of the
Linear Regression (LR) : Linear regression models the variance based features, the second training set contained all
relationship between the class and the features as the 5 of the autocorrelation -based features etc. Using these train
weighted linear combination of the features. The
weights are calculated from the training data often ing sets , we trained five separate support vector machines .
using the least square approach . As listed in Table 5 , we note that the performance of the
Random Forest ( RF ) : Random forest builds an ensemble SVR in predicting clinical assessment scores using the
learner by creating multiple decision trees on different 10 variance of the activity features is strong as compared to
bootstrap samples of the dataset . It averages the pre other major types of statistical activity features. Therefore ,
dictions from these decision trees to make the predic we hypothesized that the variance of activity performance is
tion [35] .
As listed in Table 4 , we observed that the performances of an important predictor. Additionally, we observed that skew
the machine learning algorithms in predicting the clinical 15 ness - based feature is important for predicting TUG clinical
assessment scores were similar. We also observed that the scores while it was slightly weaker for RBANS predictions .
TABLE 5
Correlation Coefficient (R) and RMSE Values between SVR predicted RBANS and
TUG Scores when SVR is trained using different types of Statistical Activity Features
Score All
Type Measure Change ACF Skewness Kurtosis Variance Features
RBANS r 0.29 0.17 0.30 * 0.21 0.49 ** 0.72 **
RMSE 25.77 21.39 19.90 25.19 17.76 14.94
TUG r 0.06 0.05 0.43 ** 0.06 0.31 * 0.45 *
RMSE 6.05 6.12 5.23 6.60 5.56 5.87
( * p < 0.05 , **p < 0.005 )

30
correlation values were all statistically significant. Because For the second CAAB feature -based experiment, we
SVR performed best overall, we conducted all of the remain- studied the relationship between the clinical assessment
ing experiments using this approach . Additionally, we scores and the statistical activity features subsets that belong
observed that the overall correlation between the predicted to various groups of ADLs . We created nine different ADL
TUG scores and the actual TUG scores were weaker than the 35 groups, each of which contains a combination of one or
predicted RBANS and actual RBANS scores . The weaker more activities ( out of seven activities ) and / or mobility. For
correlation was likely due to the fact that there were only each combination, we created a training set containing all
two activity performance features (mobility and leave home) statistical activity features belonging to the activities in that
that represent the mobility of an individual. Other activities combination . In total , we create nine different training sets .
such as cook , bed to toilet , and relax do not adequately 40 As listed in Table 6 , we make the following three observa
tions :
represent the mobility of a resident.
1 ) In terms of single variables, sleep had the highest
TABLE 4 correlation with RBANS ( r = 0.51 ). In contrast, mobility
showed little correlation with either clinical score .
Overall Prediction Performance of
the Different Learning Algorithms 45 2 ) We observed that correlation is higher when we com
bined variables . Specifically, including automatically
Score Type Measure SVR LR RF recognized ADLs improved the correlation further for
RBANS 0.72 ** 0.64** 0.52 **
both RBANS ( r = 0.61) and TUG ( r = 0.48 ). RBANS
r
RMSE 14.90 20.25 13.66 showed highest correlation when all features are used
TUG r 0.45 ** 0.41 * 0.41 ** 50 ( r = 0.72 ) .
RMSE 5.87 7.62 5.22 3 ) In the case of TUG , the only two variable combinations
that lacked a significant correlation included mobility.
( * p < 0.05 , **p < 0.005 ) Once again , adding automatically - recognized activities
2 ) CAAB prediction performance based on activity fea generally increases the correlation.
ture subsets : We performed a second set of prediction - based 55 TABLE 6
experiments using different subsets of statistical activity
features to study and find the important sets of features as Correlation Coefficient ( R ) and RMSE Values between
listed as follows: SVR -predicted RBANS and TUG Scores when SVR is Trained
1 ) We evaluated the prediction performances of the learn Using Features from Different Activities
ing algorithm when it was trained using different sub- 60 Mobility +
sets of statistical activity features. Score Type Measure Sleep Mobility ADL Leave Home
2 ) We evaluated the result of using statistical activity RBANS 0.51 ** 0.08 0.35 * 0.18
features that belong to various subsets of ADLs . r
RMSE 17.53 21.66 20.15 24.49
In the first experiment, we studied the significance of five TUG r 0.26 0.05 0.35 0.34 *
major types of statistical activity features ( autocorrelation, 65 RMSE 6.19 6.18 5.48 5.48
skewness, kurtosis, variance , and change) that CAAB
extracts from the activity performance features. To perform
US 10,896,756 B2
37 38
TABLE 6 - continued To perform this experiment, we created a training set in
which the statistical activity features ( shown in Table 3 )
Correlation Coefficient ( R) and RMSE Values between were calculated from the sensor data that was randomly
SVR- predicted RBANS and TUG Scores when SVR is Trained
Using Features from Different Activities labeled with the activity instead of using our AR algorithm
ADL + Sleep + Sleep + Sleep + ADL + Mobility + All
5 to automatically generate activity labels . We performed this
Leave Home Mobility ADL Leave Home ADL Features
experiment using the following three steps: 1 ) We labeled
raw sensor events by randomly choosing the activity labels
0.27 0.41 * 0.61 ** 0.57 * 0.50 ** 0.72 ** from the activity set . We chose an activity assuming a
22.01 19.55 17.51 19.14 19.47 14.94
0.43 * 0.20 0.48 ** 0.41 0.13 0.45 * 10
uniform probability distribution over all activity classes . 2 )
5.50 6.57 5.55 6.01 6.79 5.87 We extracted statistical activity features from the sensor data
( * p < 0.05 , **p < 0.005 )
labeled with the random activities . 3 ) We trained SVR using
the statistical features and used clinical assessment scores as
These results show that a relationship exists between ground truth . Performance measures were computed as
RBANS and TUG clinical assessment scores with combined 15 described in the previous sections .
smart home -based parameters of sleep and ADLs . Our As shown in FIG . 11 , we see that the strength of the
observations are interesting and align with results from prior correlation coefficients between predicted and actual clinical
clinical studies that have found relationships between sleep assessment scores were weak and that the RMSE values
and ADL performance with cognitive and physical health were high for the random approach . We also observed that
[ 24 ], [36 ]. Furthermore, we also note that our observations 20 the performances of the learning algorithms trained with
are computed by making use of automated smart home features obtained from the AR labeled activities were sig
sensor data and actual clinical assessment scores . The smart nificantly better than the random labels . Thus, we concluded
home sensor data are ecologically valid because the smart that activity recognition plays a vital role in CAAB and that
home collects data from the real world environment and the CAAB predictions using statistical activity features
CAAB extracts features without governing, changing, or 25 extracted from AR labeled sensor data are meaningful and
manipulating the individual's daily routines . not obtained by chance .
3 ) CAAB performance using different parameters: We Classification Experiments
performed two different experiments to study the effect of To evaluate the performance of CAAB using various
parameter choices on CAAB . In these two experiments, we classification - based experiments, we first discretized the
trained the learning algorithm using the complete set of 30 continuous clinical assessment scores into two binary
features . We first studied how the activity features extracted classes and then used a learning algorithm to classify smart
at different window sizes affected the final performances of home residents into one of these two clinical groups . Per
the learning algorithm . econd , we repeated the steps of the forming these experiments allowed us to use traditional
first experiment to study the effect of using different trend supervised learning -based methods and performance mea
removal techniques. 35 sures to evaluate CAAB , in contrast with the regression
In the first experiment, we compared performance using approaches. We trained the learning algorithms using the
different window sizes and the SVR learning algorithm . The CAAB - extracted statistical activity features. For all of the
results are summarized in FIG . 10. We observe that the classification -based experiments, we used a support vector
strength of the correlation between the actual clinical assess- machine ( SVM ) as the learning algorithm [ 35 ] . SVM iden
ment scores and predicted scores using features derived 40 tify class boundaries that maximize the size of the gap
from smaller and mid - sized window is stronger than the between the boundary and data points. We performed the
larger - sized windows. One possible explanation is that following four different classification experiments: 1 ) We
larger windows encapsulate more behavior trends and day- first evaluated classification performances of the SVM in
to - day performance variation may be lost . Therefore , we use classifying discretized RBANS and TUG clinical assess
mid- sized (30 for RBANS and 55 for TUG) windows for all 45 ment scores when they are trained with different subsets of
of our experiments. In the second experiment, we compare statistical activity features and activity performance fea
three different trend removal techniques. We create three tures. 2 ) In the second experiment, we repeated the first
different training sets that result from removing a Gaussian experiment by discretizing RBANS and TUG scores into
trend , a linear trend , and no trend removal. The results are binary classes at different thresholds. 3 ) Next, we studied the
showed in FIG . 10. We observe that the strength of the 50 classification performances of the learning algorithms
correlation coefficients is stronger and often RMSE values trained using the activity features obtained from the sensor
are smaller when we remove a Gaussian trend from the data labeled with random activities . 4 ) Finally , we evaluated
observations. Thus, in all of our remaining experiments, we the classification performance ( error) by using a permuta
remove a Gaussian trend from the data . tion - based test to ensure that the accuracy results were not
CAAB Performance Using Random Activity Labels 55 obtained by a chance . We evaluated the classification per
In our final prediction experiment, we compared CAAB formance of the learning algorithm using area under the
performance using AR - labeled activities to CAAB perfor- curve, G -mean , accuracy and error and generate them using
mance using random activity labels. There were three main leave -one - out cross - fold validation .
objectives of this experiment. First , we wanted to determine ROC curves assess the predictive behavior of a learning
the importance of the role that the AR algorithm plays in 60 algorithm independent of error cost and class distribu
CAAB . Second , we wanted to verify that CAAB is not tion . The area under the ROC curve ( AUC ) provides a
making predictions based on random chance . Third , we let measure that evaluates the performance of the learning
prediction performance based on random activity labels algorithm independent of error cost and class distribu
serve as a baseline or lower bound performance for com tion .
parison purposes. We expected CAAB performance using 65 G -Mean is the square root of the product of the true
AR - labeled activities to significantly outperform the base- positive and true negative rate [ 35 ] . G -Mean =
line performance. V ( truepositiveratextruenegativerate)
US 10,896,756 B2
39 40
Accuracy is the percent of the correct predictions made by TABLE 8 - continued
the learning algorithm by the total number of predic 52.94 62.00 76.00 80.00 44.00 76.00
tions . Accuracy = # Correct predictions/ # Total predic 0.50 0.62 0.75 0.79 0.43 0.75
tions
Error is the percent of the incorrect predictions made by 5 ( * p < 0.05 , **p < 0.005 )
the learning algorithm by the total number of predic In the second experiment, we evaluated the impact of
tions. Error= 1 - Accuracy.
1 ) CAAB classification performance based on feature CAAB performance of discretizing the continuous clinical
subsets : Similar to the prediction -based experiments, we assessment scores into binary classes at different cutoff
thresholds. The objective of this experiment was to identify
first studied
activity the importance
features of different
and subsets subsets. Forof statistical
of activities the first 10 the range of thresholds that the learning algorithm can
experiment, we discretized clinical assessment scores discriminate. For this experiment, we first discretized
(RBANS and TUG) into binary classes using an equal RBANS and TUG scores into binary classes at different
frequency binning technique. We then trained multiple thresholds. For this experiment, we used all the features to
train the SVM with AdaBoost and generate performance
SVMs to learn the relationship between CAAB -extracted 15 metrics
activity features and these discretized clinical assessment
using leave one out cross validation. We used
SVM / AdaBoost to handle the class imbalance in the dataset
scores. We make three observations based on the classifi- if there existed one [ 35 ] . The AdaBoost algorithm improves
cation performances presented in Tables 7 and 8 . the accuracy of the " weak ” learner by assigning greater
1 ) From Table 8 , we observe that the performance of the weight to the examples that the learning algorithm initially
learning algorithm that is trained with the AR - labeled activi- 20 fails to correctly classify [ 35 ] . The advantages of boosting
ties including sleep and ADLs performs generally better than the classifier to learn an imbalanced class is that since
using other single variables . boosting weights the samples, it implicitly performs both
2 ) From Table 7 , we observe that the classification per- up -sampling and down - sampling with little information loss
formances of the SVM when trained with variance based and is also known to prevent overfitting [ 35 ] . As shown in
activity features are better for both RBANS and TUG scores . 25 FIG . 12 we observed some variations in the performance of
It appears that the skewness - based feature is only important the learning algorithms when they were trained with class
for classifying RBANS clinical scores and not for the TUG labels that were discretized at different thresholds; however,
classifications . the majority of the classification performances were better
3 ) We note that the CAAB performance in the classifica- than random classification performances (i.e. , 50 % accuracy
tion- based experiments involving smart home -based param- 30 for binary classes).
eters of sleep and ADLs are similar to the performances in Additionally, based on FIG . 12 , we made four more
the prediction -based experiments . observations:
TABLE 7
Classification Performance (Accuracy and AUC ) of the SVM in Classifying Clinical
Assessment Scores (RBANS and TUG ) Discretized Using Equal Frequency Binning.
SVM was Trained Using Statistical Activity Features From All Activities .
Score All
Type Measure Change ACF Skewness Kurtosis Variance Features
RBANS Accuracy 26.92 57.69 73.07 57.69 63.46 71.75
AUC 0.27 0.58 0.73 0.58 0.63 0.71
TUG Accuracy 66.00 42.00 46.00 62.00 62.00 76.00
AUC 0.65 0.39 0.44 0.60 0.62 0.75
45
TABLE 8 CAAB performance is generally better when the RBANS

clinical score is discretized at thresholds within the
Classification Performance (Accuracy and AUC ) of the SVM in lower range of RBANS ( 85-100 ) performances and
Classifying Clinical Assessment Scores ( RBANS and TUG )
Discretized Using Equal Frequency Binning. SVM was Trained 50 within the higher range of RBANS ( 125-130 ) perfor
Using Features From Different Activities . mances. It appears that the learning algorithm does
Mobility +
successfully distinguish between the two extreme
Score Type Measure Sleep Mobility ADL Leave Home groups.
CAAB classification performance is best when the con
RBANS Accuracy
AUC
76.92
0.76
57.69 46.15
0.46
61.53
0.62
55 tinuous TUG clinical score is discretized at scores 12
0.57 and 17. We note that a score of 12 and above on the
TUG Accuracy 78.00 62.00 66.00 52.00
AUC 0.77 0.61 0.64 0.52 TUG puts individuals into the falls risk category [ 38 ] .
Given that the TUG test measures the time that is
Correlation Coefficient ( R ) and RMSE Values between SVR required to comfortably complete the Timed Up and Go
predicted RBANS and TUG Scores when SVR is Trained Using 60 task , it appears that the learning algorithm can discrimi
Features from Different Activities nate between the “ slow performers ” and the “ fast
ADL + Sleep + Sleep + Sleep + ADL + Mobility + All
performers .”
Leave Home Mobility ADL Leave Home ADL Features However, we note that similar to the prediction -based
experiment, performance of the classifier in classifying
61.53 75.00 73.08 75.00 48.05 71.15 65 TUG based scores is weaker than the performance
0.62 0.75 0.73 0.75 0.49 0.71 while classifying RBANS scores . As we mentioned
previously, this weaker performance is likely due to the
US 10,896,756 B2
41 42
fact that there are only two activity performance fea TABLE 10
tures (mobility and leave home) that represent the
mobility of an individual. Average Error and P - Value for Test Using SVMs
and Activity Features Extracted from the Dataset
Additionally, we note that CAAB performance in classi Derived from Randomly Labeled Activities
fying both TUG and RBANS clinical labels are mod- 5 Original Test 1
erate to poor when the clinical scores are discretized
into binary classes at the intermediate thresholds. We Class Label Error Err (std ) ?
obtained moderate classification performances because 0.57
the two classes are more likely to have “ similar” RBANS 0.53 ( 0.07) 0.65
activity performance and are therefore harder to dis 10 TUG 0.38 0.37 (0.11 ) 0.48
tinguish from each other.

In the fourth experiment, we compared classification CONCLUSION
performance using AR - labeled activities and random activ
ity labels. Similar to the prediction -based experiment, we 15 We described our CAAB approach to modeling a person's
expect the classification performance based on AR labeled
activities to outperform the random method . As illustrated in collects activity behavior based on smart home sensor data . CAAB
FIG . 13 , we observe that AR -based classification outper sensor data , models activity performance, extracts
forms classification with random activity labels and that the relevant statistical features, and utilizes supervised machine
results are similar to the earlier regression - based experi- 20 representsto apredict learning standard clinical assessment scores . This
longitudinal approach in which a person's own
ments ( t-test on g -mean , p <0.05 ) .
2 ) Permutation - based test : In the final experiment, we routine ate their
behavior and changes in behavior are used to evalu
functional and mobility - based health . We validated
determined whether the aforementioned performance results
were obtained because of chance, rather than because of the our approach by performing several classification and pre
effectiveness of CAAB . With the permutation -based evalu- 25 diction -based experiments. We found statistically significant
correlations between CAAB -predicted and clinician pro
ation method, we calculated a p -value to test a null hypoth vided RBANS and TUG scores. Our experiments were
esis about the relationship between the class labels and conducted using smart home data from 18 smart home
features . This p - value is calculated as a fraction of times that residents and the majority of residents are cognitively
the performance of CAAB on the dataset that is obtained by 30 healthy . We note that CAAB is not intended to replace
shuffling ( permuting ) the class labels exceeded the perfor existing clinical measurements with the smart home -based
mance of CAAB on the original dataset. Similar to the first predictions but may provide a tool for clinicians to use . We
classification -based experiment, we first discretized RBANS also note that an advantage of CAAB is that sparsely
at a threshold of 105.5 and TUG at a threshold of 12.5 using
an equal frequency
proposed in Ojala andbinning
Garrigatechnique
[ 38 ] . We. We a test 35 ously-collected smarthome data and predictionsthe.. continu
performedpermuted
randomly
measured clinical scores can be enhanced using
the class labels to study the relationship between class labels APPENDIX B - EXPERIMENTAL USE OF
and the features . ( H ) : The null hypothesis is that there exists ACTIVITY CURVES
no relationship between the data and the class labels .
Table 9 presents the results from the AR annotated data . 40 An activity curve model provides a big data -based tool for
Based on the null hypotheses H , we made the following representing
observation : the statistically significant (p<0.05 ) result for valuable for a alonger - term behavioral model . Such a tool is
variety of applications including human
the null hypothesis (H ) indicates that there exists a relation automation, health monitoring , and automated health assess
ship between the sensor-based activity performance and ment. In this section , we explain how the activity curve
discretized RBANS and TUG labels . 45
model and the PCAR algorithm can be instrumental in
TABLE 9 performing automated functional assessment. Activities of
daily living such as sleeping, grooming, and eating are
Average Error and P - Value for Test Using SVMs essential everyday functions that are required to maintain
and Activity Features Extracted from the Dataset 50 independence and quality of life. Decline in the ability to
Derived from AR - Annotated Activities
independently perform these ADLs has been associated with
Original Test 1 a host of negative outcomes , including placement in long
Class Label Error Err ( std ) p term care facilities, shortened time to conversion to demen
tia , and poor quality of life for both the functionally
RBANS 0.27 0.52 (0.11 ) 0.009 ** 55 impaired individuals and their caregivers [ 66 ] , [ 67 ] , [ 68 ] . We
TUG 0.24 0.42 ( 0.05) 0.019 * use smart home sensor data to derive activity curves that
( * p < 0.05 , **p < 0.005 ) model the activity routines of a smart home resident. Our
PCAR algorithm detects changes in those activity routines .
This experiment was repeated using activity features We note that changes in ADL patterns are one of the many
derived from randomly -labeled activities . Table 10 lists the 60 forms of behavioral changes that are frequently associated
results . Based on the p -values , we failed to reject the null with changes in cognitive and physical health . We hypoth
hypothesis (H ) that there exists no relationship between the esize that activity curve will allow us to detect changes in
class labels and features. Thus, we conclude that there exists ADL patterns which possibly can provide valuable infor
a relationship between the smart home sensors -based activ- mation about a change in the health condition . To validate
ity features and standard clinical assessment scores ( RBANS 65 our automated assessment technique, we utilize smart home
and TUG ) and that the performance results are not obtained sensor data that was collected from real world smart home
by chance . test beds with older adult residents. We apply robust activity
US 10,896,756 B2
43 44
recognition algorithms [ 69 ] , [ 70] to label these sensor- The activities were annotated with an activity recognition
monitored data with the corresponding activity labels . algorithm as described above . Clinical tests were adminis
Synthetic Dataset tered every six months including TUG and RBANS mea
First , we validate the performance of the proposed PCAR sures of cognitive status .
algorithm by running it on a synthetic activity curve. We 5 Experimental Results
create a synthetic activity curve by compiling synthetic Preprocessing
activity distributions . The synthetic activity distribution We used two different preprocessing techniques to pre
models the patterns of two activities , an arbitrary activity A process activity curves .
and an “ other ” activity . We generate synthetic activity dis Mean smoothing: We ran a mean smoothing filter of size
tributions for each time interval t for N days by applying the 10 3 on activity distributions comprising the activity curve
following three steps. Here I represents the length of each to smooth out noise and minor variations. In this step ,
time interval. we replace the estimate at time interval t with the
Generate a random value p ( 0spsl ) , which represents the average estimate of activity distributions at times t - 2 ,
t- 1 , and t .
average time that is spent in performing activity A 15 Add -One smoothing: Activity distributions for certain
during time interval t. activities can be zero . For example, we rarely eat and
Generate two vectors , S and S ' , each of length N. Generate cook at midnight so activity distributions of these
the vector S from a normal distribution i.e. S ~ Normal activities at midnight are often zero . We perform add
(p , 1 ) . Each element of vector S ' is generated by one smoothing on all of the elements of activity dis
subtracting the corresponding value in S from 1 , i.e. , 20 tributions. In add -one smoothing, we add a constant
1 - SES ' . a= 1 in every elements of the activity distributions.
Create an activity distribution that models patterns of two Add- one smoothing technique is often used in natural
activities at a time interval t . The elements of the language processing to smooth unigram estimates and
activity distribution is [s, 1 - s]. We combine these indi- has an effect of removing zero entries.
vidual synthetic activity distributions at different time 25 Studying Aggregated Activity Curves
intervals into activity curves. When we introduce an FIG . 14 is an example aggregated activity curve that
activity change, we multiply the average time spent models eight different activities , including seven recognized
performing an activity A by a constant factor in each activities (i.e. , sleep , bed toilet transition , eat, cook , relax,
time interval. personal hygiene) and an “ other ” activity. This sample
Using the aforementioned method , we create two different 30 aggregated activity curve was derived using an aggregation
sets of activity curves. The first set Y does not contain any window of size x = three months based on actual single
changes and another different set of daily activity curves Z resident smart home sensor data and 5 min time intervals .
contains changes in all activity distributions every 90 days . We observe that this smart home resident usually goes to
We run the PCAR algorithm on an aggregated activity curve sleep at around 9:00 PM and wakes up at around 7:00 AM .
of size 90 days using the time interval index alignment. 35 We also observe that the resident exhibits a fairly fixed
PCAR successfully detects all changes in the dataset where schedule for eating breakfast, lunch, and dinner.
activity changes were made and does not produce any false Comparing Activity Distributions
positives in the synthetic dataset that does not contain For most individuals, our activities follow a common
changes. pattern based on factors such as time of day. Thus, the
Next we demonstrate how PCAR can be used to detect 40 activity distributions that model activities at different time
changes in real smart home data. For this study, we recruited intervals belonging to different times of a day will be
18 single -resident senior volunteers from a retirement com- different. In our
munity and installed smart home sensors in their homes [ 71 ] . first experiment, we assess whether our proposed activity
( See FIG . 1. ) The smart home sensors unobtrusively and curve can capture such differences in activity distributions .
continuously monitor resident activities . We continuously 45 We calculate an aggregated activity curve for five -minute
collected raw sensor events for an extended period ( -2 time intervals using the first three months of activity
years ) from all of the residents. At the same time , standard- annotated sensor
ized clinical , cognitive, and motor tests were administered data from one of our smart homes . We calculate a pairwise
biannually to the residents. distance ( symmetric KL divergence) matrix between activity
Participants 50 distributions from this aggregated activity curve. We plot
Participants included 18 senior residents ( 5 females, 13 this pairwise distance matrix in a heat map shown in FIG .
males ) from a senior living community. All participants were 15. From the heat map , we observe that the distance between
73 years of age or older, and had a mean level of education activity distributions varies according to the time of day. We
of 17.52 years . At baseline , participants were classified as observe that the darkest colors appear along the diagonal
cognitively healthy ( N = 7 ), at risk for cognitive difficulties 55 when we compare activity distributions for the same time of
(N=7 ) or experiencing cognitive difficulties ( N = 4 ). One day. In contrast, we observe that the hottest colors ( greatest
participant in the cognitively compromised group met the distance ) occurs when comparing activities at midnight
criteria for dementia [72 ], while the other three individuals (when the resident typically sleeps ) to activities in mid
met the criteria for mild cognitive impairment ( MCI ) [ 73 ] . afternoon when the resident is quite active . Additionally, we
Smart Home Test Environment 60 also see different clusters emerge ( for instance , between
The 18 smart home test beds are single - resident apart- times 0 and 100 ) corresponding to times of day. These
ments, each with at least one bedroom , a kitchen , a dining observations provide intuitive visual evidence that the activ
area, and one bathroom . The sizes and layouts of these ity curve is capturing generalizable differences in activity
apartments vary from one apartment to another. The apart- routine at various times of the day.
ments are equipped with combination motion / light sensors 65 In the next experiment, we study how the activity distri
on the ceilings and combination door/temperature sensors on bution distances within an activity curve (the y axis in FIG .
cabinets and doors ( as shown in FIG . 1 ) . 16 ) change as a function ofthe time interval size (the x axis ) .
US 10,896,756 B2
45 46
For this experiment, we calculated an average pairwise
distance between activity distributions within aggregated ALGORITHM for BehaviorAndHealth Correlation (t)
activity curve for each time interval size . We observe that as 1: // t – testing time points
the time interval increases , the average pairwise distance 2: // Return correlation coefficient.
BehaviorChangeScores = [ ]
between daily activity distributions decreases . Such a 5 3:
4: ClinicalChangeScores = [ ]
decrease in distances is observed because activity distribu 5: i=0
tions at larger sized time intervals are overwhelmed by 6: repeat
activities that take larger duration ( such as sleep ) . As a 7: 21 = AggregateActivityCurvesAtTime (t; + 3 months )
result , smaller differences between such activity distribu 8:
Sit1 - Aggregate ActivityCurvesAtTime (t; + 1 + 3 months)
=
tions are harder to detect. 10 9: Dist = EmpiricalDistribution (? ; , Eit1 , Np)

10 : Si = PCAR (Ei , Sit1 , Dist)
Aggregation Window Size 11 :
12 :
S2 = ClinicalScores (t;, t; + 1 )
Append ( S1 , BehaviorChangeScores)
In the next experiment, we determine the minimum length 13 : Append ( S2 , ClinicalChangeScores)
of an aggregation window that is required to calculate a 14 : i=i+1
stable aggregated activity curve for our smart home data . 15 15 : until
FIG . 17 shows the variations in the length of an aggregate 12 : return Correlation ( S1 , S2 ) ;
window at different interval sizes calculated using all the
available sensor data . We observe that the length of the We make the following observations:
aggregation window is larger for the smaller interval sizes We obtain statistically significant correlations between
and smaller for the larger interval sizes . We can explain such 20 activity change scores and TUG scores (FIG . 18B ) .
differences in length of the aggregation window based on the No correlations exist between activity change scores
observations we made between average pairwise distances obtained from random predictions and TUG scores.
and interval sizes in FIG . 17. At larger interval sizes , activity No correlations exist between smart home based activity
distributions are dominated by activities that take a long change scores and RBANS scores.
time to complete ( such as sleep ) . Thus , the distance between 25 Often , the strength of correlations at larger time interval
two activity distributions for such activity curves are sig sizes is weak because at larger time intervals activities
nificantly lower than the distance between two activity are either dominated by sleep activity or other activity.
distributions for activity curves at smaller time intervals . Hence, changes in activity distributions at large time
Hence, we obtain a stable activity curve using a smaller 30 intervals are comparatively harder to detect .
aggregation window size for larger interval sizes . Continuous Change Scores
Change Scores and Correlations In the previous section , we predicted change scores at six
In this section, we study the strength of the correlations month intervals to correlate the smart home - based behavior
between the changes detected in activity routines by the can alsoscores
change with changes in standard clinical scores . We
PCAR algorithm and the corresponding standard clinical 35 running the PCAR these
calculate change scores more frequently by
algorithm on the activity curves that lies
scores (RBANS and TUG) for a smart home resident. within a sliding window of size six months and shifting this
Specifically, we calculate correlations between change sliding window by one month (30 days). We will refer to
scores calculated by applying PCAR on activity curves such frequent change scores as continuous change scores .
derived using activity labeled smart home sensor data and We can use continuous change scores to monitor the " per
corresponding clinical scores ensuring that each pair of the 40 formance” of a smart home resident's everyday behavior.
smart home change score and clinical score was observed at FIG . 19 shows how the continuous change scores of two
around the same time . smart home residents have varied with time . Each point in
To obtain such correlations, we first calculate aggregated a plot represents a total change score obtained by using the
activity curves for two three -month aggregation windows , PCAR algorithm to compare behavior six months prior with
W and W2 . Next, we apply PCAR to these activity curves 45 current behavior. First , we plot continuous change scores of
to obtain a smart home - based change score . We also obtain a resident whose health status has declined (FIG . 19 ) . We
clinical scores measured at time points, t and tz . We repeat observe that after a year, the total change score of this
this step for all available pairs of consecutive testing time resident started to fluctuate . Such fluctuations indicate
points for all 18 residents. Finally, we calculate Pearson changes in the average daily routines of this resident.
correlation and Spearman rank correlation between the 50 resident
Similarlywhose
, we plot continuous change scores of another
health has been in excellent condition for the
activity change scores and the corresponding clinical scores entire data collection period (FIG . 19 ) . We observe that the
to evaluate the strength of the relationship . The process is PCAR algorithm detects very few changes in the average
summarized in Algorithm for BehaviorAndHealthCorrela daily routines of this
tion(t) below . To evaluate our automated health correlation 55 Individual Activity resident
Change
.
based on smart home data , we derived correlation coeffi The change scores calculated in the previous sections
cients between change scores obtained from the smart quantify overall changes in average daily routines for the
home - based activity curve model with the standard health entire collection of known activities . In this experiment, we
clinical scores (TUG and RBANS scores ) . To conduct this can quantify total changes in the average daily routines of
experiment, we ran 1500 permutation iterations and derived 60 some specific activities by running PCAR algorithm on a
change scores for both alignment techniques. We repeated reduced activity set . The elements in this reduced activity set
the experiments for different time interval sizes . As a base- are activities that we want to monitor and the “ other activi
line for comparison, we generated random change scores by ties ” class is used to represent all of the remaining activities .
randomly predicting a change between activity distributions For example, if we want to monitor sleep and bed to toilet
instead of using the PCAR algorithm . Tables 1 and 2 shown 65 activities , we put three elements ( sleep , bed to toilet and
in FIGS . 18A - 18B list the correlations between these two other) in the reduced activity set . Using this reduced set of
scores for different time interval sizes . activity, we can use PCAR to obtain continuous change
US 10,896,756 B2
47 48
scores . FIG . 20 shows the continuous sleep change scores of continuous stream with an activity label that corre
the same two smart home residents for whom we studied the sponds to a detected activity of the individual, the
continuous change scores in FIG . 19. We see that PCAR automatic labeling transforming the data into measure
detected changes in the overall sleep routine of the first ments of activities of daily living;
resident while it does not detect any sleep routine changes 5 automatically extracting activity performance features
for the other resident. from the activity labeled sensor data , wherein each
activity performance feature includes a time - based fea
CONCLUSIONS ture and a sensor -based feature ;
automatically extracting statistical features from the
We developed an activity curve model to represent daily 10 extracted activity performance features, the extracted
activity - based behavior routines. The activity curve models statistical features further measuring attributes over
the activity distributions of the activities at different times of time of each extracted activity performance feature
a day. Using the activity curve model , we developed and between at least two clinical cognitive testing point
tested the PCAR algorithm to identify and quantify changes times ; and
in the activity routines. We validated our model by perform- 15 submitting indications of the extracted statistical features
ing experiments using synthetic data and longitudinal smart to a machine learning engine configured to predict and
home sensor data . PCAR is able to represent behavior output a cognitive assessment score using the sensor
patterns through big data analysis. event data and the extracted statistical features, the
The current activity model considers activity distributions machine learning engine modeling relationships
using different interval sizes . Further, while performing 20 between clinical cognitive assessment scores and activ
experiments with an activity curve model , we chose a subset ity performance features to correlate the predicted
of activities that are considered important in daily life. We cognitive assessment score to a clinical cognitive
also note that the activities that did not fit into these seven assessment .
predefined categories were termed “ other ” . We note that the 2. The method of claim 1 , the outputting the cognitive
“ other” data is very large , complex , and represents important 25 assessment score further comprising forwarding the pre
activities that could be added to our activity vocabulary in dicted clinical cognitive assessment to a clinician for further
future embodiments. diagnosis of cognitive state .
We developed the PCAR algorithm to quantify changes in 3. The method of claim 1 wherein the plurality of smart
an activity routine . PCAR makes use of a smart home sensor sensors comprise one or more motion sensors , light sensors ,
data of an individual collected over a period to quantify 30 door sensors , or temperature sensors , wherein each sensor is
changes in the activity routine and outputs change scores . associated with an identifier that identifies a location in the
This algorithmic approach is important because activity residence .
routines vary among individuals. 4. The method of claim 1 wherein the automatically
In this experiment, we studied the relationship between labeling of the raw output data of each sensor event with an
the output from the PCAR algorithm and the standard 35 activity label includes labeling the data with activities indi
clinical and physical health scores . We found moderate cating two or more of mobility, sleep , or activities of daily
correlations between the change scores and standard TUG living.
scores . However, we found that the correlations between 5. The method of claim 4 wherein the activities of daily
smart home - based change scores and standard cognitive living comprise cooking , eating, relaxing, personal hygiene,
scores (RBANS) were not as strong as we expected because 40 or leaving home .
the majority of the older adults for whom we analyzed the 6. The method of claim 4 wherein the sleep label includes
data are healthy older adults. We note that clinicians have sleeping and a bed to toilet transition .
frequently argued for the existence of a relationship between 7. The method of claim 1 wherein the time -based feature
changes in ADL patterns and changes in the health . We want includes an indication of duration of the activity and the
to use our idea of an activity curve to detect changes in ADL 45 sensor - based feature includes an indication of number of
patterns which possibly can be associated with a change in corresponding sensor events.
cognitive and physical health conditions. Similarly, we also 8. The method of claim 1 wherein extracting activity
demonstrated methods to evaluate the “ average perfor- performance features further comprises extracting statistical
mance ” of a smart home resident by continuously monitor- activity features from the extracted activity performance
ing changes in the overall daily routine as well as a set of 50 features and the extracted statistical activity features include
specified activities. one or more of measures of variance, autocorrelation, skew
The work described here confirms that pervasive com- ness , kurtosis, or change.
puting methods can be used to correlate an individual's 9. The method of claim 1 wherein the sensor data is
behavior patterns and clinical assessment scores. Visualiza- received continuously and wherein the activity recognition
tion tools for the activity curve model could be incorporated. 55 logic annotates and transforms the data by viewing the data
The invention claimed is : in continuously moving evaluation windows over the data
1. A method in a computing system for facilitating a such that each evaluation window provides context for
cognitive assessment of an individual, comprising; evaluating a newly received input of sensor data .
receiving from a plurality of smart sensors installed in a 10. The method of claim 9 wherein the sensor data is
residence, a continuous stream of raw output data of 60 received in near real time .
sensor events, the sensor events corresponding to a 11. The method of claim 1 wherein indications of the
plurality of activities engaged in by the individual in extracted activity performance features are compared with
the residence, the raw output data of each sensor event actual clinical assessments to train the machine learning
including at least a date , time , sensor identifier, and engine such that clinical data of the resident is used as a
sensor value; 65 baseline for predictive assessment.
using automated activity recognition logic , automatically 12. The method of claim 1 wherein the machine learning
labeling the raw output data of each sensor event in the engine is a support vector machine.
US 10,896,756 B2
49 50
13. A cognitive facilitator assessment computing system the extracted activity performance features comprises at
comprising ; least one of variance, autocorrelation , skewness, kurtosis, or
a memory ; change.
a computing processor; 16. The cognitive facilitator assessment computing sys
a machine learning engine; and 5 tem of claim 13 wherein the time -based feature includes an
clinical assessment activity behavior tool stored in the indication of duration of the activity and the sensor - based
memory and further comprising : feature includes an indication of number of corresponding
sensor events .
sensor input logic configured, when executed, to pro 17. A non - transitory computer -readable memory medium
vide one or more streams of continuous live smart 10 containing contents for instructing a computer process to
home based sensor data corresponding to activities facilitate a cognitive assessment of a resident of a residence
of a resident in a residence, with the residence having using a plurality of smart sensors installed in the residence,
a plurality of smart - home based sensors installed by performing a method comprising:
therein ; receiving from the plurality of smart sensors , a continuous
an activity recognizer configured , when executed , to stream of raw output data of sensor events, the sensor
receive the continuous sensor data real time and to
15
events corresponding to a plurality of activities
label the sensor data with activity labels to transform engaged in by the individual in the residence, the raw
the data into measurements of activities of daily output data of each sensor event including at least a
living ; date, time , sensor identifier, and sensor value ;
an activity performance features extractor configured , 20
automatically labeling the raw output data of each sensor
when executed , to automatically extract activity per event in the continuous stream with an activity label
formance features from the activity labeled sensor that corresponds to a detected activity of the resident,
data , the activities corresponding to activities of the automatic labeling transforming the raw output data
daily living and / or mobility, wherein each extracted into measurements of activities of daily living ;
activity performance feature includes a time -based 25
automatically extracting activity performance features
feature and a sensor- based feature; from the activity labeled sensor data , wherein each
a statistical feature extractor configured, when extracted activity performance feature includes a time
executed, to automatically extract statistical features based feature and a sensor - based feature;
from the extracted activity performance features, the automatically extracting statistical features from the
extracted statistical features further measuring attri 30 extracted activity performance features, the extracted
butes over time of each extracted activity perfor statistical features further measuring attributes over
mance feature between at least two clinical cognitive time of each extracted activity performance feature
testing point times ; and between at least two clinical cognitive testing point
a cognitive assessment predictor configured, when times ; and
executed, to receive the extracted performance fea 35 submitting indications of the extracted statistical features
to a machine learning engine to predict a cognitive
tures and to submit them to the machine learning assessment score using the continuous sensor event
engine to automatically predict a cognitive assess data and the extracted statistical features, the machine
ment score using the sensor event data and the learning engine modeling relationships between clini
extracted statistical features, the machine learning cal cognitive assessment scores and activity perfor
engine modeling relationships between clinical cog- 40 mance features to correlate the predicted cognitive
nitive assessment scores and activity performance assessment score to a clinical cognitive assessment.
features to correlate the predicted cognitive assess 18. The memory medium of claim 17 wherein the auto
ment score to a clinical cognitive assessment.
14. The cognitive facilitator assessment computing sys matically labeling of the raw output data of each sensor
event in the continuous stream with an activity label
tem of claimconfigured
is further 13 wherein the executed
, when cognitive, assessment
to train the predictor
machine 45 includes labeling the data withactivities indicating mobility,
learning engine by initially submitting extracted statistical sleep , and activities of daily living .
features of the extracted activity performance features with ties19.ofThedaily memory medium of claim 18 wherein the activi
living comprise cooking, eating, relaxing,
actual clinical assessments . personal hygiene , or leaving home.
15. The cognitive facilitator assessment computing sys
tem of claim 13 wherein the extracted statistical features of

US10896756

Uploaded by

Copyright:

Available Formats

US10896756

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

US10896756

Uploaded by

Copyright:

Available Formats

US010896756B2

Receive sensor eve data rav

Ruta a lively recognizer 29'clo

5x95 cavity teltunar

Extract statstical any saturaz

Output YeoAgnitive over

( 51 ) Int . Ci . Chen , C. et al ., “ CASASviz : Web -based Visualization of Behavior

( 56 ) References Cited Ouchi , Y. et al., “ Impaired instrumental activities of daily living

koments Diagnosi 205 206

tfCoAlrBs amcetinvsudrienscgplriednctailngase mnt fortCAcoutirvltsey apctivertnys

CAcutriv teyECnhgaine MReopdrutlieng

SAFCsuoyngectimvnael SIenpsuortEngine Recognizer ActivtyPerfomanc EFxetarucos

Receive sensor event data in raw

Run activity recognizer logic to

Extract activity performance

Extract statistical activity features

Qutput predicted cognitive and/or

ONCM09:004201o09.-405-k251 OFEatM09:00420109-.250-519 ONRM09:00420e209-l.5a0-21x5 ONCM09:004202o09.-285-k215 ONCM09:004202o09-.258-k125

ONM09:00420109.-405-251 OFFM09:00420109-.250-519 ONM09:00420209-.50-215 ONM09:00420209-.258-125 ONM4209:000309.-215-765

) normalized( interval that

oforecissandthetNarer,.Inoaliaetcrmugpclasoeintghpaiesdcl fsthetbycisp,.Inairbeghmnstpfaiosercnmtapidlgce byiscs,a.Fdthepfotw)setof(uanebihclartmouintslraglaemhiydc 7

Other Progams Data Repositre 2806

OCotmhpuetr MReadiblae 2805

SFACasoycgienltmavoer Smart-home1/1 803

) normalized(interval that in activity in spent time .Avg

size window aggregation of Distribution

measure of spread . 7: while Tj < ( T2 - W ) do

( * p < 0.05 , **p < 0.005 )

TABLE 8 CAAB performance is generally better when the RBANS

tinguish from each other.

tions are harder to detect. 10 9: Dist = EmpiricalDistribution (? ; , Eit1 , Np)

You might also like