RESEARCH ARTICLE

Psychometric Properties of a Generic,

Patient-Centred Palliative Care Outcome
Measure of Symptom Burden for People with
Progressive Long Term Neurological
Conditions
Wei Gao1*, Vincent Crosby2, Andrew Wilcock3, Rachael Burman1, Eli Silber4,
a11111
Nilay Hepgul1, K Ray Chaudhuri5, Irene J. Higginson1, on behalf of the OPTCARE Neuro
trial¶
1 King’s College London, Faculty of Life Sciences and Medicine, Department of Palliative Care, Policy and
Rehabilitation, Cicely Saunders Institute, London, United Kingdom, 2 Palliative Medicine, Nottingham
University Hospitals NHS Trust, Nottingham, United Kingdom, 3 Palliative Medicine and Medical Oncology,
School of Medicine, University of Nottingham, Nottingham, United Kingdom, 4 Department of Neurology,
Kings’ College Hospital, London, United Kingdom, 5 National Parkinson Foundation Centre of Excellence,
OPEN ACCESS Kings College Hospital and Kings College, London, United Kingdom

Citation: Gao W, Crosby V, Wilcock A, Burman R, ¶ Membership of the OPTCARE Neuro trial is provided in the Acknowledgments.
Silber E, Hepgul N, et al. (2016) Psychometric * [email protected]
Properties of a Generic, Patient-Centred Palliative
Care Outcome Measure of Symptom Burden for
People with Progressive Long Term Neurological
Conditions. PLoS ONE 11(10): e0165379. Abstract
doi:10.1371/journal.pone.0165379

Editor: Jerson Laks, Universidade Federal do Rio

de Janeiro, BRAZIL
Background
Received: October 19, 2015
There is no standard palliative care outcome measure for people with progressive long
term neurological conditions (LTNC). This study aims to determine the psychometric prop-
Accepted: September 15, 2016
erties of a new 8-item palliative care outcome scale of symptom burden (IPOS Neuro-S8) in
Published: October 25, 2016 this population.
Copyright: © 2016 Gao et al. This is an open
access article distributed under the terms of the Data and Methods
Creative Commons Attribution License, which
permits unrestricted use, distribution, and Data were merged from a Phase II palliative care intervention study in multiple sclerosis
reproduction in any medium, provided the original (MS) and a longitudinal observational study in idiopathic Parkinson’s disease (IPD), multi-
author and source are credited.
ple system atrophy (MSA) and progressive supranuclear palsy (PSP). The IPOS Neuro-S8
Data Availability Statement: Due to the ethical was assessed for its data quality, score distribution, ceiling and floor effects, reliability, fac-
restriction, the datasets underlying the findings
tor structure, convergent and discriminant validity, concurrent validity with generic (Pallia-
cannot be shared publicly. However, aggregate and
nonidentifiable data are provided in the tables, and tive care Outcome Scale) and condition specific measures (Multiple Sclerosis Impact
requests to access the original data can be sent to Scale; Non-motor Symptoms Questionnaire; Parkinson’s Disease Questionnaire), respon-
[email protected] or [email protected]. siveness and minimally clinically important difference.
Funding: Funding received by WG, VC, AW, RB,
ES, KRC, IJH from the National Institute for Health Results
Research, Health Services and Delivery Research
Programme (12/130/47) http://www.nets.nihr.ac. Of the 134 participants, MS patients had a mean Extended Disability Status Scale score
uk/programmes/hsdr. Funding received by WG and 7.8 (SD = 1.0), patients with an IPD, MSA or PSP were in Hoehn & Yahr stage 3–5. The

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 1 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

IJH from the Collaboration for Leadership in IPOS Neuro-S8 had high data quality (2% missing), mean score 8 (SD = 5; range 0–32), no
Applied Health Research and Care South London, ceiling effects, borderline floor effects, good internal consistency (Cronbach’s α = 0.7) and
National Institute for Health Research which is a
partnership between King’s Health Partners,
moderate test-retest reliability (intraclass coefficient = 0.6). The results supported a moder-
St. George’s, University London, and St George’s ately correlated two-factor structure (Pearson’s r = 0.5). It was moderately correlated with
Healthcare NHS Trust. http://www.clahrc- generic and condition specific measures (Pearson’s r: 0.5–0.6). There was some evidence
southlondon.nihr.ac.uk/palliative-and-end-life-care.
for discriminant validity in IPD, MSA and PSP (p = 0.020), and for good responsiveness
IJH is a National Institute for Health Research
senior investigator. The funders had no role in and longitudinal construct validity.
study design, data collection and analysis, decision
to publish, or preparation of the manuscript.
Conclusions
Competing Interests: The authors have declared
IPOS Neuro-S8 shows acceptable to promising psychometric properties in common forms
that no competing interests exist.
of progressive LTNCs. Future work needs to confirm these findings with larger samples
and its usefulness in wider disease groups.

Introduction
Progressive long term neurological conditions (LTNC) are a group of irreversible and degener-
ative diseases of the nervous system that share some common characteristics, such as an
increasing deterioration in neurological function over time, leading to increasing disability,
cognitive impairment and dependence on others. As the disease progresses, complex physical,
psychosocial and spiritual issues can arise which require palliative care and integrated care pro-
visions from multiple agencies [1–3]; the primary care focus also moves from cure to comfort,
and the patient perspectives become central [4]. Little research is conducted on how best to
integrate and coordinate these care services, and on the effectiveness of different care models
among this population. One major barrier may be the lack of appropriate patient centred palli-
ative care outcome measures (PCOM).
A number of disease specific PCOMs are in existence for LTNCs, e.g. the Multiple Sclerosis
Impact Scale (MSIS-29)[5], the Parkinson’s Disease Questionnaire (PDQ-39, PDQ-8) [6, 7],
and Non-Motor Symptoms Questionnaire (NMSQuest) [8], but all of these measures fail to
adequately address the progressive nature of the LTNC and the resulting complex needs
including palliative care needs. For example, none of the existing measures feature pain, dys-
pnoea and other distressing symptoms which are frequently reported by patients and are rec-
ommended as triggers for palliative care referral [1, 2, 9, 10].
The 10-item palliative care outcome scale (core POS) is a widely used PCOM for palliative
care patients. However, it was designed for general palliative care purposes and when applied to
LTNCs it is not as sensitive as the measure primarily focusses on key symptoms that require palli-
ative care input [10–12]. A brief PCOM for palliative care needs, comprising five typical symp-
toms (pain, nausea, vomiting, mouth problems and sleeping difficulty), was found to have
satisfactory psychometric properties and was sensitive to change in patients with multiple sclero-
sis (MS) [11–13]. In an observational study in patients with idiopathic Parkinson’s disease (IPD),
multiple system atrophy (MSA) and progressive supranuclear palsy (PSP), a 20-item measure
covering more comprehensive aspects of palliative care needs including symptoms was used [14].
It appears to be a promising tool but has not yet been through formal psychometric evaluation.
Based on the well validated core POS [15–17], several commonly used condition specific POS
symptom versions [10–12], and clinical management guidelines for LTNCs [18, 19], we devel-
oped a new patient reported, integrated palliative care outcome measure that may be used to eval-
uate the outcome for people with progressive LTNC (IPOS Neuro).

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 2 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

A key component in the IPOS Neuro is the symptom burden, a concept that encompasses
both the severity of the symptoms and the patient's perception of the impact of the symptoms
[20]. This study provides a formal evaluation of the psychometric properties of the symptom
burden specifically associated with eight core symptoms in the IPOS Neuro. For the conve-
nience of reference, we named this subscale version the IPOS Neuro-S8. We examined five psy-
chometric aspects: data quality, floor and ceiling effects, internal consistency and test-retest
reliability, cross-sectional construct validity. We also provided a preliminary validation on
responsiveness and longitudinal validity, and an initial estimate of the minimally clinically
important difference (MCID).

Data and Methods

Ethical Approval
The Ethical approval for the MS trial was granted by the King’s College Hospital NHS Trust
Local Research Ethics Committee (protocol number: 01-04-018); the IPD,MSA & PSP study
was approved by both the Research Ethics Committee of the Institute of Psychiatry and King’s
College Hospital NHS Foundation Trust (REC number: 05/Q0703/196). Written consent was
given by all participants in both studies, through consent forms approved by the Research Eth-
ics Committees.

Setting and Design

Data was pooled from two studies: A) a randomised Phase II trial evaluating effectiveness of a
short-term palliative care intervention in people severely affected by multiple sclerosis (MS)
[11]; B) a longitudinal community based study in IPD, MSA and PSP [14]. Both studies were
conducted in South East England (including London, Kent and Sussex) with the MS trial pri-
marily focusing on patients living in South East London. Patients were recruited from a neuro-
sciences Centre based at King’s College Hospital (KCH). This Centre is the second largest
regional neuroscience centre in the UK and serves an estimated population of 3.5 million
people.

Eligibility Criteria
Patients with MS were included if their usual clinician deemed them to have one or more of
unresolved symptoms, psychosocial concerns, end-of-life issues, progressive illness, or complex
needs(i.e., palliative care needs); patients with a clinical diagnosis of IPD, MSA or PSP, who fell
into Hoehn and Yahr (HY) stages 3–5 for IPD or equivalent motor disability for MSA and PSP
[21].
In both studies, patients with urgent needs for symptom control, or support, or who were
deteriorating rapidly or dying were excluded. Patients living in nursing homes were also
excluded. The MS trial patients were randomly allocated to either the control or the interven-
tion group, and the two groups were comparable for all factors. There was no clinically detect-
able change at 6 weeks on any clinically important variables [11].

Measures
The 8 key symptom subscale of the Integrated Palliative care Outcome Scale for neuro-
logical conditions (IPOS Neuro-S8). A generic, patient-centred palliative outcome measure
of symptom burden for people with progressive LTNCs. It is based on a validated symptom
burden measure in advanced multiple sclerosis—palliative care outcome scale, 5 symptoms
(POS-MS-5)[11]. The POS-MS-5 assesses five key symptoms—pain, nausea, vomiting, mouth

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 3 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

problems and sleeping difficulties over the past three days. To extend its use to the other neuro-
logical conditions, three further symptoms (breathlessness, spasms and constipation) prevalent
in LTNCs are included to form the IPOS Neuro-S8 [18, 19, 22]. Each item is rated on a Likert
scale from 0 (no problem) to 4 (overwhelming problem), with a total score range from 0 to 32
(S1 File). We propose to incorporate it into a 45-item integrated palliative care outcome mea-
sure (IPOS Neuro), with 37 items asking about symptom experience and the remaining items
asking about information needs, practical concerns, anxieties of the patient and family, and
their overall feeling of being at peace (S2 File).
The Core Palliative care Outcome Scale (Core POS). A well validated and commonly
used generic palliative care outcome measure [13, 15, 16]. It comprises of 10 items asking
patients about their physical symptoms, information and practical needs, patient and family
anxiety and well-being. The impact of the items on patients over the past three days is scored
using a five-point Likert scale system, ranging from 0 (no problem) to 4 (overwhelming prob-
lems). The total score range is 0 to 40.
The Multiple Sclerosis Impact Scale (MSIS-29). A disease-specific health related quality
of life measure, with two subscales assessing physical and psychological impact of multiple sclero-
sis (MS)[5, 23]. It has a 20-item physical impact scale and a 9-item psychological impact scale.
Each item on the MSIS has a five-point response option (1–5): “not at all,” “a little,” “moder-
ately,” “quite a bit” and “extremely.” Scores on the physical impact scale can range from 20 to
100 and on the psychological impact scale from 9 to 45, with lower scores indicating little impact
of MS and higher scores indicating greater impact. In this study, we used these two subscales.
The Parkinson’s disease Questionnaire (PDQ-8). An 8-item self-completed, Parkinson’s
disease specific questionnaire. Each item is rated: 0 (never) to 4 (always/cannot do) in the past
one month. Items are: problems getting around in public, difficulty dressing self, felt depressed,
embarrassed in public due to having disease, problems with close personal relationships, prob-
lems with concentration, unable to communicate with people properly, painful muscle cramps
or spasms. The PDQ-8 Summary Index is expressed as a percentage of the sum of the items
scores on the maximum possible scale score [6, 7].
The Non-Motor Symptoms Questionnaire (NMSQuest-30). A 30-item self-adminis-
tered, Parkinson’s disease specific questionnaire with three response categories (“yes”, “no”
and “don’t know”) for each item. The items are divided into six domains: neuropsychiatric
symptoms, autonomic disorders, disorders of smell, sleep disorders, sensory symptoms and
other symptoms [8, 24]. Total NMSQuest scores range from 0 to 30 based on the total number
of “yes” responses. Scores indicate how many different non-motor symptoms are present.
Other measures. The Extended Disability Status Scale (EDSS) is a single item 10-point cli-
nician rated classification scheme used to assess physical disability for patients with MS [25].
The score range 0 (normal neurological exam) to 10 (death due to MS) with a step increase of
0.5. EDSS 1.0 to 4.5 refer to people with a high degree of ambulatory ability, 5.0 to 9.5 refer to
the loss of ambulatory ability. The modified Hoehn & Yahr scale (HY) is a measure of the
symptom progression for IPD, MSA and PSP, where stage 3, 4 and 5 respectively indicate
“mild to moderate bilateral disease; some postural instability; physically independent”, “Severe
disability; still able to walk or stand unassisted” and “Wheelchair bound or bedridden unless
aided” [21]. The EuroQoL 5-Dimensions Questionnaire (EQ-5D) is a widely used generic
instrument of health related quality of life [26]. It includes a descriptive part and a visual ana-
logue scale for current health status (EQ-VAS, score range: 0 = the worst possible health status
to 100 = the best possible health status). The descriptive part consists of five questions asking
respondent’s current health in dimensions, i.e. mobility, self-care, usual activities, pain/discom-
fort and anxiety/depression. The rating on these five items is converted to a value (EQ-index)
using the UK weighting schemes [27].

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 4 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

Data Collection and Follow Up Schedule

Details of data collection and follow up schedule have been described elsewhere [11, 14, 28]. In
brief, data were collected through face-to-face interviews in both studies. Socio-demographic
information, clinical characteristics and the summary scores of patient reported measures were
recorded at baseline. The patients in both studies were followed up at 12 weeks, and the
patients with MS had a further data collection point at 6 weeks.

Statistical Analysis
Total scores and item specific scores were described using mean and standard deviation (SD).
Individual items were examined for their frequency distribution and expressed as a percentage.
Floor and ceiling effects were measured by the number of respondents receiving the low (0–2)
and high range (30–32) summary scores, or the minimum (0) and maximum (4) score on the
individual items, respectively. The floor and ceiling effects were deemed present when more
than 15% of patients recorded the extreme scores [29]. Data quality was assessed by the com-
pleteness of responses.
To evaluate the dimensions of the scale, exploratory factor analysis (EFA) and confirmatory
factor analysis (CFA) were used. The EFA was conducted using the maximum likelihood
method with promax rotation. The number of factors were determined by Spree plot and Kai-
ser’s criterion of an Eigen value>1. A factor loading greater than 0.30 was considered signifi-
cant. The factorial structure of the scale was verified by CFA [30]. Several fit indices were
selected to assess the CFA model fit: chi-square test statistics, root-mean-squared error of
approximation (RMSEA), standardised root mean square residual (SRMR), and Comparative
Fit Index (CFI) [31, 32]. RMSEA is a measure of the average of the residual variance and
covariance; good models have RMSEA values that are at or less than 0.08. SRMR is a measure
of the mean absolute value of the covariance residuals; values below 0.05 indicate good fit; CFI
is an index that falls between 0 and 1, with values greater than 0.95 considered being indicators
of good fitting models.
The internal consistency of the scale was calculated with Cronbach's alpha coefficient (mini-
mum acceptable value for alpha was 0.7) [33]. Pearson’s correlation coefficient (r, 95%CI) was
used to evaluate the concurrent and the convergent validity. The IPOS Neuro-S8 was anticipated
to be more strongly correlated with physical domain than non-physical domain, e.g. in MS,
higher in MSIS physical than MSIS psychological, and in IPD, MSA & PSP higher in NMSQuest
than in PDQ-8. The interpretation of an absolute r value is as follows: 0.35—low or weak corre-
lations, 0.36 to 0.67—modest or moderate correlations, and 0.68 to 1.0—strong or high correla-
tions [34]. Discriminant validity was assessed using the known-groups validation method.
Patients with higher level of disability were expected a priori to have a higher need for palliative
care compared with patients with lower level of disability (EDSS<8 versus 8+ in MS, HY3 versus
HY4&5 in IPD, MSA & PSP, respectively). The mean difference between groups was compared
with the two sample t-test.
The test-retest reliability was explored with the intraclass correlation coefficient (ICC),
using the baseline and six-week data from the MS trial only. The responsiveness of the IPOS
Neuro-S8 was evaluated using the change scores from baseline to 12 week follow up. Two
aspects were assessed and compared using independent t-test: responsiveness to intervention
(MS trial), and to disease trajectory by diagnosis (IPD versus MSA & PSP). The responsiveness
in MS sample was also analysed with effect size, interpreted using Cohen’s rules [35]. Longitu-
dinal concurrent validity was tested using Pearson’s correlation coefficient r on the change
scores between measures.

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 5 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

The minimal clinically important difference (MCID) was derived using the distribution
based approach [36]. The MCID was estimated by the standard error of measurement (SEM),
one half and one third of the standard deviation of the change score [37, 38]. Strictly speaking,
SEM is a measure of the precision of the scale. As a precision measure, it has an arbitrary crite-
rion (e.g. <1/2 or 1/3 SD) to be deemed as appropriate [39]. However, 1 SEM can be used as an
approximation to the MID, along with other distribution-based methods for interpreting
PROMs that produce dissimilar results [40].
All statistical tests were two-sided, P<0.05 was considered statistically significant. Data
were analysed by SAS 9.4 (SAS Institute Inc, Cary, USA).

Results
Characteristics of the Study Participants
The pooled sample consists of 134 patients (Table 1). The average age of the study participants
was 62 years old (SD = 12). Patients with MS were younger than patients with IPD, MSA or
PSP (53 vs 66 years, p<0.001). Most patients with MS were women (69%), while gender was
more balanced in IPD, MSA and PSP sample (55% vs 45%). 94% of patients with MS were in
primary- or secondary-progressive stage (44% and 50%, mean EDSS score = 7.8(SD = 1.0)).
Patients with MSA and PSP (71% in H&Y stage 4 & 5) were in more advanced disability stages
than those with IPD (60% in H&Y stage 4 & 5). The disease duration was significantly shorter
in MSA and PSP (4 years) than in MS (15.5 years) or IPD (12 years) (p<0.0001).

Scale- and Item-Level Descriptive Statistics and Frequency Distribution

The mean total score of the IPOS Neuro-S8 was 7.8 (SD = 4.8); the score distribution appeared
to be near normal with a kurtosis of -0.008 and a skewness of 0.54 (p value for Shapiro-Wilk
test<0.001) (Table 2). All of the five response categories (0–4) were used in both conditions.
Only a small proportion of patients scored the highest possible score (4) on the individual

Table 1. Demographic and clinical characteristics in MS and IPD, MSA & PSP samples.
Characteristics Value All(n = 134) MS(n = 52) IPD, MSA & PSP
All(n = 82) IPD(n = 50) MSA(n = 17) PSP(N = 15)
Age Mean(SD) 61.6(12) 53.0(10) 66(9) 66.5(8) 67(8) 69(12)
Median(min, max) 64(33,86) 52(33,75) 68(38, 86) 67(40,80) 68(51,81) 71(38,86)
Gender Woman% 73(54%) 36(69%) 37(45%) 23(46%) 6(35%) 8(53%)
Diagnosis Primary: secondary: other - 23:26:3 NA
Disease stage/progressiveness Mean EDSS score(SD) - 7.8(1.0)
Median EDSS (min-max) - 8.0(5.5–9.5)
H&Y 3:4:5 - - 29:37:34% 40:38:22% 12:41:47% 13:27:60%
Duration of illness since diagnosis Mean(SD) 12.4(8.5) 17.2(9.6) 9.3(6.0) 11.9(5.8) 5.9(4.0) 4.6(2.7)
Median(min, max) 12(0,54) 15.5(0,54) 8.0(1, 25) 12(1,25) 5(1,14) 4(1,11)
Employment Retired 72(54%) 11(21%) 61(74%) 38(76%) 11(65%) 12(80%)
In employment/self employed 7(5%) 1(2%) 6(7%) 6(12%) 9(0%) 0(0%)
Unable to work(due to illness) 55(41%) 40(77%) 15(18%) 6(12%) 6(35%) 3(20%)
Ethnic group White% 113(84%) 47(90%) 66(80%) 37(74%) 17(100%) 12(80%)
Living arrangement Lives with carer 104(78%) 41(79%) 63(77%) 36(72%) 14(82%) 13(87%)
Lives alone 23(17%) 9(17%) 14(17%) 10(20%) 2(12%) 2(13%)
Other 7(5%) 2(4%) 5(6%) 4(8%) 1(6%) 0(0%)

Hoehn & Yahr stage refers to: 3 Mild to moderate bilateral disease; some postural instability; physically independent; 4 Severe disability; still able to walk or
stand unassisted; 5 Wheelchair-bound or bedridden unless aided.

doi:10.1371/journal.pone.0165379.t001

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 6 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

items. Other than nausea and vomiting, all items were roughly symmetrical around the score
range 0–3. The mean total score of the IPOS Neuro-S8 was higher in IPD, MSA & PSP than in
MS (9.0 vs 6.0, p<0.001); the average scores for the pain, shortness of breath, mouth problems
and difficulty in sleep items followed a similar pattern and were significantly different between
MS and IPD, MSA & PSP (p<0.05). No difference was found in spasms, nausea, vomiting and
constipation scores by condition (p>0.09).

Missing Data, Floor and Ceiling Effects

The rate of missing data for the IPOS Neuro-S8 was low, with only 5.8% of total scores missing
in the MS sample and no missing for IPD, MSA & PSP patients (Table 2). The items missing in
the MS data was in a small range (3.8–5.8%). The missing pattern appeared to be scale missing
and was not related to individual items or observed factors. There was evidence for a borderline
floor effect, with 14.9% of the patients scoring 0–2; the floor effect was evident in patients with
MS (28.9%) but not for IPD, MSA & PSP patients (6.1%). There was no ceiling effect for the
total score in the combined data or for both diagnostic groups. The floor effect was present for
all items, with 23 to 91% of the patients receiving the lowest possible score (0).

Factor Structure
The EFA indicated a two-factor structure (P = 0.67, X2 = 10.2, df = 13), with factor 1 loaded
with five items and factor two with three items (Table 3). All fit indices in the CFA met accept-
able level model fit. The two factors were moderately correlated (Pearson’s r = 0.48), suggesting
two associated but distinct constructs. The factor loadings on spasms (0.31 for factor 1) and on
constipation (0.28 for factor 2) were weak.

Internal Consistency and Test-Retest Reliability

The Cronbach’s alpha for the combined sample was 0.66; it was higher in MS (0.70) than in
patients with IPD, MSA or PSP (0.60). Given that the IPOS Neuro-S8 had only eight items
with two moderately correlated factors, we interpreted the internal consistency of the scale as
acceptable. The test-retest analysis revealed the agreement between the baseline and the week 6
assessments was fair, with an ICC value of 0.58 (95%CI: 0.28–0.83). The baseline total scores
were not significantly different from those of week 6 (p for paired t-test = 0.80).

Construct Validity
The IPOS Neuro-S8 was moderately correlated with the core POS (Pearson r 0.50, 95%CI: 0.36–
0.62) (Table 4). The correlation between IPOS Neuro-S8 and core POS were similar (r = 0.58) in
the two diagnostic groups. The correlation with existing disease specific measures (MSIS-Physi-
cal/ Psychological, PDQ, NMSQuest) was also moderate, ranging from 0.46 to 0.59 but statisti-
cally significant (p<0.05). The IPOS Neuro-S8 was negatively correlated with both EQ-5D and
health status (-0.39 and -0.25), and positively correlated with the HADS-14 (0.29).
The mean scores were different according to the EDSS score in MS sample (4.8 for EDSS
<8 versus 6.9 for EDSS 8+), the difference did not reach the significance level (p = 0.08). In the
IPD, MSA & PSP group, the H&Y staging system distinguished patients well, with higher
scores corresponding to more severe symptoms (7.2 H&Y stage 3 vs 9.7 H&Y stage 4 & 5), and
the difference was statistically significant (p = 0.020).
The two factors identified in the CFA were both strongly correlated with the total score
(Pearson’s r = 0.91 for Factor 1 and 0.71 for Factor 2); but were only weakly related to each

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 7 / 15

 Table 2. Descriptive statistics of IPOS Neuro-S8 scores in the combined data and by condition.
Item All (N = 134) Multiple Sclerosis (N = 52) IPD,MSA & PSP (N = 82)
Mean(SD) % Mean(SD) % Mean(SD) %
0 1 2 3 4 missing 0 1 2 3 4 missing 0 1 2 3 4 missing
Total* 7.8(4.8) 14.9 0.0 1.5 6.0(4.7) 28.9 0.0 5.8 9.0(4.5) 6.1 0.0 0.0
Pain 1.8(1.2) 23.1 12.7 29.1 29.1 4.5 1.5 1.5(1.3) 30.8 21.2 19.2 19.2 5.8 3.8 2.0(1.1) 18.3 7.3 35.4 35.4 3.7 0.0
Spasms 1.4(1.2) 34.3 11.9 33.6 16.4 2.2 1.5 1.4(1.1) 26.9 21.2 32.7 11.5 3.8 3.8 1.4(1.2) 39.0 6.1 34.1 19.5 1.2 0.0

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016

Shortness of breath 0.7(1.0) 57.5 19.4 14.9 6.0 0.7 1.5 0.3(0.7) 71.2 21.2 1.9 0.0 1.9 3.8 0.9(1.1) 48.8 18.3 23.2 9.8 0.0 0.0
Nausea 0.4(0.9) 73.9 11.9 7.5 3.7 1.5 1.5 0.4(0.9) 71.2 15.4 5.8 1.9 1.9 3.8 0.5(0.9) 75.6 9.8 8.5 4.9 1.2 0.0
Vomiting 0.2(0.6) 91.0 2.2 2.2 2.2 0.7 1.5 0.2(0.7) 86.5 3.8 3.8 0.0 1.9 3.8 0.1(0.6) 93.9 1.2 1.2 3.7 0.0 0.0
Constipation 1.2(1.2) 44.0 15.7 18.7 17.9 1.5 2.2 0.9(1.2) 50.0 17.3 13.5 11.5 1.9 5.8 1.3(1.2) 40.2 14.6 22.0 22.0 1.2 0.0
Spasms 1.4(1.2) 34.3 11.9 33.6 16.4 2.2 1.5 1.4(1.1) 26.9 21.2 32.7 11.5 3.8 3.8 1.4(1.2) 39.0 6.1 34.1 19.5 1.2 0.0
Difficulty in sleep 1.1(1.3) 47.8 11.9 20.1 14.9 3.0 2.2 0.8(1.3) 59.6 9.6 11.5 7.7 5.8 5.8 1.3(1.2) 40.2 13.4 25.6 19.5 1.2 0.0
Mouth problems 1.1(1.2) 45.5 17.9 18.7 13.4 3.0 1.5 0.5(1.0) 73.1 5.8 13.5 1.9 1.9 3.8 1.5(1.2) 28.0 25.6 22.0 20.7 3.7 0.0

*0–2 for the minimum, and 30–32 for the maximum in case of total score; 0 for the minimum and 4 for the maximum in item-specific score. Floor or ceiling effects are considered to be
present if more than 15% of respondents fell in the extreme score categories[29].

doi:10.1371/journal.pone.0165379.t002

8 / 15
Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions
 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

Table 3. Standardized Factor Loadings (Standard Error, SE), factor correlation (SE) and the fit indi-
ces in the confirmatory factor analysis of the IPOS Neuro-S8.
Item Factor1(5 items) Factor2(3 items)
Pain 0.61(0.09)
Shortness of breath 0.43(0.10)
Nausea 0.99(0.10)
Vomiting 0.64(0.08)
Constipation 0.28(0.09)
Spasms 0.31(0.10)
Difficulty in sleep 0.49(0.09)
Mouth problems 0.49(0.09)
Correlation of the two factors 0.48(0.11)
Fit statistics P = 0.19(Chi-square = 24.2, DF = 19)
Standardised RMR(SRMR) = 0.057
RMSEA = 0.046
Comparative fit index = 0.9634
doi:10.1371/journal.pone.0165379.t003

other (Pearson’s r = 0.35). The association pattern was similar in both conditions, providing
further proof that the two factors reflect a different but related symptom profile.

Responsiveness, MID and Longitudinal Construct Validity

The IPOS Neuro-S8 was responsive to change. This was more evident for patients with MS
when comparing the patients in the intervention arm to those in the control arm. The t-test
result was statistically significant (p = 0.030, t = 2.23, df = 46). The score change in IPD and the
combined MSA & PSP was in the same direction (mean change -0.45 and -0.88), but none of
the changes reached significance level and the changes between groups were also not significant
(p = 0.54, t = 0.62, df = 70). The effect size (ES) of the short-term palliative care intervention in
MS patients was small (<0.5). The MID was 3.1 as measured by SEM, greater than half and a
third of SD (1.7 and 1.1).
Longitudinally, the IPOS Neuro-S8 showed reasonable correlation with the core POS and
with the disease specific measures. However, few of them reached statistical significance due to
the sample size reduction caused by attrition.

Discussion
This study provided initial evidence on the cross sectional and longitudinal reliability and
validity of the IPOS Neuro-S8, a new PCOM for people with progressive LTNCs. The IPOS
Neuro-S8 showed acceptable to promising psychometric properties for use across neurological
conditions, including advanced MS, late stage progressive IPD, PSP and MSA. While most
studies involving severely ill people often suffer from small numbers [41, 42], this evaluation
was based on a relatively large sample (N = 134) therefore making it statistically more robust.
The reliability of the IPOS Neuro-S8 appears adequate based on a Cronbach’s alpha value of
0.66 with only 8 items, and supported by the two factor solution and their moderate correlation
(Pearson’s r = 0.48) [43, 44]. The IPOS Neuro-S8 is correlated reasonably well with the non-
condition specific core POS. The core POS, as a multidimensional measure, only two out of its
ten items are questions about pain and one other physical symptom. Therefore, a high correla-
tion between IPOS Neuro-S8 and core POS is not anticipated.
The correlation between IPOS Neuro-S8 and the diagnosis specific scales (MSIS-29,
NMSQuest-30 and PDQ-8) was as expected (Pearson’s r range: 0.46–0.59), e.g. higher in physical

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 9 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

Table 4. Cross-sectional and longitudinal reliability and validity of the IPOS Neuro-S8.
Psychometric properties Measure Time All (N = 134) MS (N = 52) IPD,MSA &PSP (N = 82)
scale
Cross-sectional
Internal consistency (Cronbach alpha) Full scale T0 0.66 0.70 0.60
Construct validity
Concurrent (Pearson’s r, 95%CI) Core POS T0 0.50(0.36 to 0.58(0.36 to 0.74) 0.58(0.41 to 0.71)
0.62)
MSIS physical T0 - 0.59(0.36 to 0.75) -
MSIS T0 - 0.46(0.20 to 0.66) -
psychological
NMSQuest T0 - - 0.58(0.43 to 0.71)
PDQ-8 T0 - - 0.48(0.29 to 0.63)
Discriminative or known group (Mean EDSS (<8 vs 8+) T0 - 4.8(2.1) vs 6.9(5.6)1 -
(SD))
H&Y 3 vs 4&5 T0 - 7.2(4.1) vs 9.7(4.5)2
Convergent (Pearson’s r, 95%CI) Factor 1 vs total T0 0.92(0.88 to 0.91(0.85 to 0.95) 0.91(0.86 to 0.94)
score 0.94)
Factor 2 vs total T0 0.71(0.62 to 0.73(0.57 to 0.84) 0.71(0.58 to 0.80)
score 0.79)
Factor 1 vs Factor T0 0.37(0.22 to 0.40(0.14 to 0.61) 0.35(0.14 to 0.52)
2 score 0.51)
Longitudinal
Test-retest reliability ICC (95%CI) T0 vs - 0.62(0.36 to 0.82) -
wk6
Construct validity
Concurrent (Pearson’s r, 95%CI) Core POS T0 to 0.30(0.12 to 0.16(-0.13 to 0.42) 0.56(0.38 to 0.70)
wk12 0.45)
MSIS physical T0 to - 0.41(0.12 to 0.63) -
wk12
MSIS T0 to - 0.05(-0.25 to 0.33) -
psychological wk12
NMSQuest T0 to - - 0.18(-0.06 to 0.39)
wk12
PDQ-8 T0 to - - 0.27(0.03 to 0.47)
wk12
Responsiveness(change score) Mean (SD, 95%CI) T0 to -0.42(3.4, -1.02 -0.15(4.1, -1.33 to 1.04) -0.60(2.8, -1.26 to 0.06)
wk12 to 0.19)
To intervention: Intervention vs - -1.31(4.3, -3.04 to 0.43) vs -
standard care 1.23(3.4, -0.28 to 2.74)3
To disease trajectory by type of - -0.45(2.9, -1.29 to 0.40) vs
diagnosis (IPD vs MSA & PSP) -0.88(2.7, -1.99 to 0.23)4
Minimally important difference(MID)*: SEM T0 to 3.1 2.9 2.9
distribution based approach wk12
1/2 SD T0 to 1.7 2.1 1.4
wk12
1/3 SD T0 to 1.1 1.4 0.9
wk12

1: p for two sample t-test = 0.08

2: p for two sample t-test = 0.020
*Standard error of measurement (SEM) = SDbaseline × sqrt (1-ICC). The MID measures for MS group was based on the patients in the intervention group
only.
3: mean (sd, 95%CI) for intervention and standard care group, respectively. The difference was significant (p = 0.030).
4: mean (sd, 95%CI) for IPD and the MSA&PSP combined, respectively. The difference was not statistically significant (p = 0.54).

doi:10.1371/journal.pone.0165379.t004

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 10 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

and lower in psychological aspects. One reason for the moderate association may be that none of
these measures were designed specifically for palliative care needs or for people severely affected
by their illness; thus, intrinsically these measures may not capture the same needs as those cap-
tured by the IPOS Neuro-S8 [5, 7, 8, 23]. The time frames used in the disease specific measures
may be another reason. For example, NMQuest-30 and PDQ-8 ask symptoms experienced over
the past month (or 4 weeks) and the MSIS is based on the past two weeks. In contrast, the IPOS
Neuro-S8 asks about the past three days, aiming to capture the immediate care needs. For
patients with neurological conditions who require palliative care input, the disease trajectory is
more unpredictable and more frequent assessments are recommended as best practice [9].
Our results support the continued development and validation of the IPOS Neuro-S8. If
proved by further validation studies, it can be a useful outcome measure in research settings, par-
ticularly for patients with progressive LTNCs, as it is brief and shows promising psychometric
properties. It may also facilitate the clinical use of the IPOS Neuro-S8 as a screening measure in
non-palliative care settings, to increase the opportunities of identifying patients with palliative
care needs and trigger the referral process. In palliative care clinical practice, it may be used to
monitor care intervention outcomes and disease progression. At the organisation level, it can be
a simple, standardised auditing tool. It also makes patient’s self-monitoring possible.
Several limitations are noted in our study. We could not test face validity and content valid-
ity in this study, but all eight items of the IPOS Neuro-S8 were important for patients and rele-
vant in clinical practices [9, 18]. Although having been able to assess a range of psychometric
properties, this evaluation was based on two existing datasets using very different study
designs–a longitudinal observational study and a randomised clinical trial. Previous research
revealed that eligibility criteria for patient selection in clinical trials can be very restrictive,
therefore these patient samples may not be as representative as those from observational stud-
ies [45]. Some very “severe” patients were excluded as well–less than 5% got the highest possi-
ble score on the individual items; we did not have data for other prevalent neurological
conditions that may have a higher need for palliative care, e.g. motor neuron diseases. As a con-
sequence of the high attrition rate and the loss of statistical power, the longitudinal psychomet-
ric evaluation should be treated as preliminary. Furthermore, the psychometric testing in this
study was primarily based on classical test theory; it may be of interest to compare how the
results differ from those derived from modern measurement theories.

Conclusions
In conclusion, our results demonstrated the acceptable to promising psychometric properties of
the IPOS Neuro-S8 in two common long term neurological conditions. The IPOS Neuro-S8
showed good data quality, borderline floor effects and no ceiling effects, good internal consistency
and moderate test-retest reliability. A moderately correlated two-factor structure was confirmed.
The IPOS Neuro-S8 was moderately correlated with both the generic core POS and the condition
specific measures (MSIS-29, NMSQuest-30, PDQ-8). There was evidence for discriminant validity
in IPD, MSA and PSP (by disease progression). Preliminary evidence suggested good responsive-
ness and longitudinal construct validity but needs further research. Future work needs to confirm
these findings with larger samples and its usefulness in wider LTNC disease groups.

Supporting Information
S1 File. The 8 key symptoms of the Integrated Palliative Outcome Scale for patients with
long term neurological condition (IPOS Neuro-S8).
(DOCX)

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 11 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

S2 File. The Integrated Palliative Outcome Scale for patients with long term neurological
condition (IPOS Neuro).
(DOCX)

Acknowledgments
OPTCARE Neuro is a multicentre clinical trial funded by the National Institute for Health
Research, Health Services and Delivery Research Programme (NIHR HS&DR, 12/130/47), led
by King’s College London, Cicely Saunders Institute, Department of Palliative Care, Policy &
Rehabilitation, UK. The views expressed in this publication are those of the authors and not
necessarily those of the NHS, the National Institute for Health Research or the Department of
Health.
Co-Chief Investigators: Higginson, Gao.
Co-Investigators: Al-Chalabi, Benz, Burman, Byrne, Chaudhuri, Crosby, Evans, Hotopf,
Jackson, Leigh, McCrone, Murtagh, Pickles, Silber, Wilcock, Young.
Project Manager: Hepgul, van Vliet (1st Apr 14 to 1st Apr 15).
SITE PIS: Brighton–Lindsay; Cardiff–Byrne; Liverpool–Young; London–Burman, Higgin-
son, Murtagh; Nottingham–Crosby.
Trial Statisticians: Gao, Pickles.
Health Economists: McCrone, Yi.
Collaborators: Turner-Stokes, C Murphy.
Intervention Leads: Nottingham–Crosby; London—Bajwah, Dawkins; Liverpool–Groves;
Cardiff–Byrne; Brighton—Lindsay.
Optcare Neuro Team Members: Prof Ammar Al-Chalabi, Dr Sarah Awan, Dr Sabrina Baj-
wah, Dr Cynthia Benz, Dr Rachel Burman, Dr Anthony Byrne, Prof K Ray Chaudhuri, Dr Vin-
cent Crosby, Ms Joanna Davies, Ms Marsha Dawkins, Dr Catherine Evans, Ms Mim Evans, Ms
Sarah Farnan, Dr Wei Gao, Dr Karen Groves, Dr Nilay Hepgul, Prof Irene Higginson, Prof
Matthew Hotopf, Dr Diana Jackson, Mrs Paramjote Kaler, Dr Nigel Leigh, Dr Fiona Lindsay,
Ms Cathann Manderson, Prof Paul McCrone, Ms Caroline Murphy, Dr Fliss E M Murtagh,
Mrs Jenifer Newton, Ms Caty Pannell, Ms Louise Pate, Prof Andrew Pickles, Dr Eli Silber, Miss
Debbie Tonkin, Prof Lynne Turner-Stokes, Dr Liesbeth van Vliet, Dr Andrew Wilcock, Dr
Deokhee Yi, Prof Carolyn Young.
Study Steering Committee Members: Prof Marie Fallon (Chair), Dr Cynthia Benz, Prof
Mogens Groenvold, Prof William Hollingworth, Ms Denise Howel, Prof Huw Morris, Mr Fos-
ter Murphy, Dr Diane Playford, Prof Julia Riley, Prof Jane Seymour, Dr Andrew Wilcock.
Data Monitoring And Ethics Committee Members: Prof Mike Bennett (Chair), Prof Gunn
Grande, Dr David Oliver, Prof Raymond Voltz, Prof Stephen Walters.
Patient And Public Involvement Committee Members: Dr Cynthia Benz, Mr Sanjay
Chadha, Mr David Charlton, Mr Colin Fellows, Ms Helen Findlay, Mrs Savita Jain.
The UKCRC-registered King’s Clinical Trials Unit at King’s Health Partners is part funded
by the National Institute for Health Research (NIHR) Biomedical Research Centre for Mental
Health at South London and Maudsley NHS Foundation Trust and King’s College London and
the NIHR Evaluation, Trials and Studies Coordinating Centre (NETSCC). We would like to
thank Ms Joanna Kelly for the management of the eCRF database, and Mrs Beverley White-
Alao for providing advice on the Trial Management process.
The Collaboration for Leadership in Applied Health Research and Care (CLAHRC) South
London is part of the National Institute for Health Research (NIHR), and is a partnership
between King’s Health Partners, St. George’s, University London, and St George’s Healthcare
NHS Trust. This publication is independent research supported by the National Institute for

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 12 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

Health Research Collaboration for Leadership in Applied Health Research & Care Funding
scheme. The views expressed in this publication are those of the author(s) and not necessarily
those of the NHS, the National Institute for Health Research or the Department of Health.
Prof Irene J Higginson is an NIHR Senior Investigator.
This study involved datasets from two studies: 1) a randomised Phase II trial evaluating
effectiveness of a short-term palliative care intervention in people severely affected by multiple
sclerosis (MS), and 2) a longitudinal community based observational study in idiopathic Par-
kinson’s disease(IPD), multiple system atrophy(MSA) and progressive supranuclear palsy
(PSP).
We thank the MS Society who funded the MS study (MS Society, 676/01); the staff involved
in design, data collection, interviews and other aspects of the study, including Prof. Peter Nigel
Leigh, Prof. Paul McCrone, Polly Edmonds, Sam Hart, Tariq Saleem, Bella Vivat, Troy Cart-
wright, Gay Foxwell, Barbara Gomes, Farida Malik and Michael Walton; and the Project Advi-
sory Committee (Keith Andrews, Fiona Barnes, Cynthia Benz, Colin Campbell, Sharon
Haffenden, Martin Hunt, Robin Luff, David Oliver, Michael Ritchie, Sally Plumb and Carolin
Seitz) that oversaw the development of methods and trial conduct. The MS team members at
King’s College Hospital, as well as at Lambeth, Southwark and Lewisham PC, who referred and
collaborated, are greatly acknowledged. Most importantly, we thank the patients and families
for sharing their thoughts, experiences and concerns with us.
We thank Caty Pannell and Frances who helped to collect and enter data and follow-up
interviews for IPD, MSA & PSP study. We thank Parkinson’s UK and the PSP Society who
helped the plan the aims and identified individuals affected by Parkinson’s and related disor-
ders to form a consumer group who advised on the IPD, MSA & PSP study. We thank the clini-
cal staff who helped to identify patients, including Anne Martin, Julia Johnson; the Project
advisory Committee who oversaw the development of methods; Prof Lynne Turner-Stokes,
Prof Richard Brown, Dr Nora Donaldson and Dr Chris Clough who helped to design and plan
the study; and most importantly the patients and families for sharing their thoughts, experi-
ences and concerns with us.

Author Contributions
Conceived and designed the experiments: WG IJH.
Analyzed the data: WG IJH.
Wrote the paper: WG VC AW RB ES NH KRC IJH.

References
1. Chahine LM, Malik B, Davis M. Palliative care needs of patients with neurologic or neurosurgical condi-
tions. Eur J Neurol. 2008; 15(12):1265–72. doi: 10.1111/j.1468-1331.2008.02319.x PMID: 19049542.
2. NEoLCP, NCPC. End of life care in long term neurological conditions: A framework for implementation.
2011.
3. Department of Health. The National Service Framework for Longterm Conditions. London: 2005.
4. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute (PCORI) national pri-
orities for research and initial research agenda. JAMA. 2012; 307(15):1583–4. doi: 10.1001/jama.
2012.500 PMID: 22511682.
5. Hobart J, Lamping D, Fitzpatrick R, Riazi A, Thompson A. The Multiple Sclerosis Impact Scale (MSIS-
29): a new patient-based outcome measure. Brain. 2001; 124(Pt 5):962–73. PMID: 11335698.
6. Jenkinson C, Fitzpatrick R, Peto V, Greenhall R, Hyman N. The Parkinson’s Disease Questionnaire
(PDQ-39): development and validation of a Parkinson’s disease summary index score. Age Ageing.
1997; 26(5):353–7. PMID: 9351479.

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 13 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

7. Jenkinson C, Fitzpatrick R, Peto V, Greenhall R, Hyman N. The PDQ-8: Development and validation of
a short-form parkinson’s disease questionnaire. Psychology & Health. 1997; 12(6):805–14. doi: 10.
1080/08870449708406741
8. Chaudhuri KR, Martinez-Martin P, Schapira AH, Stocchi F, Sethi K, Odin P, et al. International multi-
center pilot study of the first comprehensive self-completed nonmotor symptoms questionnaire for Par-
kinson’s disease: the NMSQuest study. Mov Disord. 2006; 21(7):916–23. Epub 2006/03/21. doi: 10.
1002/mds.20844 PMID: 16547944.
9. Marie Curie. Triggers for palliative care: Improving access to care for people with diseases other than
cancer. London: 2015 06/2015. Report No.
10. Saleem TZ, Higginson IJ, Chaudhuri KR, Martin A, Burman R, Leigh PN. Symptom prevalence, sever-
ity and palliative care needs assessment using the Palliative Outcome Scale: a cross-sectional study
of patients with Parkinson’s disease and related neurological conditions. Palliat Med. 2013; 27(8):722–
31. Epub 2012/12/05. doi: 10.1177/0269216312465783 PMID: 23208011.
11. Edmonds P, Hart S, Wei G, Vivat B, Burman R, Silber E, et al. Palliative care for people severely
affected by multiple sclerosis: evaluation of a novel palliative care service. Mult Scler. 2010; 16
(5):627–36. Epub 2010/03/23. doi: 10.1177/1352458510364632 PMID: 20305044.
12. Higginson IJ, McCrone P, Hart SR, Burman R, Silber E, Edmonds PM. Is short-term palliative care
cost-effective in multiple sclerosis? A randomized phase II trial. J Pain Symptom Manage. 2009; 38
(6):816–26. Epub 2009/10/17. doi: 10.1016/j.jpainsymman.2009.07.002 PMID: 19833477.
13. Sleeman KE, Higginson IJ. A psychometric validation of two brief measures to assess palliative need
in patients severely affected by multiple sclerosis. J Pain Symptom Manage. 2013; 46(3):406–12.
Epub 2012/12/01. doi: 10.1016/j.jpainsymman.2012.08.007 PMID: 23195389.
14. Higginson IJ, Gao W, Saleem TZ, Chaudhuri KR, Burman R, McCrone P, et al. Symptoms and quality
of life in late stage Parkinson syndromes: a longitudinal community study of predictive factors. PLoS
One. 2012; 7(11):e46327. Epub 2012/11/13. doi: 10.1371/journal.pone.0046327 PMID: 23144781;
PubMed Central PMCID: PMCPMC3492372.
15. Siegert RJ, Gao W, Walkey FH, Higginson IJ. Psychological well-being and quality of care: a factor-
analytic examination of the palliative care outcome scale. J Pain Symptom Manage. 2010; 40(1):67–
74. Epub 2010/05/18. doi: 10.1016/j.jpainsymman.2009.11.326 PMID: 20471782.
16. Hearn J, Higginson IJ. Development and validation of a core outcome measure for palliative care: the
palliative care outcome scale. Palliative Care Core Audit Project Advisory Group. Qual Health Care.
1999; 8(4):219–27. Epub 2000/06/10. PMID: 10847883; PubMed Central PMCID: PMCPMC2483665.
17. Bausewein C, Le Grice C, Simon S, Higginson I. The use of two common palliative outcome measures
in clinical care and research: a systematic review of POS and STAS. Palliat Med. 2011; 25(4):304–13.
Epub 2011/04/06. doi: 10.1177/0269216310395984 PMID: 21464119.
18. Turner-Stokes L, Sykes N, Silber E, Guideline Development G. Long-term neurological conditions:
management at the interface between neurology, rehabilitation and palliative care. Clin Med. 2008; 8
(2):186–91. PMID: 18478869.
19. Turner-Stokes L. Long-term neurological conditions: management at the interface between neurology,
rehabilitation and palliative care: National Guidelines. London: 2008 03/2008. Report No.
20. Cleeland C. Symptom burden: multiple symptoms and their impact as patient-reported outcomes. J
Natl Cancer Inst Monogr. 2007; 37:6.
21. Hoehn MM, Yahr MD. Parkinsonism: onset, progression, and mortality. Neurology. 1967; 17(5):427.
doi: 10.1212/wnl.17.5.427 PMID: 6067254
22. Saleem T, Leigh PN, Higginson IJ. Symptom prevalence among people affected by advanced and pro-
gressive neurological conditions—a systematic review. J Palliat Care. 2007; 23(4):291–9. Epub 2008/
02/07. PMID: 18251448.
23. Riazi A, Hobart JC, Lamping DL, Fitzpatrick R, Thompson AJ. Multiple Sclerosis Impact Scale (MSIS-
29): reliability and validity in hospital based samples. J Neurol Neurosurg Psychiatry. 2002; 73(6):701–
4. Epub 2002/11/20. PMID: 12438473; PubMed Central PMCID: PMCPMC1757371. doi: 10.1136/
jnnp.73.6.701
24. Chaudhuri KR, Sauerbier A, Rojo JM, Sethi K, Schapira AH, Brown RG, et al. The burden of non-motor
symptoms in Parkinson’s disease using a self-completed non-motor questionnaire: a simple grading
system. Parkinsonism Relat Disord. 2015; 21(3):287–91. doi: 10.1016/j.parkreldis.2014.12.031 PMID:
25616694.
25. Kurtzke JF. Rating neurologic impairment in multiple sclerosis: an expanded disability status scale
(EDSS). Neurology. 1983; 33(11):1444–52. PMID: 6685237.
26. Brooks R. EuroQol: the current state of play. Health Policy. 1996; 37(1):53–72. PMID: 10158943.

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 14 / 15

 Psychometric Properties of an Outcome Measure for Long Term Neurological Conditions

27. Dolan P. Modeling valuations for EuroQol health states. Med Care. 1997; 35(11):1095–108. PMID:
9366889.
28. Higginson IJ, Costantini M, Silber E, Burman R, Edmonds P. Evaluation of a new model of short-term
palliative care for people severely affected with multiple sclerosis: a randomised fast-track trial to test
timing of referral and how long the effect is maintained. Postgrad Med J. 2011; 87(1033):769–75. Epub
2011/10/08. doi: 10.1136/postgradmedj-2011-130290 PMID: 21978993.
29. McHorney CA, Tarlov AR. Individual-patient monitoring in clinical practice: are available health status
surveys adequate? Qual Life Res. 1995; 4(4):293–307. Epub 1995/08/01. PMID: 7550178.
30. Costello AB, Osborne JW. Best Practices in Exploratory Factor Analysis: Four Recommendations for
Getting the Most From Your Analysis. Practical Assessment, Research & Evaluation. 2005; 10(7).
Epub July 2005.
31. J.B. S, STAGE FK, KING J, NORA A, BARLOW EA. Reporting Structural Equation Modeling and Con-
firmatory Factor Analysis Results: A Review. The Journal of Educational Research. 2006; 99(6):15.
32. Hu Lt, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria
versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal. 1999; 6(1):1–55.
doi: 10.1080/10705519909540118
33. Terwee CB, Bot SD, de Boer MR, van der Windt DA, Knol DL, Dekker J, et al. Quality criteria were pro-
posed for measurement properties of health status questionnaires. J Clin Epidemiol. 2007; 60(1):34–
42. Epub 2006/12/13. doi: 10.1016/j.jclinepi.2006.03.012 PMID: 17161752.
34. Taylor R. Interpretation of the Correlation Coefficient: A Basic Review. Journal of Diagnostic Medical
Sonography. 1990; 6(1):35–9. doi: 10.1177/875647939000600106
35. Cohen J. A power primer. Psychol Bull. 1992; 112(1):155–9. PMID: 19565683.
36. Wright A, Hannon J, Hegedus EJ, Kavchak AE. Clinimetrics corner: a closer look at the minimal clini-
cally important difference (MCID). J Man Manip Ther. 2012; 20(3):160–6. Epub 2013/08/02. doi: 10.
1179/2042618612y.0000000001 PMID: 23904756; PubMed Central PMCID: PMCPMC3419574.
37. Norman GR, Sloan JA, Wyrwich KW. Interpretation of changes in health-related quality of life: the
remarkable universality of half a standard deviation. Med Care. 2003; 41(5):582–92. doi: 10.1097/01.
MLR.0000062554.74615.4C PMID: 12719681.
38. Revicki D, Hays RD, Cella D, Sloan J. Recommended methods for determining responsiveness and
minimally important differences for patient-reported outcomes. J Clin Epidemiol. 2008; 61(2):102–9.
doi: 10.1016/j.jclinepi.2007.03.012 PMID: 18177782.
39. Aaronson N, Alonso J, Burnam A, Lohr KN, Patrick DL, Perrin E, et al. Assessing health status and
quality-of-life instruments: attributes and review criteria. Qual Life Res. 2002; 11(3):193–205. Epub
2002/06/21. PMID: 12074258.
40. Wyrwich KW, Wolinsky FD. Identifying meaningful intra-individual change standards for health-related
quality of life measures. J Eval Clin Pract. 2000; 6(1):39–49. PMID: 10807023.
41. Jordhoy MS, Kaasa S, Fayers P, Ovreness T, Underland G, Ahlner-Elmqvist M. Challenges in pallia-
tive care research; recruitment, attrition and compliance: experience from a randomized controlled
trial. Palliat Med. 1999; 13(4):299–310. Epub 2000/02/05. PMID: 10659099.
42. Evans CJ, Harding R, Higginson IJ. ’Best practice’ in developing and evaluating palliative and end-of-
life care services: a meta-synthesis of research methods for the MORECare project. Palliat Med. 2013;
27(10):885–98. Epub 2013/01/17. doi: 10.1177/0269216312467489 PMID: 23322647.
43. Cortina JM. What Is Coefficient Alpha? An Examination of Theory and Applications. Journal of Applied
Psychology. 1993; 78(1):17.
44. Tavakol M, Dennick R. Making sense of Cronbach’s alpha International Journal of Medical Education.
2011;(2: ):3.
45. Hannan EL. Randomized Clinical Trials and Observational Studies: Guidelines for Assessing Respec-
tive Strengths and Limitations. JACC: Cardiovascular Interventions. 2008; 1(3):211–7. http://dx.doi.
org/10.1016/j.jcin.2008.01.008. doi: 10.1016/j.jcin.2008.01.008 PMID: 19463302

PLOS ONE | DOI:10.1371/journal.pone.0165379 October 25, 2016 15 / 15