Bachelor’s Degree in Bioinformatics (UPF-UPC-UB)

Final Grade Project

Emotional analysis prediction using qualita-

tively representations of multiple psycho-phys-
iological time-series signals.
Guillem Hernández1
Scientific director: 2Beatriz Lòpez

1G. Hernandez, BDBI ESCI-UPF, Barcelona, Spain, on an internship at the University of Girona, Spain
([email protected]); 2B. Lopez, PhD, eXit research group, University of Girona, Spain.
([email protected]).

Abstract
Motivation: This report provides techniques to deal with time-series emotional data. Transforming time-series
data into qualitatively representations enables to reduce the complexity to mine the data while not altering the
dimensionality by having a vocabulary. The method uses several individuals with an audiovisual stimulus and
records multiple psychophysiological signals. Signals can be retrieved with a simple wristband, providing in time
data and auto supervised, the best possible scenario for supervised and deep learning to take part.
Results: Multiple classifications and regression techniques are used to predict different emotions and accomplish
a proper classification, reducing the error of prediction progressively. The methods are based on the mining of
sequence information from the representations, giving insight into pattern recognition and sequence alignment to
take knowledge from the data.
Supplementary information: Supplementary data are available at GitLab link, https://gitlab.com/ghg98/tfg-
exit-emotional_prediction.git

there is still much more work to do to have a precise map-

1 Introduction ping of the emotional spectra of species.
In fact, there is no learned principle to mine this kind of
Emotions are mental states of living beings associated with
signals, always falling in models with no statistical
thoughts, feelings or responses to different stimuli from the
significance, thus relying on simple preprocessing
environment of living beings. Are responsible for the gov-
techniques and feature-driving methods. Preprocessing
ernment of psychological changes influencing the behav-
steps are an essential key to get a significant result.
iour of individuals. From an evolutionary point of view,
Simplistic normalisation procedures do not perform well
emotions such as fear or anger could have had an essential
when dealing with differences among subjects and when
role in natural selection, inducing certain adaptive behav-
isolating the significant signal variations. The fact that
iours in favour of species survival. The fact that there is no
emotions involve subjective experience, cognitive
cognitive process required to have emotional states makes
processes, psychophysiological changes and expressive
most species prone to have emotional pre-cognitive re-
behaviour entails unavoidable individual variability. The
sponses.
weak differentiation among responses leads to rigid models
of emotions, which become more abstract and predictive
Psycho-physiological data has been successfully used to
intercorrelated. Possibilities to have more that one emotion
pinpoint the origin of some familiar, easily-distinctive emo-
at a time and varying them among individuals are not often
tions such as fear, anger, love and happiness. Nevertheless,
G. Hernández et al.

taken into account. There are easily predictable signals and

more significant stimulus.

This report aims at establishing a method to mine psycho-

physiological data to improve emotional prediction.

Fig. 1. Pipeline. Preprocessing and SAX conversion before different

1.1 Related work classification strategies to improve the classificators.

Studies correlate Heart Rate Variability and Power-Spec-

2.1 Data
tra-Density (PSD) analysis with the effect of emotions to
characterise physiological illnesses such as depressions or
panic-disorders (McCraty et al., 1995). Signals retrieved The psycho-physiological data used in this study belong to
with wristbands, such as electrodermal activity (EDA) or the DEAP data set (Database for Emotional Analysis using
skin temperature are key factors to predict depression pat- Physiological Signals). This is a multimodal database for
terns following Hamilton Depression Rating Scale (HDRS) the analysis of human affective states induced by audiovis-
(Ghandeharioun and Fedor et al., 2017). Beyond studies on ual inputs (forty-one-minute long videos). The dataset en-
depression, wearables have been useful in detecting relapse compasses the records of thirty-two healthy participants,
prevention (Kerz et al., 2016), anxiety (McGinnis et al., with the same number of females and males.
2017) drug addiction (Fletcher et al., 2011), stress (Pino To quantitatively describe emotions, the dataset uses a
2014, Choi 2017) or schizophrenia (Torous et al., 2018). modification of Russell’s valence-arousal scale by placing
Such gadgets provide individualised data that allow the emotional states on a two-dimensional plane with arousal
modelling of individuals mental Health. The electrodermal and valence in the axis (Barrett, L. F., & Russell, J. A. et
activity has been used to identify excitement, and respira- al., 1999). While arousal and valence explain most of the
tory volume is used to provide insights into negative-va- variation in the emotional field when included dominance
lence emotions (Wu et al., 2011). and liking as variables to the model, the results improved.
Arousal, valence, dominance and liking scales have a con-
tinuous classification range continuously from 1 to 9. The
1.2 Objectives results are obtained from each participant self-assessment
after each stimulus presentation.
This work aims to establish a method to learn emotional
patterns from psycho-physiological data to enable their pre- The dataset gathers 32 data sources, one for each partici-
diction. To that end, a wide range of problems has been ad- pant, that collects 40 physiological signals for each stimu-
dressed. Within-between variability among samples, di- lus.
mensionality and feature-driving methods (no adapted to
time-series data) makes the mining task difficult, needing DEAP data set, although vast in terms of mixed signals rec-
new methods to have reliable results. Time series is prepro- orded, provide a small number of them that can be used to
cessed and discretised in qualitative representations. Dif- predict emotions in non-experimental environments. Since
ferent methods relying on distance sequence classification the ultimate goal is to predict emotions in a daily life envi-
and pattern mining are used as classificators. Results are ronment, only signals that can be retrieved by a wristband
compared and parameters tested to evaluate the reliability such as “empàtica-e4”: Galvanic skin resistance (GSR),
of classifications. electrocardiogram (ECG) and skin temperature (TEMP)
To cope with these problems, the objective of this project have been used.
is to propose a method capable of shrinking the complexity
of the data with its major drawbacks while enabling an ad-
2.2 Preprocessing
equate classification.
In a first preprocessing step, while participants are watch-
ing the video clips, the study records physiological signals.
2 Materials Methods A smoothing and downsampling from 512Hz to 128Hz re-
Figure 1 shows an overview of the workflow conducted to duces computation time.
address the different problems towards the objective, as-
sembling the different methods used.
Emotional analysis prediction

In a second step, a class value is generated from the arousal, (1)

𝑛
valence, dominance and liking information: class “0” for 𝑤
𝑖
𝑤
low/inactive (1-3.666) response, “1” for a neutral response 𝐶̅ 𝑖 = ∑ 𝐶𝑗
(3.667-6.333) and “2” for hight/active response (6.334- 𝑛 𝑛
𝑗= (𝑖−1)+1
𝑤
9).
Eq. 1. PAA. Calculation of dimensionality reduction blocks in PAA segments.
An oversampling of the class ‘1’ is observed. Therefore a
subsampling method is applied to reach a much more
equally distributed training data. The test-train partitioning To get w dimensions’ transformation of the time-se-
of the data with 10-fold cross-validation gives reliability to ries of n dimensions, data is divided into w equal
the results when performing this procedure. sized “blocks” called episodes (Williams et al., 1986)
defined by a time interval and a qualitative context.
Last pre-processing is related to obtain new features from The mean value of the data falling within a block is
blood volume pressure. Heart Rate (HR) and Heart Rate calculated, and a vector of these values becomes the
Variability (HR) has shown to produce good results in dimensionality-reduced representation
terms of prediction (McCraty, Rollin et al., 1995). A
MAX30100 pulse oximeter filter algorithm extrapolates This normalisation method is called Piecewise
from the blood volume pressure the HR / HRV, detecting Aggregate Approximation (PAA) representation
peaks of higher volume. The algorithm consists of a “dc (Keogh et al., 2001; Yi & Faloutsos, 2000).
removal” and a “mean-median filter”. Classification
strategies take place with the raw ECG signal and the (2) Breakpoints produce equal-sized areas under
filtered HRV. Gaussian distribution, dividing the space of the data.
When the PAA segment falls into the area inherits the
corresponding letter from a precomputed vocabulary
2.3 Qualitative representations
(PAA coefficients have associated símbols
Qualitative symbols, also called primitives by some authors determined by breakpoints). The time series is,
(Rengaswamy, Venkatasubramanian, et al., 1995) repre- therefore, discretised into a string or word of length
sent time series while enabling the extraction of features “w”.
that allow the following classification using other tools,
mainly pattern recognition procedures. The characteristic function can be other rather than
Gaussian, but since time series are physiological
These representations enable not only to reduce the com- signals, the natural distribution can better explain the
plexity of the system but do not alter the dimensionality by fluctuation.
having a vocabulary (a finite set of qualitative descrip-
tions). They are assignations of qualitative symbols from a
precomputed dictionary to equally sized blocks of time se-
ries.

In particular, this work uses Symbolic Aggregate

approXimation (SAX) (Lin et al., 2003). The symbolic
representation procedure has two steps (see Figure 2):

(1) Time series is normalised using a mean filter through

a simple spanning window technique.

A time series C of length n can be represented in a w Fig. 2. Steps performed by SAX algorithm. (a) Raw time-series. (b)
PAA formation and normalisation. (c) Gaussian distribution of space.
dimensional space by a vector 𝐶̅ = 𝑐̅1 ,… , 𝑐̅𝑤 . The ele-
(d) qualitatively discretisation.
ment 𝑖 𝑡ℎ 𝑜𝑓 𝐶̅ is calculated by the following equation
G. Hernández et al.

Although the SAX representations are widely used, they matrix, from where one-neighbour classifier and svm
show a particular drawback. Parameters must be set performs the classification.
manually to minimise the error (see Figure 3). Alphabet
length, PAA segments length, slopes (prominence of the The method uses two distance measurements:
slopes in the result) and stiffness (percentage of the length
“n” of the time series to discretise) are parameters that (1) EpDtw:
must be set before the discretisation. That involves the Dynamic time warping (DTW) (Sankoff and Kruskal
expertise of data scientists since the settings may change in et al., 1983). It uses dynamic programming to align
different datasets. the time series to minimise distance (Bellman and
Stuart, 1962; Sakoe and Chiba, 1978; Silverman and
Morgan, 1990). With the time series using PAA
segments, the algorithm uses a variation (Piecewise
dynamic time warping, ePDTW). It finds the align-
ment by stretching or shrinking the time series along
the time axis, warping them non-linearly (can be used
to find similarity regions). Distance is the accumula-
tion of local distances between corresponding ele-
ments.
A significant drawback is that the user must choose a
compression rate for the dimensionality reduction.

Fig. 3. Granularity effect. (a) If the reduction scale is large (n/w), the (2) qssi:
chance of information loss increases. (b) if it is small, the PAA segments
Qualitatively Sequence Similarity Index (qssi)
show noise and do not adequately reduce dimensionality.
algorithm matches similar items (Cuberos et al.,
2002). Allows greater flexibility by formulating the
Several authors have adopted SAX since its creation. With
alignment in terms of similarity rather than in terms
the symbolic representations, PCA enables the reduction to
of the distance between sequences. Not aligning the
1-dimensional time series and find the characteristic
sequences over the full length, makes the algorithm
patterns (Tanaka et al., 2005). Minimum Description
faster.
Length (MDL) has been used to extract the most frequently
occurring pattern over SAX sequences (Lin et al., 2001).
QSSI - vital signs, proposed in this work, uses as mainframe
Using PLA to compare patterns in a postprocessing step
QSSI to generate an algorithm specially designed for
with SAX-representations has good results (Hung and Anh
psycho-physiological signals, enabling to perform
et al., 2007).
classification over multiple signals at a time for the same
stimulus. The algorithm needs a series of parameters for
2.4 Distance Sequence Classification SAX representations specially tested to reinforce the
There is no learned principle to mine time-series data in an model, adapting it to the type of signals.
efficient way. Calculating the distance between two time-
series could be costly due to dimensionality. Extraction of One-nearest-neighbour classification and svm are not
symbolic representations enables efficiency while still enough if several psycho-physiological signals need to be
gathering primary information. combined to avoid drawbacks of emotional data. Therefore,
Euclidean, Hamming or Manhattan distance measurements new classification methods are proposed, based on a
are sensible to distortions in time, making them unreliable consensus analysis over the classification of the different
when comparing sequences. signals to give a final result.

Alternative to them, QSSI is an algorithm used in QSSI-vs knn majority voting

engineering environments to compare two-time series to It is designed to take as predicted the class with more enti-
classify voltage sags (Gamero Argüello et al., 2019). The ties among the “k” nearest neighbours. Posterior consensus
method proposes two different distance measurements to analysis takes the majority class among the predictions of
calculate over SAX representations and generate a distance all different signals.
Emotional analysis prediction

QSSI-vs knn aggregational voting Prefixspan3 is a modification of the prefixspan algorithm.

The algorithm is based on the time of arrival (distance). A The raw algorithm was too flexible, enabling infinite time
sumatorium of the distances of each possible outcome is misalignments, which makes the prediction unreliable (the
generated. pattern could be found over the total length since the first
Consensus analysis makes a normalisation between symbol is matched). Moreover, it only detects if a pattern
classifications of all the signals. It takes the minimal is present.
distance predicted class.
The prefixspan3 version allows (see Figure 4):
2.5 Pattern sequence classification (text mining)
- Setting a maximum distance to find the pattern, based
Similarity sequence classification is based on the alignment on the percentage of the total length “n” of the se-
of sequences to evaluate the similarity from a structural quence.
point of view rather than distance in (Agrawal and Srikant,
1995). Two approaches are explored in this work: - Detecting the number of appearances for each pattern,
Prefixspan3 and MSA_2. not only whether it is present. That implies giving
more reliability to repeated patterns.
2.4.1 Prefixspan3
The prefixspan3 algorithm allows for a searching method
similar to “bag of words algorithm”, but since sequences
are not real words with a robust meaning, certain flexibility
can be allowed. “Bag of words algorithm” search the num-
ber of appearances of words in different classes, and as-
signs a class to a new instance based on the matching of the
words on it. In a similar method, prefixspan3 finds the dis-
criminant patterns between classes that have higher fre-
quency to compare with each instance; it is a ‘bag of flexi-
ble patterns’ algorithm.

In this work, the algorithm is used on multiple signals. For

each signal separates the balanced training set among the
different classes, and finds the patterns of each class. After-
wards, it finds the discriminant patterns among the different
classes of a signal (the patterns that are only present in each
class). A database of discriminant patterns is performed for
each class, allocating the pattern and the support.

Classification for each test is achieved by calculating the

number of outcomes of each pattern. The similarity index
Fig. 4. Pipeline prefixspan3. The pipeline of the prefixspan3 algorithm is computed using the number of outcomes and the support
gathered with the knn-aggregational voting workflow.
for each pattern. The index is used with the QSSI-
aggregational voting to perform a classification before the
Prefixspan3 is based on prefix-projected Sequential pattern consensus analysis.
mining, Prefixspan (Pei J. et al., 2001). It is used on quali-
tative representations of signals (sequence of symbols). 2.4.2 MSA_2
Multiple sequence alignment, MSA, is a method for align-
The algorithm is used to discover all sequential patterns un-
ing a database of different related sequences.
der a predefined support threshold (percentage of se-
quences presenting the pattern). The algorithm performs
The algorithm pipeline is the following:
well for small datasets. In vast datasets, the time complex-
(1) Computing a matrix of distances between all pairs
ity is huge O(n^2).
of input sequences (based on similarity index)
G. Hernández et al.

(2) Estimation of phylogenetic guide tree based on a classification that would be reduced if the classification
distance matrix was binary.
(3) Progressive global alignment according to a
guided tree 3.1.2 ECG analysis
(4) Guided tree reestimation and realignment Studies show that Heart Rate Variability and Power-Spec-
(5) iterative refinement tra-Density (PSD) analysis are correlated with the effect of
(6) Post-processing and visualisation. emotions and are able to characterise physiological ill-
nesses such as depressions or panic-disorders (McCraty et
Global sequence alignment uses the NW algorithm al., 1995). Heart rate (HR) and Heart Rate Variability
(Needleman & Wunsch et al., 1970) for aligning sequences (HRV) can be extrapolated from electrocardiogram signals
over their entire length to build up an optimal alignment using a MAX30100 Pulse Oximeter.
using previous solutions for alignments of smaller subse-
A MAX30100 Pulse Oximeter filter has been tested. The
quences. The algorithm uses dynamic programming
algorithm uses “mean-median filter” along with a “dc re-
(Gusfield et al., 1997) to find alignment maximising the
moval” to produce a much smooth signal from which HR
similarity score. can be discretised.
The classification does not improve with the discretisation,
Aligning is stated in terms of transforming sequence A into and the filter has to be improved since it is normalising the
B while using a set of operations. Algorithm rewards match signal in such a way that useful information is being lost.
and penalises indels (insertions/deletions). Producing a
matrix for a posterior trace-back. More research has been 3.1.3 Balancing
carried out to create gap-costs that allow block insertions
Balancing does not produce better results. In all the datasets
and deletions (Gotoh, 1982; Sankoff & Kruskal,1983) there is an oversampling of class “1”, neutral. Probably,
there is more probability for a stimulus to produce a neutral
MSA_2 uses Multiple Sequence Alignment as a mainframe.
response than to give an empowered or lowered response.
For each signal, sequences are separated regarding their
class. A database of aligned sequences is computed for each For that reason, when balancing with a subsampling, get-
class (alignment profile). For each query, the sequence is ting rid of useful information (is more likely for a specific
aligned to each class MSA profile and computes the simi- stimulus to give a neutral response) produces worst results.
larity. Class is inherited from the class profile giving the
minimum similarity measurement. Afterwards, a consensus
3.2 Results of QSSI algorithm
analysis is performed between all signals.
The percentage of error produced by the QSSI algorithm for
single signals with a one-nearest neighbour classifier shows
3 Results and Discussion randomness in almost all the classifications (results over
A number of experimental methods have been carried out 0.66, as the method has three classes (see Table 1).
to test the different steps of the methodology.

Table 1. % error on QSSI-1nn for single signals. Mean (Error - Arousal,

3.1 Preprocessing
Error - Dominance, Error - Liking and Error – valence).

3.1.1 Classification strategy SIGNAL qssi EpDTW

Classification tends to use binary models (low/empowered)
In the emotional analysis field; a binary model is useless. ECG 0,66015 0.61714
This means the individual is always in a reinforced or mi- GSR 0,61327 0,61912
TEMP 0,66992 0.64253
nored state, not allowing for a neutral state. For example, a
stimulus could like/dislike but also triggers no response at
all.
3.3 Results of QSSI-vital signs
Using these claims, the algorithm discretises the continuous Figure 5 demonstrates the overlapping of emotional data.
scale of classification into a 3 class model. This model To improve the results, further classification with consen-
shows certain drawbacks. Having more classes will pro- sus analysis of each vital signal has been performed to im-
duce a high probability to fall in incorrect classes in each prove the overall prediction.
Emotional analysis prediction

3.4 Results of prefixspan3

The classification on the structure of the sequences pro-
vides the most common patterns included in a database of
sequences.

By producing a database of each class in every signal, a

support threshold of 0.75 (the minimum percentage of se-
quences in the database presenting the pattern) is used to
make a database of patterns for each class and signal. Dis-
crimination of the patterns get rid of patterns not specific of
a class and shrinks the initial database, getting rid of small
uninformative patterns.
The pattern database, with the pattern and a similarity index
(calculated using the support and the number of outcomes),
Fig. 5. PCoA on DEAP arousal data. Principal components analysis of is used to dynamically find the pattern in each query.
arousal data. Demonstrating overlapping of the data
Results are shown in Table 5. They have been obtained with
3.2.1 QSSI-vs majority voting a support threshold of 0.75. When setting support higher
than 0.75, although getting more significant patterns, the
Using a knn classificatory with multiple signals, results are
discrimination lets some of the class pattern databases with-
improved in all classifications. Knn does not define the
out a single instance. In the other hand, when support is
classification based only on one instance, which makes a
smaller than 0.75, prefixspan is very expensive in terms of
more robust model.
time complexity, and the patterns do not have sufficient
support to serve as predictors.
Table 3. % error on QSSI-vs-knn_majority_voting (k = 35) using multiple
signals (ecg,gsr,temp). We need a support threshold smaller than 0.75 to get a large
enough pattern database (with discriminant entries for each
SIGNAL qssi EpDTW class) but is impossible in terms of computational re-
sources. For that reason, if there is no pattern in the class
Arousal 0.546875 0.5859375 database, the algorithm performs 35knn_aggrega-
Dominance 0.5078125 0.4921875 tional_voting from QSSI-vs to make the consensus.
Liking 0.578125 0.5625
Valence 0.5859375 0.546875
Table 5. % error on prefixspan3-35knn_aggregational_voting
SIGNAL QSSI EpDTW

3.2.2 QSSI-vs aggregational voting ECG 0.6875 0.6875

A knn based on the time of arrival noticeably improves the GSR 0.63281 0.63281
result in EpDTW distance matrix. Pondering the time of ar- TEMP 0.65625 0.65625
MULTIPLE 0.64063 0.64063
rival to give prominence improves the result.
The algorithm does not improve the previous results. Fur-
Table 4. % error on QSSI-vs-knn_aggregational_voting (k = 35) using ther experimentation with different SAX sequences will
multiple signals (ecg,gsr,temp). probably improve the results.

SIGNAL qssi EpDTW

3.5 Results of MSA_2
Arousal 0.5859375 0.6015625 MSA takes care of the patterns while also taking care of the
Dominance 0.65625 0.5859375 order of appearance. Prefixspan3 finds the more significant
Liking 0.578125 0.59375 patterns but does not retrieve the location on the sequence.
Valence 0.6796875 0.671875 Since the location of patterns can be used as a predictor,
MSA implementation alienates the sequence while giving
This method shows the worst results when compared with insight on the position of alignment. Results are shown in
majority voting. Nevertheless, in balanced data, aggrega- Table 6.
tional voting improves majority voting results, but never
reaches the ones achieved with the unbalanced dataset. Table 6. % error on MSA_2 for arousal prediction.
G. Hernández et al.

Liking 0.57812 0.46687 0.73437 0.625

SIGNAL QSSI EpDTW
Valence 0.65625 0.54687 0.625 0.6774

ECG 0.65625 0.65625

GSR 0.7265625 0.7265625
TEMP 0.640625 0.640625 The best prediction is achieved using knn_majority voting
MULTIPLE 0.6484375 0.6484375 (taking as k = 35). When comparing the g with the whole
dataset, results when using only participant data are better.

3.6 Participant analysis

3.7 Parameter variation analysis
Emotional data is very subjective and depends on many pa- A number of parameters testing have been carried out to
rameters. The patient must perform a self-classification of check the effect of parameter variation on the results.
the emotions, and the scales may change from one patient
to another. Because of this, and the fact that there are indi-
vidual variations within psychophysiological signals of pa- 3.7.1 vocabulary effect
tients makes the classification even more difficult The length of vocabulary is one of the parameters that de-
fine the final dimension of qualitative representations.
To solve these significant drawbacks, the model must adopt Choosing a proper length is critical to have better results.
representations able to handle individual differences in re-
sponses and account for essential signal variations, while
including flexible sequential behaviour. That improvement
could be used to make a self-learning method that uses the
individual's data, rather than other data, to continually im-
prove. By doing that, differential behaviour patterns among
individuals are eliminated (Henriques et al., 2014).

To minimise within-between individual differences, each

individualised dataset can be split between predictor and
predicted. Inputs are always the same for the 32 partici-
pants.

Results are shown in table 6 and 7. Best results are achieved Fig. 6. Vocabulary length effect. Error in Arousal prediction for different
using that technique, demonstrating that the main source of vocabulary effects.
noise responses differences between individuals. The better
predictor in all cases is QSSI-vital signs with knn majority Figure 6 shows the result of analysing the impact of differ-
voting (using as k=35) ent vocabulary sizes, from 4 to 8 symbols. There is no ap-
parent effect of the length of the vocabulary on the predic-
tions (for different predictions and different signals there is
Table 6. % error on participant data using EpDTW distance measure- not a significant relevance).
ment.
QSSI-1nn QSSI-vs-Knn-MJ QSSI-vs-Knn-AG MSA_2 3.7.2 PAA effect
Best results are achieved using PAA segments of 64. Since
Arousal 0.52343 0.4843 0.625 0.7096
Dominance0.5859 0.42128 0.6718 0.6826
Data is downsampled to 128hz, segments on 64 still gather
Liking 0.61718 0.46875 0.67187 0.625 essential information while reducing substantially the di-
Valence 0.648437 0.515625 0.67187 0.6774 mensionality getting rid of the granularity problem.

3.7.3 k effect
The k effect demonstrates the hypothesis that when using
Table 7. % error on participant data using QSSI distance measurement. individualised data, the results are better than using as a
train and test the whole dataset.
QSSI-1nn QSSI-vs-Knn-MJ QSSI-vs-Knn-AG MSA_2

Arousal 0.48437 0.468775 0.63281 0.7083

Since the emotional classification and physiological signals
Dominance 0.57812 0.42968 0.6875 0.7019 heavily depend on the individual (introducing noise in the
Emotional analysis prediction

dataset), to reduce noise the noise, and therefore improve 4 Conclusions

the prediction, the model should focus in using as predic- Emotional data show significant limitations and drawbacks
tors signals from the same individual as the test. This rea- in terms of predictive analysis. Subjectiveness in self-as-
soning produces a high improvement in the prediction sessment, with no clear empirical support, creates a limita-
when using participant data, by dropping the overall error tion on the classificatory step. Parameters must be clear,
from 0.60939 when using as predictors all individuals to and a method as objective as possible must be chosen to
0.468775. avoid distortions between users. Moreover, the complexity
of the data, with time-series structure, have been a limita-
tion and a jigsaw for years in the computational machine
learning field.
The need for new methods to reduce the complexity of the
signals while enabling for a proper prediction is clearly
stated when mining physiological signals.

To tackle this problem, this report provides a methodology

based on the simultaneous use of different psychophysio-
logical signals to improve the mining of emotional data.
Discretising time-series into character strings reduces the
complexity and also provides a valid input for feature-
driven models and text mining approaches. This method in-
tegrates sequential and feature-driving methods to perform
a reliable classification. Several classificators have been
tested to improve the prediction. The best prediction is
achieved after a knn classification of a distance matrix by
sequence distance methods of QSSI and EpDTW, two
methods based on computing distance between sequences
after alignment approaches.

In order to improve the results, methods must focus on cre-

ating individualised signal profiles to achieve an accurate
emotion prediction. Focusing on the structure of the signal
Fig. 7. K effect on prediction error. (1) Effect of k in individualised data.
as a whole, by aligning and getting the distances, and with
(2) effect of k in the whole dataset.
features associated with states are the best way to improve
the results. The better results are achieved using each indi-
In some of the predictions (Arousal and liking), for knn vidual as an independent instance (responding to an input
methods (it is more noticeable in aggregational voting), the in an individualised way) rather than treating all individuals
best k in individualised data is very close to the one when equally. Since emotions can be different between individu-
using the whole dataset as it is shown in fig. 7. That effect als, and physiological signals too, the best option is to cre-
suggests there is much noise in the whole dataset, that is not ate an individualised model, refining it progressively to ac-
as heavy in the individualised one. Variability between in- curate the prediction.
dividuals, produced by different physiological responses to Best results are achieved when using as predictors, se-
the same stimulus and self-assessment could be the princi- quences from the same individual as the predicted ones.
pal source of variation. Models to mine that type of signals Moreover, the k effect analysis shows clear evidence that
must use individualised data to get better results. the number of predictors to choose tends to be nearby to the
number of instances of each individual.

Self-learning approaches, individualised, what integrate a

higher variety of signals (that can be provided by a gadget
such as a wristband), with an objective self-assessment
tool, will be the key to achieve robust models that can mine
the signals to achieve reliable emotional prediction tools in
a daily environment.
G. Hernández et al.

Acknowledgements mobile technologies. Depression and Anxiety, 34(7), 603–609.

https://doi.org/10.1002/da.22646

Chih-Fong Tsai, “Bag-of-Words Representation in Image Annotation: A Re-

Funding view,” ISRN Artificial Intelligence, vol. 2012, Article ID 376804, 19 pages,
2012. https://doi.org/10.5402/2012/376804.
This work has been supported by the eXiT research group,
McCraty, Rollin & Atkinson, Mike & A.Tiller, William & Rein, Glen & D. Watkins,
{UdG}. Alan. (1995). The Effects of Emotions on Short-Term Power Spectrum Analy-
sis of Heart Rate Variability. The American journal of cardiology. 76. 1089-93.
10.1016/S0002-9149(99)80309-9.
Conflict of Interest: none declared.
Barrett, L. F., & Russell, J. A. (1999). The Structure of Current Affect: Controversies
and Emerging Consensus. Current Directions in Psychological Science, 8(1),
References 10–14. https://doi.org/10.1111/1467-8721.00003

Ghandeharioun, A., Fedor, S., Sangermano, L., Ionescu, D., Alpert, J., Dale, C., Son-
tag, D., Picard, R. "Objective assessment of depressive symptoms with machine
learning and wearable sensors data," International Conference on Affective
Computing and Intelligent Interaction (ACII), San Antonio, Texas, October
2017
Pei J., Han J., Mortazavi-Asl J., Pinto H., Chen Q., Dayal U., Hsu M., “PrefixSpan:
Mining Sequential Patterns Efficiently by Prefix-Projected Pattern Growth”,
17th International Conference on Data Engineering (ICDE), April 2001.

Agrawal R., Srikant R., “Mining sequential patterns”, Proceedings 1995 Int. Conf.
Very Large Data Bases (VLDB’94), pp. 487-499, 1999.

Han J., Dong G., Mortazavi-Asl B., Chen Q., Dayal U., Hsu M.-C., ”Freespan: Fre-
quent pattern-projected sequential pattern mining”, Proceedings 2000 Int. Conf.
Knowledge Discovery and Data Mining (KDD’00), pp. 355-359, 2000.

Wojciech Stach, “http://webdocs.cs.ualberta.ca/~zaiane/courses/cmput695-

04/slides/PrefixSpan-Wojciech.pdf”.

Erken, Eray. How to Measure Heart Rate with MAX30100 Pulse Oximeter. 30 April
2016. https://www.youtube.com/watch?v=36cELpNGJZYHow to Measure
Heart Rate with MAX30100 Pulse Oximeter (accessed December 1, 2016).

Kennedy, Stephen M. “AN INTRODUCTION TO PULSE OXIMETERS:

EQUATIONS AND THEORY.” The University of Wisconsin-Madison. 20
April 2015. http://www.imt.liu.se/FoUtb/kurser/oxikurs/restricted/pulseoxime-
tersequationsan dtheory-stephenkennedy.pdf (accessed January 15, 2017).

Maxim. “Pulse Oximeter and Heart-Rate Sensor IC for Wearable Health.” MaximIn-
tegrated. September 2014. https://datasheets.maximinte-
grated.com/en/ds/MAX30100.pdf (accessed November 18, 2016). Sang
Strogonovs, Raivis. "Implementing Pulse Oximeter Using MAX30100 - MORF -
Coding And Engineering". Morf.Lv, 2019,.
McCraty, Rollin et al. "The Effects Of Emotions On Short-Term Power Spectrum
Analysis Of Heart Rate Variability". 1995.
Pous, Carles et al. “Modeling Reuse On Case-Based Reasoning With Application To
Breast Cancer Diagnosis.” 2019.
Henriques, Rui et al. “Seven Principles to Mine Flexible Behavior from Physiologi-
cal Signals for Effective Emotion Recognition and Description in Affective In-
teractions.” PhyCS (2014).
H. Xiong et al. “A brief survey on sequence classification.”. SIGKDD Explorations
(2010).
Espinosa-Carrasco, Jose et al. Pergola: Boosting Visualization And Analysis Of Lon-
gitudinal Data By Unlocking Genomic Analysis Tools. 2019.
Koelstra, Sander. "DEAP: A Dataset For Emotion Analysis Using Physiological And
Audiovisual Signals". Eecs.Qmul.Ac.Uk, (2019),
http://www.eecs.qmul.ac.uk/mmv/datasets/deap/.
Gamero Argüello, Fco. Ignacio. "Pattern recognition based on qualitative represen-
tation of signals. Application to situation assessment of dynamic sys-
tems". Dugi-Doc.Udg.Edu, 2019, https://dugi-doc.udg.edu/bitstream/han-
dle/10256/7039/tfiga.pdf?sequence=5.

Aamodt, A., & Plaza, E., (1994). Case-based reasoning: Foundational issues, meth-
odological variations, and system approaches. AI Communications. Retrieved
from http://iospress.metapress.com/index/316258107242JP65.pdf

Ang, H. H., Gopalkrishnan, V., Zliobaite, I., Pechenizkiy, M., & Hoi, S. C. H. (2013).
Predictive Handling of Asynchronous Concept Drifts in Distributed Environ-
ments. IEEE Transactions on Knowledge and Data Engineering, 25(10), 2343–
2355. https://doi.org/10.1109/TKDE.2012.172

Armengol, E., (2016). Assessment of Diabetic Complications Based on Series of

Records. In First AI4D workshop, ECAI (pp. 28–30).

Aung, M. H., Matthews, M., & Choudhury, T. (2017). Sensing behavioural

symptoms of mental health and delivering personalised interventions using
Annexe, plot 1.

Result of the different algorithms on unbalanced data set with QSSI and EpDTW measurements. [% error of prediction, the blue
line is the random threshold, the red line is lower prediction error]

Annexe, plot 2.

Result of the different algorithms on balanced data set with QSSI and EpDTW measurements. [% error of prediction, the blue
line is the random threshold, the red line is lower prediction error] .
G. Hernández et al.

Annexe, plot 3.

Result of the different algorithms on balanced data set with QSSI and EpDTW measurements. [% error of prediction, the blue
line is the random threshold, the red line is lower prediction error].

Annexe, plot 4.

Result of the different knn algorithms for different k in the unbalanced dataset. [k value ranges from 1 to 128, algorithms are knn-
majority voting and knn-aggregational voting].
Emotional analysis prediction

Annexe, plot 5.

Result of the different knn algorithms for different k in the balanced dataset. [k value ranges from 1 to 128, algorithms are knn-
majority voting and knn-aggregational voting].

Annexe plot 6.

Result of the different knn algorithms for different k in participant data set. Average of the 32 participants. [k value ranges from
1 to 36, algorithms are knn-majority voting and knn-aggregational voting].