$FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Contents lists available at ScienceDirect

Accident Analysis and Prevention

journal homepage: www.elsevier.com/locate/aap

Construction accident narrative classication: An evaluation of text mining

techniques

Yang Miang Goh , C.U. Ubeynarayana
Safety and Resilience Research Unit (SaRRU), Dept. of Building, School of Design and Environment, National Univ. of Singapore, 4 Architecture Dr., 117566, Singapore

A R T I C L E I N F O A B S T R A C T

Keywords: Learning from past accidents is fundamental to accident prevention. Thus, accident and near miss reporting are
Accident classication encouraged by organizations and regulators. However, for organizations managing large safety databases, the
Construction safety time taken to accurately classify accident and near miss narratives will be very signicant. This study aims to
Data mining evaluate the utility of various text mining classication techniques in classifying 1000 publicly available con-
Support vector machine
struction accident narratives obtained from the US OSHA website. The study evaluated six machine learning
Text mining
algorithms, including support vector machine (SVM), linear regression (LR), random forest (RF), k-nearest
neighbor (KNN), decision tree (DT) and Naive Bayes (NB), and found that SVM produced the best performance in
classifying the test set of 251 cases. Further experimentation with tokenization of the processed text and non-
linear SVM were also conducted. In addition, a grid search was conducted on the hyperparameters of the SVM
models. It was found that the best performing classiers were linear SVM with unigram tokenization and radial
basis function (RBF) SVM with uni-gram tokenization. In view of its relative simplicity, the linear SVM is re-
commended. Across the 11 labels of accident causes or types, the precision of the linear SVM ranged from 0.5 to
1, recall ranged from 0.36 to 0.9 and F1 score was between 0.45 and 0.92. The reasons for misclassication were
discussed and suggestions on ways to improve the performance were provided.

1. Introduction et al., 2016). The results appear to be promising, but there are concerns
that the success of automatic classication of accident narratives is very
Workplace safety and health is a major concern in the construction sensitive to the dataset and the eectiveness of classication algorithms
industry in many countries (Zhou et al., 2015). To improve the in- may not be consistent across dierent datasets. There is also a wide
dustrys safety and health performance, the industry needs to learn from range of text mining techniques and the usefulness of dierent techni-
past accidents eectively (Chua and Goh 2004). However, accident ques in the context of accident narrative classication need to be
reports are typically unstructured or semi-structured free-text data that evaluated. Even though automatic classication of accident narratives
require signicant manual classication before statistical analyses can does not generate new knowledge per se, it may be argued that with
be conducted to facilitate interventions. These classication tasks are higher eciency, more incident data can be collected and more de-
typically conducted at organizational and national levels. Due to the tailed analytics can be conducted to produce useful insights that would
resource-intensiveness of the classication process, signicant amount not be available when fewer incidents were classied by human coders.
of resources need to be spent on classifying accident narratives, but the This study aims to evaluate the utility of various text mining clas-
consistency of the classication is hard to be ascertained. On the other sication techniques in classifying publicly available accident narra-
hand, organizations that choose not to classify accident narratives, will tives obtained from the US OSHA website (Occupational Safety and
suer loss of precious data for learning and accident prevention. Health Administration, 2016). This study also contributes to future
There had been an increased interest in automatic classication or studies on accident narrative classication by making available a da-
auto-coding of accident narratives through the application of text taset of 4470 construction accident narratives to other researchers (see
mining techniques. These studies typically aim to improve the con- Appendix A). The dataset includes 1000 narratives labelled in this study
sistency, productivity and eciency of accident narrative classication and 3470 narratives that were not labelled. The subsequent sections
(e.g. Chen et al., 2015b; Marucci-Wellman et al., 2011; McKenzie et al., provide an overview of current text mining research on accident nar-
2010b; Taylor et al., 2014; Tixier et al., 2016; Vallmuur 2015; Vallmuur ratives, the text data that were used in this study, an overview of the

Corresponding author.
E-mail address: [email protected] (Y.M. Goh).

http://dx.doi.org/10.1016/j.aap.2017.08.026
Received 15 May 2017; Received in revised form 24 July 2017; Accepted 26 August 2017
(OVHYLHU/WG$OOULJKWVUHVHUYHG
Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Recall =
(TP + FN )
text mining techniques implemented in this study, the results of the TP
evaluation, and discussion and recommendations for future research on (3)

2 Precision Recall
F 1Score =
text mining of accident narratives.

Precision + Recall (4)

2. Literature review

2.1. Text mining techniques 2.3. Past studies on accident narrative classication

Text mining is a well-researched eld. One of the common tasks in There were several other studies that applied text mining techniques
text mining is classication of text data (Sebastiani 2002). Text classi- in the analysis of injury narratives. In Chen et al. (2015a), the study
cation is the task of assigning one or more class labels to a document aimed to automatically classify narratives in emergency room medical
using a predened set of classes or labels. The supervised machine reports into common injury cause codes. The authors argued that injury
learning approach to text classication relies on an initial set of corpus narratives have unique characteristics that make them dierent from
(or collection of documents) with known class labels. This corpus is general documents and a detailed experiment is needed to evaluate the
split into training and testing datasets in order to train and then as- usefulness of dierent text mining techniques for their dataset. It was
certain the performance of the classier. The classier is trained by found that the use of matrix factorization coupled with support vector
observing the characteristics of the training dataset through dierent machine (SVM), gave the best classication performance. The authors
machine learning algorithms. reported recall ranging from 0.48 to 0.94 and precision ranging from
Data for text classication is typically represented using vector 0.18 to 0.95 for dierent classication labels. McKenzie et al., (2010a)
space model. In this model, each document is represented as a vector of also attempted to classify emergency department injury narratives for
terms. Another way to look at these terms is that they are essentially a the purpose of injury surveillance to support an evidence-based public
bag of words (Bird et al., 2009). Terms are features that represent a health response. The study compared keyword search, index search,
document, which could be a single word, phrase, or string. To distin- and text mining. Text mining was conducted using a content text
guish between documents in a corpus, each feature for each document mining software, Leximancer (Leximancer Pty Ltd, 2016), and it was
is given numeric values to show the importance of that term to the found that text mining approach provided the best performance. Bertke
document (Keikha et al., 2008).A commonly used vector space model is et al. (2012) made use of Nave Bayesian classiers to classify workers
the term frequencyinverse document frequency (tf-idf) representation medical compensation claims into three claim causation categories,
(Peng et al., 2014). In the tf-idf representation, values, or xik weights, i.e. musculoskeletal disorder (MSD), slip trip fall (STF), or others
reecting the importance of each given feature of a document is given (OTH). The study found that the Nave Bayesian classier was able to
by achieve approximately 90% accuracy for MSD, STF and OTH classi-

x ik = fik log
cations. However, it was observed that when OTH was being broken
N
ni
up into lower level classications, the performance of the classier
(1) dropped signicantly.
where fik is the frequency of feature i in document k, N is the number of Tanguy et al. (2015) evaluated an aviation safety report corpus,
documents in the corpus, and ni is the number of documents where which contains 136,861 documents, using support vector machine. As
feature i occurs. Once the document is represented using a suitable part of the pre-processing and experimental design, numerous forms of
vector space representation model, the data can be trained and classi- text units were created. Some of the text units explored include word,
ed using typical data mining techniques such as decision tree, neural word stems (e.g. falling is converted to its word stem, fall), and N-
network, support vector machine and Bayesian network (Raschka 2015; grams of words and stems. An N-gram refers to a set of N adjacent
Witten 2011). words (Bird et al., 2009). The study found that use of bi-gram and tri-
gram of stemmed narratives produced the best results in their pre-
2.2. Performance metrics liminary classications. They constructed a binary classier for each
target label (e.g. air trac management, bird strike, runway excursion
This study adopts the use of recall, precision and F1 score (or F- and glider towing related event) and that means 37 classiers were
measure) (Buckland and Gey 1994) to evaluate the performance of the trained. However, the authors only reported the results for seven of the
machine learning algorithms experimented. Table 1 and Equations (2) classiers, the precision ranged from 0.6 to 0.96, recall was 0.360.93,
to (4) dene these metrics. Essentially, precision is a measure of how and F1 score was 0.450.95. The authors highlighted that the perfor-
accurate the positive predictions are and recall is a measure of how mance for each classier is dependent on issues such as rarity, di-
many of the actual positives the model can identify (Williams 2011). F1 culty and inconsistency of the text data.
score combines precision and recall to provide an overall assessment of Taylor et al. (2014) trained Fuzzy and Nave Bayesian models to
performance of the classier. As these metrics are widely used and assign mechanism of injury and injury outcome for a set of re-related
discussed in the literature, readers can refer to text mining or machine near miss narratives obtained from the National Fireghter Near-Miss
learning textbooks (e.g. Bird et al., 2009; Witten 2011) for their de- Reporting System. Their algorithms achieved sensitivity (same as re-
tailed description. call) of between 0.602 and 0.74. Taylor et al. (2014) also made a

Precision =
comparison with ve other studies and claimed that their ndings are

(TP + FP )
TP
are comparable with the growing body of seminal studies on narrative
(2) autocoding.
For the construction industry, there were several studies that uti-
Table 1 lized text mining approaches in areas such as dispute resolution (Fan
True and false positives and negatives (adapted from Bird et al. (2009)).
and Li 2013), cost overrun (Williams and Gong 2014) document re-
Relevant Irrelevant trieval (Yu and Hsu 2013) and classication of eld inspection records
(Chi et al., 2016). Specically in the domain of construction accident
Retrieved True Positives (TP) False Positives (FP) (Type I narrative classication, Tixier et al. (2016) made use of a customized
error)
term lexicon (keyword dictionary) as well as a set of rules to auto-
Not retrieved False Negatives (FN) (Type II True Negatives (TN)
error) matically classify construction incident narratives. The study was con-
ducted on a dataset with 2201 accident narratives. The lexicons were


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

used to reduce the wide range of terms into a smaller set of keywords rules described in Table 2. These labels were based on the classica-
and rules were then manually crafted to improve the classication tions used in Workplace Safety and Health Institute (2016). Since more
performance of the approach. The authors reported a F1 score of 0.95; than one event can occur during an accident, the labelling of cases
which is signicantly better than several text mining papers that they followed the principle of identifying the rst occurrence of uncontrolled
reviewed. Even though hand crafting of rules has the advantage of or unintended action. The number of possible labels was also reduced
being more accurate; there are concerns that the rule-creation process is by combining labels with similar meanings. For example, in the case of
very tedious. The approach may not be sustainable when the rule set Fig. 1, the narrative could have been labelled as Struck by ying ob-
grows in proportion to the size of the dataset. In addition; the gen- ject, but was classied as Struck by moving object, which is more
eralizability of the rules will need to be evaluated against other data- general. Another example is the combination of falling from height
sets. and fall on same level into falls.
It can be observed that despite the successes of machine learning The narrative and title were then combined to form a single para-
algorithms in classifying emergency department narratives, there is a graph of text. Stopwords, which are high-frequency words with little
lack of study on the use of machine learning algorithms in classifying lexical content (Bird et al., 2009), were removed. Words could exist in
narratives in construction accident reports. Thus, this study aims to dierent forms, but are semantically similar, e.g. plural, and past tense
experiment with a variety of text mining machine learning algorithms forms. The use of several dierent forms of the same word during
and evaluate their potential in automatically classifying accident nar- training and can lead to poorer performance. Thus, word stemming was
ratives. carried out using the Snowball Stemmer (Shibukawa 2013), to obtain
word stems in which the dierent forms of the words is aggregated to a
single type. As an example, Fig. 2 is the processed text of the narrative
3. Dataset
in Fig. 1. Fig. 3 shows the word count for the original text and processed
text for each label. It can be observed that the number of words in the
3.1. Data source
original text had been compressed after processing. Fig. 3 also shows
that the word count for the text narratives is varied and there are many
The dataset used in this study is based on accident narratives col-
outliers for each label which may cause diculties for the classiers.
lected from the US OSHA website (Occupational Safety and Health
The outliers, in this context, refer to a narrative with word count that
Administration, 2016). A dataset of 16,323 accident records (occurring
lies outside the expected range of word count. The expected word count
between 1983 and 2013) was downloaded and a dataset consisting of
range for each label is dened as

[(Q1 1.5 IQR), (Q3 + 1.5 IQR)]

4471 construction industry cases were compiled based on the Industry
Code (SIC code) (Goh 2016).For the purpose of this study, only the title
and accident narrative information were used to classify the cases.
Fig. 1 shows a sample record, which includes the title, narrative and where Q1 is the 25th percentile, Q3 is the 75th percentile and IQR is the
label. It should be noted that the original data was not labelled and the inter-quartile range.
labels were created during data pre-processing.
4. Experiments conducted
3.2. Data pre-processing
Prior to experimentation, each stemmed narrative was broken into
One thousand cases were manually labelled based on the labelling strings of N-consecutive words, N-grams, in a process called

Fig. 1. Sample accident narrative and title.


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Table 2
Labels used for labelling the dataset, their criteria and frequency.

Case Labels Description Count %

1 Caught In/Between Objects Fracture, cuts, lacerations, amputations caused by being caught in between objects, generally referring to hand tools 68 6.8%
2 Collapse of Object Involving cases that started with structural failure 212 21.2%
3 Electrocution Direct electric shock or any burns caused by electrical faults 108 10.8%
4 Exposure to Chemical Contact with any toxic/corrosive chemical substances 29 2.9%
Substances
5 Exposure to Extreme Extreme temperatures caused by frost, hot liquid or gases, including hypothermia 17 1.7%
Temperatures
6 Falls Involving slip, trip cases and where the victim is falling from elevation but not due to structural failure 236 23.6%
7 Fires and Explosion Injuries caused by direct res and explosion but not electrical burns 47 4.7%
8 Struck by Falling Object victim is hit by falling object from height but object is not due to structural failure 43 4.3%
9 Struck by Moving Objects The victim is hit by a moving object (that is not in free fall) 134 13.4%
10 Trac Injury happen due to workers driving a vehicle or a moving vehicle hits worker. 63 6.3%
11 Other To accommodate cases that do not fall in the above categories. Some of the less occurring categories were merged into 43 4.3%
others as the number of occurrences was too low (drowning, suocation)
Total 1000 100.0%

Fig. 2. Sample processed text.

Fig. 3. Both box plots for word counts on 11 class labels.


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

tokenization (Bird et al., 2009). Tokenization is the process of breaking of the labels and this helps to account for the imbalance frequency of
up a sentence of strings into pieces such as words, phrases which are dierent labels (see Table 2). Subsequently, a grid search function
then referred to as tokens. Tokens are used as the input data in text (scikit-learn Community, 2016) was implemented to identify the sui-
mining. Uni-gram tokenization breaks a longer string into one word table tuning parameters or hyperparameters by implementing a brute-
tokens, bi-gram is a token consisting of two words and tri-gram is a force exhaustive search (solving a problem in which by analyzing
token with three words. A corpus of documents can thus be represented nearly all possible solutions) (Raschka 2015)through many models in
by a matrix with one row per document and one column per token (e.g. sequence and returning the best model as measured by a pre-de-
one word). The general process of turning a collection of text docu- termined performance metric.
ments into a set of numerical feature vectors is called vectorization. In the linear SVM model, the C value is the key hyperparameter that
Tokenization is also called the n-grams representation (scikit-learn needs to be tuned. The C parameter is essentially a regularization
Community, 2016). Uni-grams and bi-grams, which consist of one and parameter, which determines how much the classier adapts to the
two, word length tokens, were used in this experiment. These were then training samples (scikit-learn Community, 2016). C is the parameter,
converted into the tf-idf document term matrix representation (see which helps to strengthen the misclassication of training data. Lower
Equation (1)). This tf-idf matrix, coupled with the case labels, was then values of parameter C will smoothen the decision surface with longer
used to train the classiers. running times. Higher values of parameter C will correctly classify the
Using stratied sampling, about 25% of the 1000 labelled cases training data with lesser the running times. Seven C values on the log
(251) were set aside as an independent test set to compare the perfor- scale ranging from 103 to 103were used and the selection was based
mance of the dierent approaches evaluated. The open-source algo- on performance measured using F1 score. In addition, ten-fold cross-
rithms used in this study were primarily derived from Python 2.7 validation was utilized during the search. The same test set was then
(Python Software Foundation, 2016), scikit-learn library version 0.17 used to measure the performance of the SVM model. During the linear
(scikit-learn Community, 2016) and Natural Language Toolkit (nltk) SVM experimentation, dierent combinations of uni-gram, bi-gram and
library version 3.1 (Bird et al., 2009). A preliminary experiment was tri-gram were experimented with. Finally, the radial basis function
conducted on the dataset to identify the best performing learning al- (RBF), a non-linear function, was experimented with search function
gorithm to focus on. The preliminary multinomial classication ex- based on variations in the C and gamma parameters. The gamma
periment included six machine learning classiers: support vector ma- parameter has similar function as the C parameter and it can be seen as
chine (SVM), linear regression (LR), random forest (RF), k-nearest the inverse of the radius of inuence of samples selected by the model
neighbor (KNN), decision tree (DT) and Naive Bayes (NB). SVM and NB as support vectors (scikit-learn Community, 2016). Nonlinear classi-
were used and recommended in previous studies and the remaining cations can be handled with the help of the parameter gamma. The
algorithms were arbitrarily selected. Each of these algorithms are well parameter gamma belongs to the Gaussian kernel family, a small
established and are carefully described in machine learning textbooks gamma indicates a Gaussian kernel with large variance, where as a
(e.g. Bishop 2006). Thus, the remainder of this paragraph only provides large gamma reveals a Gaussian kernel with low variance. More im-
brief descriptions of those algorithms. SVM algorithm creates hyper- portantly the kernel function in SVM classier contains all the in-
planes based on training data and then optimizes hyperplanes, which formation about the relative positions of the input data in the feature
are the basis for classifying data points among classes. Logistic regres- space. The actual learning algorithm of SVM based only on the kernel
sion algorithm predicts the probability of occurrence of an event by function and thus the algorithm can be carried out without explicit use
tting data to a logit function. Nave Bayes algorithm uses Bayes rule of the feature space.
with strong independent assumptions between features. It simplies the
calculation of probabilities by assuming that the probabilities of each
feature belonging to a given class label are independent of all other 5. Results and ndings
features. K nearest neighbor classier is a simple algorithm that stores
all available class labels and classies new class labels based on a dis- 5.1. Preliminary classication
tance vector when there is no prior knowledge about the underlying
distribution of the data. A decision tree starts with a root node, where As highlighted earlier, an initial round of classication was con-
all the training data is located. An initial split is made using a test on a ducted using six machine learning classication algorithms, so as to
feature thus, separating the data into child nodes. Further splitting can select the best performer for further experimentation. The classication
be made to the child nodes depending on the outcome of multiple tests. results, as represented by F1 score, are shown in Table 3.
The branches signify the result of the test on a feature and the leaf
nodes represent the class labels. The nodes where the nal class labels Table 3
are xed is known as the terminal node. A random forest classier F1 scores for preliminary experiment.

constitutes a set of decision trees. Each tree in the forest predicts their F1 Score
nal class label. The collection of trees then voted for the most popular
class as the nal class label. Labels SVM LR RF KNN DT NB
Throughout all the experiments, the same set of training cases was
caught in/between objects 0.60 0.36 0.53 0.34 0.46 0.89
used to train each machine learning classier independently to label collapse of object 0.66 0.48 0.54 0.52 0.56 0.25
each narrative to the 11 labels in Table 2. The classiers were then electrocution 0.95 0.91 0.95 0.69 0.92 0.67
tested on the test set and the best performing classier was used in exposure to chemical substances 0.62 0.00 0.00 0.40 0.40 0.40
subsequent experiments aiming to improve the performance of the exposure to extreme temperatures 0.67 0.00 0.00 0.40 0.25 0.00
falls 0.78 0.63 0.74 0.66 0.76 0.17
classier.
res and explosion 0.74 0.56 0.71 0.50 0.64 0.63
Consistent with other studies like Tanguy et al. (2015) and Chen others 0.43 0.00 0.29 0.20 0.48 0.00
et al. (2015b), the SVM classier (Burges 1998) was the best performer struck by falling object 0.14 0.00 0.00 0.11 0.27 0.61
during the preliminary experiment. The initial SVM was set up as a struck by moving objects 0.58 0.48 0.45 0.44 0.55 0.48
One-vs-Rest (OVR) linear classier with parameters C = 10 and ba- trac 0.67 0.48 0.40 0.54 0.54 0.36
Average 0.62 0.35 0.42 0.44 0.53 0.41
lanced class weight (scikit-learn Community, 2016). A OVR technique
extends a binary classier for a multi-class problem by training one SVM Support Vector Machine, LR Linear Regression, RF Random Forest, KNN K-
classier for each label or class (Raschka 2015). The balanced mode Nearest Neighbour, DT Decision Tree, NB-Nave Bayes; numbers in bold and italics are
automatically adjusts weights inversely proportional to the frequencies the highest value for the label.


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Table 4 5.3. Misclassication

F1 Score for experiments using dierent tokenisation and optimisation.
For the linear SVM model, misclassication is less likely in narra-
F1 Score
tives labelled as electrocution, falls, re and explosion, and trac,
SVMLinear SVMRBF but more likely in cases with labels, others, struck by falling object,
exposure to chemical substances and struck by moving objects. The
Label (1,1) (1,2) (1,3) (2,2) (2,3) (1,1) commonly mislabeled cases were evaluated by creating a confusion
caught in/between objects 0.61 0.58 0.58 0.36 0.20 0.56 matrix, and qualitative evaluation of the mislabeled cases were re-
collapse of object 0.67 0.62 0.60 0.40 0.39 0.71 corded in Table 6.
electrocution 0.92 0.94 0.93 0.75 0.72 0.90
exposure to chemical substances 0.55 0.60 0.25 0.25 0.25 0.60 6. Discussion
exposure to extreme 0.67 0.67 0.67 0.00 0.00 0.67
temperatures
falls 0.83 0.76 0.78 0.70 0.66 0.83 MisclassicationOne of the key reasons for the misclassication was
res and explosion 0.80 0.80 0.75 0.47 0.27 0.83 probably due to the tf-idf vector space model not being able to capture
others 0.45 0.29 0.17 0.15 0.17 0.45 the context in which the words are used in the incident. For example,
struck by falling object 0.52 0.29 0.15 0.17 0.17 0.50
the tf-idf vector space does not retain any information about the order
struck by moving objects 0.57 0.60 0.62 0.44 0.37 0.57
trac 0.76 0.62 0.59 0.42 0.36 0.76 of terms after it is converted into the tf-idf matrix. Even though the
Average 0.67 0.61 0.55 0.37 0.32 0.67 tokenization of terms into bi-gram or tri-gram may help to retain some
of these context, the experimentation conducted shows that uni-gram
SVM Support Vector Machine; RBF radial basis function; (1,1) uni-gram; (1,2) provided the best results. Thus, the classier may give too much at-
uni-gram and bi-gram; (1,3) uni-gram and tri-gram; (2,2) bi-gram; (2,3) bi-gram
tention to terms that do not provide clear linkage to the direct cause of
and tri-gram; numbers in bold and italics are the highest value for the label.
the accident. Another key challenge for classication of accident nar-
ratives is that the narrative might be overly focused on aspects not
A best macro average F1 score, or the average F1 score for all labels,
directly related to the cause of the accident. For instance, for the nar-
of 0.62 was achieved using the SVM classier. Even though the second
rative in Fig. 4, the number of words describing the work environment
best classier (in accordance to average F1 score) was DT, NB classier
was excessive in relation to the number of words describing the actual
performed very well for two of the labels, caught in/between objects
direct cause of the accident, fall into the trench.
and struck by falling object. However, NB performed very poorly for
Some labels were fairly tricky to classify even for human readers, in
most of the other labels.
particularly the labels, collapse of object, struck by falling object,
struck by moving object. This is because natural language is not
precise and there are dierent ways to word a sentence, giving rise to
5.2. Tokenization and optimization
dierent interpretation. For instance, in Fig. 5, it is not very clear
whether it is a case of being struck by a moving object, or was the
With reference to Table 4, another six experiments were conducted
victim caught in between objects. In this particular case, the label was
to improve the performance of the SVM classier. The linear SVM with
given as struck by moving object. The inherent diculty in classi-
optimization was conducted by varying dierent tokenization of the
fying accident narrative was also discussed in Tanguy et al. (2015), who
processed text. The variations include use of uni-gram, bi-gram and
found that signicant portion of their corpus were misclassied by the
combinations of uni-gram, bi-gram and tri-gram. As can be observed
human classiers.
from Table 4, uni-gram gave the highest average F1 score.
Despite the credible performance of the linear SVM for some of the
In addition, a RBF SVM with uni-gram tokenization was trained and
categories, it is not possible to automatically label all the narratives
tested. As can be seen from Table 4, among the six classiers in Table 4,
without human intervention. If a machine learning approach is to be
the RBF SVM classier had the highest F1 score for seven of the 11
taken for labelling of accident narratives, it is suggested that the auto-
labels, while the linear SVM had the highest F1 score for six labels.
coding of narratives be done only for labels with good F1 score, e.g.
Thus, the overall performance of the non-linear SVM is similar to the
electrocution and falls. Labels with moderate level of F1 scores, e.g.
linear SVM with uni-gram. In accordance to the Occam's Razor, the
exposure to extreme temperatures and collapse of object, can be auto-
linear SVM with unigram tokenization is recommended due to its re-
coded, but additional checks by human classiers will be needed.
lative simplicity. Table 5 shows the precision, recall and F1 score of the
best performing linear SVM model.
6.1. Industry implications

The proposed text mining approach has the ability to support an

Table 5 informed culture (Reason 1997) in construction organizations. As ar-
Precision, recall and F1 score of best performing SVM model. gued by Reason (1997), an informed culture, one in which those who
manage and operate the system have current knowledge about the
Label Precision Recall F1 Score human, technical, organizational and environmental factors that de-
electrocution 1.00 0.85 0.92 termine the safety of the system as a whole, is synonymous to safety
exposure to chemical substances 0.75 0.43 0.55 culture. An informed culture is dependent on an eective safety in-
re and explosion 0.77 0.83 0.80 formation system that is able to eectively and eciently capture,
struck by moving objects 0.53 0.62 0.57 analyze and disseminate information from incidents, near misses and
exposure to extreme temperatures 1.00 0.50 0.67
others 0.57 0.36 0.45
proactive checks on the safety management system. To materialize the
falls 0.77 0.90 0.83 safety information system and informed culture proposed by Reason
struck by falling object 0.50 0.55 0.52 (1997), the ability to eciently and consistently classify incidents and
trac 0.85 0.69 0.76 near misses is critical. This is especially challenging in the construction
collapse of object 0.67 0.66 0.67
industry, which consists of temporary and resource-scarce projects that
caught in/between objects 0.63 0.59 0.61
Average 0.73 0.63 0.67 may not be prepared to develop and maintain a safety information
system.
Values above average were highlighted in bold and italics. It is proposed that the construction industry should create a web-


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Table 6
Qualitative evaluation of commonly mislabeled labels.

Actual Case Label Most Commonly Mislabelled Qualitative Evaluation of Samples

Label

1 electrocute re and explosion The narrative focused too much on the eects of electrocution, e.g. burn, ash burn, and res, which
overlaps with re and explosion
2 exposure to chemical others In the mislabeled cases, there were a lot of cardio-related terms in the narrative from the forensics
substances diagnosis, which contributed to anothers category label, and also, the chemical described in the narrative
was unique.
3 re and explosion electrocution The narrative in the re and explosion case described electrical equipment, but the cause of accident was
not related to the equipment.
4 struck by moving objects collapse of object Object involved in the incident are trees and beams which are not typically associated under the struck by
moving objects. Also, some objects broke/sheared which were confused with structural collapse.
5 exposure to extreme re and explosion The mislabeled cases were mostly about scalds and burns, while the correctly labelled extreme temperature
temperatures cases were about heat strokes.
6 collapse of object falls Ladder collapse leading to a fall is common problem.
7 falls collapse of object The victim was in an environment that was frequently associated with structural collapse e.g. trench.
However, the fall incident does not have any relation with the environment.
8 struck by falling object collapse of object The classier was unable to distinguish between objects falling as a result of structural failure or improper
securing
9 trac collapse of object Some seldom-seen vehicles e.g. train and bus, were not linked to trac cases. In some cases, vehicle
overturning was associated with collapse of object than trac
10 others falls others is not a well-dened category and it has no strong dening features
11 caught in/between objects struck by moving objects Classier was unable to pick up the cause; possibly due too much noise from other labels.

based safety information system to allow construction companies and 1998). It is a semi-supervised learning technique that seeks to make use
their employees to submit descriptions of incidents and near misses of an additional large number of unlabeled cases, as in the case of this
anonymously. It is believed that if the data is collected industry wide, a study, and use them to improve the classication performance. To be
large pool of data can then be collected to understand construction able to exploit the unlabeled cases would likely to be an advantage in
hazards more eectively; this will encourage reporting, learning and any classication system because labelling cases require additional ef-
informed culture, which will lead to better safety performance in the fort.
long run. To manage the large pool of text data, the proposed text A common problem in classication of text, which was not ex-
mining approach becomes essential. Concurrently, on-going calibra- amined in this study, is the excessive number of terms (features) in each
tion, training and testing of the algorithms will improve the accuracy of set of narrative. To reduce the number of features and possibly improve
the automatic classication systems. The safety information system will performance, a set of ontology and lexicon (e.g. Tixier et al., 2016) can
then allow other analytics and machine learning techniques to be ap- be applied to the corpus during pre-processing. Words with similar
plied. meaning can be compressed into the same term and allow the learning
algorithm to spot patterns across the corpus. Future works in this area
6.2. Future work and limitations could involve the creation of a more domain specic construction-re-
lated dictionary to better identify specic terms. A more intelligent pre-
The study showed that dierent classiers were able to produce processing of the narrative, such as using rule-based methods, could
better performance for dierent labels. None of the classiers were help strip away elaborate narratives that are not related to the incident,
unanimously the best across all labels. Thus, an ensemble approach and improve classication accuracy. In addition, dimension reduction
(Witten 2011), where several classiers are trained independently and techniques such as the Latent Semantic Analysis (LSA) (Turney and
then used collectively to select the best label for each narrative, may be Littman 2003)can be applied to the corpus. LSA considers the re-
able to produce better classication performance. Another technique lationship between words in the data and it makes use of the Singular
that should be explored is the co-training approach (Blum and Mitchell Vector Decomposition (SVD) mathematical technique to produce a new

Fig. 4. Sample text to illustrate how narrative can

over-focus on content not related to the label.


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Fig. 5. Sample text to illustrate how some narratives

can be dicult to classify.

set of concepts that is drawn from the words present in the data. Media Inc.
Bishop, C.M., 2006. Pattern Recognition and Machine Learning. Springer, New York.
In addition, this study had not explored the potential inconsistency Blum, A., Mitchell, T., 1998. Combining labeled and unlabeled data with co-training. In:
in classication when the machine learning models are applied across Proceedings of the Eleventh Annual Conference on Computational Learning Theory,
dierent datasets. Each dataset may have its own characteristics and ACM. Madison, Wisconsin, USA. pp. 92100.
Buckland, M., Gey, F., 1994. The relationship between recall and precision. J. Am. Soc.
the machine learning models developed using dierent dataset may Inf. Sci. (19861998) 45 (1), 12.
produce dierent classication when tested with a fresh set of data. It is Burges, C.J., 1998. A tutorial on support vector machines for pattern recognition. Data
recommended that further studies be conducted to reduce the incon- Min. Knowl. Discovery 2 (2), 121167.
Chen, L., Vallmuur, K., Nayak, R., 2015a. Injury narrative text classication using fac-
sistency that may arise when dierent datasets are used.
torization model. BMC Med. Inform. Decis. Mak. 15 (1), 112.
Chen, L., Vallmuur, K., Nayak, R., 2015b. Injury narrative text classication using fac-
7. Conclusions torization model. BMC Med. Inf. Decis. Making 15 (1), S5.
Chi, N.-W., Lin, K.-Y., El-Gohary, N., Hsieh, S.-H., 2016. Evaluating the strength of text
classication categories for supporting construction eld inspection. Autom. Constr.
This study produced comparable results as previous text mining 64, 7888.
studies that were focused on corpus from other industries (e.g. Tanguy Chua, D.K.H., Goh, Y.M., 2004. Incident causation model for improving feedback of safety
et al., 2015; Taylor et al., 2014). Based on the overall F1 score, this knowledge. J. Constr. Eng. Manage. Am. Soc. Civ. Eng. 130 (4), 542551.
Fan, H., Li, H., 2013. Retrieving similar cases for alternative dispute resolution in con-
study recommends the use of linear SVM with uni-gram tokenization to struction accidents using text mining techniques. Autom. Constr. 34, 8591.
automatically code accident narratives, in particular the U.S. OSHA Goh, Y.M., 2016. Accident Narratives Dataset Obtained from Occupational Safety and
accident database. A suitable alternative was non-linear SVM using Health Administration (OSHA) Fatality and Catastrophe Investigation Summaries.
OSHA, U.S.
radial basis function, but the improvement in F1 score was not sig- Keikha, M., Razavian, N.S., Oroumchian, F., Razi, H.S., 2008. Document representation
nicant. Text mining could alleviate the diculty in labelling large and quality of text: an analysis. In: Berry, M.W., Castellanos, M. (Eds.), Survey of Text
volume of text by selectively labelling a portion of the cases with high Mining II: Clustering, Classication, and Retrieval. Springer, London, pp. 219232.
Leximancer Pty Ltd, 2016. Leximancer. http://info.leximancer.com/ (April 11, 2016).
condence and leaving the ones that are harder to label to human Marucci-Wellman, H., Lehto, M., Corns, H., 2011. A combined Fuzzy and Nave Bayesian
classiers, who would have their workload reduced. Periodic updating strategy can be used to assign event codes to injury narratives. Inj. Prev. 17 (6),
of the training set will be required to check and update the classiers. 407414.
McKenzie, K., Campbell, M.A., Scott, D.A., Discoll, T.R., Harrison, J.E., McClure, R.J.,
It is believed that dierent corpus has dierent characteristics and it 2010a. Identifying work related injuries: comparison of methods for interrogating
will be useful for researchers to share the dataset that they have eval- text elds. BMC Med. Inform. Decis. Mak. 10 (1), 110.
uated, so that other researchers can assess the performance of dierent McKenzie, K., Scott, D.A., Campbell, M.A., McClure, R.J., 2010b. The use of narrative text
for injury surveillance research: a systematic review. Accid. Anal. Prev. 42 (2),
machine learning techniques on the same dataset. Since application of
354363.
machine learning approaches in classifying accident narratives provides Occupational Safety and Health Administration, 2016. Fatality and Catastrophe
the foundation for improved ability to learn from past accidents, it is an Investigation Summaries. https://www.osha.gov/pls/imis/accidentsearch.html (Mar
important area within construction management research. Thus, the 30, 2016).
Peng, T., Liu, L., Zuo, W., 2014. PU text classication enhanced by term fre-
dataset collected in this study was made publicly available to motivate quencyinverse document frequency-improved weighting. Concurrency Comput.
other researchers to search for better approaches to automatically label Pract Experience 26 (3), 728741.
accident narratives (see Appendix A). Python Software Foundation, 2016. Python Language Reference, Version 2.7. http://
www.python.org (Apr 13, 2016).
Raschka, S., 2015. Python Machine Learning. Packt Publishing, Birmingham.
Acknowledgement Reason, J., 1997. Managing the Risks of Organizational Accidents. Ashgate, Aldershot.
Sebastiani, F., 2002. Machine learning in automated text categorization. ACM Comput.
Surv. 34 (1), 147.
This study is funded by Singapore Ministry of Education (MOE) Tier Shibukawa, Y., 2013. Snowball Stemming Library Collection for Python. https://gi-
1 grant (Grant No.: R-296-000-161-112). The author acknowledges Mr thub.com/shibukawa/snowball_py (Apr 12, 2016).
Xianglong Lius assistance in pre-processing the data and developing Tanguy, L., Tulechki, N., Urieli, A., Hermann, E., Raynal, C., 2015. Natural language
processing for aviation safety reports: from classication to interactive analysis.
portions of the scripts for this study. Ms Zhenzhen Fans advice and Comput. Ind. 78, 8095.
inputs are also appreciated. Taylor, J.A., Lacovara, A.V., Smith, G.S., Pandian, R., Lehto, M., 2014. Near-miss nar-
ratives from the re service: a Bayesian analysis. Accid. Anal. Prev. 62, 119129.
Tixier, A.J.P., Hallowell, M.R., Rajagopalan, B., Bowman, D., 2016. Automated content
Appendix A
analysis for construction safety: a natural language processing system to extract
precursors and outcomes from unstructured injury reports. Autom. Constr. 62, 4556.
The dataset used in this study can be downloaded from https:// Turney, P.D., Littman, M.L., 2003. Measuring praise and criticism: inference of semantic
github.com/safetyhub/OSHA_Acc.git. Please acknowledge this article, orientation from association. ACM Trans. Inf. Syst. 21 (4), 315346.
Vallmuur, K., Marucci-Wellman, H.R., Taylor, J.A., Lehto, M., Corns, H.L., Smith, G.S.,
if the dataset is used. The original data were obtained from https:// 2016. Harnessing information from injury narratives in the big data era: under-
www.osha.gov/pls/imis/accidentsearch.html (Occupational Safety and standing and applying machine learning for injury surveillance. Inj. Prev. 22 (Suppl
Health Administration, 2016). 1), i34i42.
Vallmuur, K., 2015. Machine learning approaches to analysing textual injury surveillance
data: a systematic review. Accid. Anal. Prev. 79, 4149.
References Williams, T.P., Gong, J., 2014. Predicting construction cost overruns using text mining:
numerical data and ensemble classiers. Autom. Constr. 43, 2329.
Williams, G., 2011. Data Mining with Rattle and R: The Art of Excavating Data for
Bertke, S.J., Meyers, A.R., Wurzelbacher, S.J., Bell, J., Lampl, M.L., Robins, D., 2012.
Knowledge Discovery (Use R!). Springer, Canberra, ACT, Australia.
Development and evaluation of a naive bayesian model for coding causation of
Witten, I.H., 2011. Data Mining: Practical Machine Learning Tools and Techniques.
workers compensation claims. J. Safety Res. 43.
Morgan Kaufmann, Burlington, MA.
Bird, S., Klein, E., Loper, E., 2009. Natural Language Processing with Python. O'Reilly


Y.M. Goh, C.U. Ubeynarayana $FFLGHQW$QDO\VLVDQG3UHYHQWLRQ

Workplace Safety and Health Institute, 2016. Workplace Safety and Health Report 2015. . Zhou, Z., Goh, Y.M., Li, Q., 2015. Overview and analysis of safety management studies in
https://www.wsh-institute.sg/ (Mar 30, 2016). the construction industry. Saf. Sci. 72 (February), 337350.
Yu, W.D., Hsu, J.Y., 2013. Content-based text mining technique for retrieval of CAD scikit-learn Community, 2016. Scikit-learn Machine Learning in Python. http://scikit-
documents. Autom. Constr. 31, 6574. learn.org/ (Apr 13, 2016).