Microbiological Water Quality Test Results Extraction from

Mobile Photographs
Jifang Xing Zhang Ruixi Remmy Zen
[email protected] [email protected] [email protected]
National University of Singapore National University of Singapore National University of Singapore

Ngurah Agus Sanjaya ER Laure Sioné Ismail Khalil

[email protected] [email protected] [email protected]
Udayana University Imperial College London Johannes Kepler University

Stéphane Bressan
[email protected]
National University of Singapore

ABSTRACT 1 INTRODUCTION
An emerging and promising approach to achieving access to wa- The United Nations have identified access to water and sanitation
ter and sanitation for all leverages citizen science to collect valu- for all as the sixth of seventeen goals in the United Nations Sus-
able data on water quantity and quality, which can assist policy- tainable Development Goals [7]. However, in the developing world,
makers and water utility managers in sustainably managing water the effects of climate change, compounded by a lack of financial
resources. resources and trained personnel, contribute to making Sustainable
This paper specifically considers water quality data collected Development Goal six a particularly challenging one to reach. Al-
using a mobile phone app wielded by citizen-scientists. The cit- though cities in the developing world may provide piped water
izens use a microbiological water quality test kit to measure E. coli supply to the home, this is not synonymous to having access to
content. The test result is photographed by the citizens and passed continuous, reliable or safe water [40]. Indeed, using data collected
on to scientists for interpretation. However, reading the results by the International Benchmarking Network, World Bank, World
necessitates a trained scientific eye and is time consuming. This Health Organisation and Asian Development Bank, the authors
paper therefore puts forward an algorithm that can automatically of [15] estimated that over 309 million people have water sup-
infer the outcome of the test from a photograph. plied intermittently – excluding countries not reporting to these
To this end, we evaluate several image processing and machine databases, which would no doubt count many more.
learning algorithms for the automatic extraction of results from Intermittent Water Systems provide water discontinuously in
photographs of microbiological water quality tests. We devise and time and space, which not only results in an unreliable provision
present a new knowledge and rule-based algorithm and show that of water, but also one that is unsafe for direct consumption. Water-
it performs satisfactorily. borne pathogens proliferate in stagnant pipe-water, which is flushed
down the next time that water flows through the pipes and con-
CCS CONCEPTS sumed at the tap. Furthermore, since all pipe systems leak, when
• Computing methodologies → Visual content-based indexing the water supply is switched off, the pressure drops and contami-
and retrieval. nated groundwater or surface water can enter the system through
the cracks in the pipe walls or joints [25]. This contamination is
KEYWORDS typically indicated by the presence of E. coli.
rule-based, feature extraction, environmental engineering This failure to meet the water needs of the population occurs for
a number of reasons, including lack of financial resources, lack of
ACM Reference Format: trained personnel, poor governance, urban population growth and
Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure
climate change.
Sioné, Ismail Khalil, and Stéphane Bressan. 2019. Microbiological Water
An emerging and promising approach to tackling these chal-
Quality Test Results Extraction from Mobile Photographs . In The 21st Inter-
national Conference on Information Integration and Web-based Applications lenges is citizen science. Citizen science is the collection and some-
& Services (iiWAS2019), December 2–4, 2019, Munich, Germany. ACM, New times analysis of scientific data by members of the public in collab-
York, NY, USA, 10 pages. https://doi.org/10.1145/3366030.3368455 oration with scientists, and has typically so far been applied to the
natural world [24]. It results in the generation of large volumes of
Publication rights licensed to ACM. ACM acknowledges that this contribution was
authored or co-authored by an employee, contractor or affiliate of a national govern-
relatively cheaply acquired data, that would be otherwise difficult to
ment. As such, the Government retains a nonexclusive, royalty-free right to publish or obtain. However, citizen science projects necessitate a certain level
reproduce this article, or to allow others to do so, for Government purposes only. of engagement on the citizen’s part if they are to be successful [31].
iiWAS2019, December 2–4, 2019, Munich, Germany
© 2019 Copyright held by the owner/author(s). Publication rights licensed to ACM.
Therefore, in an effort to provide a convenient and engaging user
ACM ISBN 978-1-4503-7179-7/19/12. . . $15.00 experience, some citizen science projects such as the Networking
https://doi.org/10.1145/3366030.3368455
iiWAS2019, December 2–4, 2019, Munich, Germany Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure Sioné, Ismail Khalil, and Stéphane Bressan

Water Data project in Nepal [41], use mobile phone applications

as a data collection and analysis tool. Although mixing technology
and low-income countries may seem counter-intuitive, the use of
digital interfaces in the water domain are becoming increasingly
widespread, with mobile phone applications being used even in the
poorest of developing countries such as Ethiopia [11].
In the Networking Water Data project, local citizens are provided
with a mobile phone app, which lets them log in data on the quantity,
timeliness and perceived quality of their municipal water supply.
Participants in the project are also provided with water quality test
kits to monitor the chlorine levels and level of contamination of
their water by E. coli and total coliform bacteria. The citizens then
take pictures of the test results using their mobile phones, and send Figure 1: Photograph of the bag
them to the managing scientists for interpretation.
In this way, citizens record the extent to which they lack access
to water infrastructure services, and the extent to which their water
supply is contaminated at their locations in real-time. This allows
for in-depth investigation of the technical patterns and causes, as 2 RELATED WORK
well as providing valuable evidence of the health consequences. The The authors of [19] developed a crowdsourcing application to mon-
result is better information about where and for whom action and itor water quality by calculating the remote sensing reflectance in
investment are most urgently needed, assisting policy-makers and red, green and blue channels of the picture on a mobile application.
water sector officials in sustainably managing the water supply [36]. More directly related to the utilisation of E. coli test kits, the
Although this technique, as well as other crowdsourcing plat- authors of [1, 22] present mobile applications, mWater and Akvo
forms and applications for water monitoring [1, 19, 22], provides Caddisfly that allow users to upload photographs and manually
relevant water quality data, it is noted that the interpretation of enter water quality results. However, these two applications do not
the water quality test results is a time-consuming process which automatically infer the outcome and result from the photograph
necessitates a trained scientific eye. This is not a sustainable water but engage trained enumerators to interpret the result instead.
management practice for low-income countries where water supply Not only is this time-consuming, but it is also prone to human
management resources are already scarce. error: first when the data is inputted manually and a second time
This paper therefore investigates the automatic inference of the when the most probable number of E. coli is looked up. We are
test result from photographs of the water quality tests. Specifically, therefore interested in methods that can help us to infer the most
we consider the collection of water quality data using a similar probable number of E. coli from the colour of the compartments
crowdsourcing technique to Networking Water Data project [41], of a photograph of a compartment of the quantitative E. coli water
namely the Aquagenx [2] microbiological water quality test (E. coli) quality test kit.
kit [33]. The kit consists of a compartmentalised bag and a growth The problem can be reformulated in terms of different standard
medium. image processing tasks.
The bag is shown in Figure 1. The bag is made of five compart- On the one hand, the inference could be based on the detection
ments with different volumes. After the bag is filled with the water of the compartments in the photograph followed by the inference
sample to be tested, the growth medium tablet is added and the of the main colour of each compartment. On the other hand, the
sample is left to react for 20 to 48 hours [2]. The colour of the water inference could be a more direct classification or regression tasks
in the different compartments changes. The colour combination of that tries and directly infers the test outcome or the most probable
the different compartments determines the outcome of the test. The number of E. coli. The detection of the compartments in the pho-
user can interpret the outcome by looking at a correspondence table tograph can leverage image feature extraction algorithms such as
between the colour combination and the most probable number of edge, contour or shape extraction algorithms. Statistical machine
E. coli bacteriophage in the table provided by Aquagenx shown in learning algorithms in general and neural networks in particular
Table 1. The most probable number of E. coli bacteriophage value can be used for the image classification and for the regression tasks.
is a proxy measurement for water quality. Several algorithms have been developed for the extraction of
We evaluate several image processing and machine learning features from images and many surveys of these algorithms [14, 28,
algorithms for the automatic extraction of results from photographs 32, 37, 44, 46] have been written since the 1969 survey by Martin
of microbiological water quality tests. We notice that the former Levine [20].
are not sufficiently robust and the latter suffer from small and The features of an image can be divided into two levels: low-
imbalanced data sets. We devise and present a new knowledge and level features and high-level features. The low-level features are
rule-based algorithm and show that it performs satisfactorily. the features that are extracted without any shape information [23].
These features include colour, texture, isolated points, lines and
edges. Local binary patterns operator can be used to characterise
the local texture of each pixel [26]. Because edges are typically
characterised by sharp changes in intensity, detecting such changes
Microbiological Water Quality Test Results Extraction from Mobile Photographs iiWAS2019, December 2–4, 2019, Munich, Germany

can be done by computing gradients. The low-level features are the a general purpose graphic processing unit for a faster image clas-
basic building blocks of high-level features. sification. Convolutional neural network can be used for image
The high-level features define the shape information of the im- and pixel classification. The mask region convolutional neural net-
ages [23]. These features include shape, position, size and rotation. work [10] is a two-stage model, in which each pixel is classified
Complex pictures are composed of a set of simple shapes. into not only different categories but also different instances.
Traditional hand-crafted edge extraction algorithms usually rely Fully convolutional neural network [21] and U-Net [29] are end-
on local features in the small neighbouring region of each pixel. to-end models for pixel based classification. The authors of [27]
We present some representative works on edge, contour and shape present a comprehensive survey of convolutional neural network
extraction methods. One of the widely used edge extraction tech- algorithms and implementations for image classification.
niques is the Canny edge detector. The Canny edge detector uses a
low and high threshold to reduce false edge points after the gradient
computation [4].
Some machine learning algorithms have been devised for edge 3 METHODOLOGY
extraction. They perform a one-class classification task classify-
ing pixels into the single class of pixels belonging to an edge. The 3.1 Problem Definition and Framework
detected edges form regions. They are not required to form lines. The outcome of a water quality test with the Aquagenx microbi-
The authors of [35] propose a convolutional neural network model ological water quality test kit is given by the colour combination
called DeepContour, which consists of four convolution layers and of the compartments. Each compartment is either yellow or green.
two fully connected layers, to detect if there is an edge in a patch Our goal is to automatically infer the outcome of the test from a
and classify the edge. Here, the patch is a small region (of 45 × 45 photograph of the bag.
pixels) in a picture. Edge extraction by DeepContour is limited Figure 2 summarises the pipeline to extract the microbiological
to within a small region, which means that DeepContour cannot water quality results, i.e., most probable number of E. coli from a
utilise the information available at a larger scale. The authors of [43] photograph of a bag of a water quality test.
develop an end-to-end edge extraction algorithm based on fully The framework that we consider consists of the preprocessing
convolutional networks, which can automatically learn the hierar- of the photograph into an image that can be processed followed
chical features. One of the drawbacks of machine learning based by the classification of the image and the mapping of the predicted
edge extraction techniques is that they require large amount of outcome to the most probable number of E. coli in the sample water.
training data samples. Table 1 shows the mapping of the thirty-two possible outcomes to
A popular contour extraction algorithm is the Active Contour the most probable number.
Model or Snake. The Active Contour Model requires weak prior The photograph is a three-dimensional matrix with dimension
geometric information, e.g., the topology of the contour, to detect M ×N ×3, where M and N are the width and the height, respectively,
the contour in an image [12]. This method performs well in sit- and the three layers corresponds to the red, green and blue colour
uations where the target geometric information is available. The of the photograph in pixels.
algorithm minimises a formulation of the energy between the image The outcome of a test is denoted as c = [c 1, . . . , c 5 ] where c i ∈
and the contour by transforming known geometric information a {G, Y}, where “G” is short for green and “Y” is short for yellow.
fitting it to the detected edges. There are only two parameters in the For instance, the outcome of the photograph shown in Figure 1 is
model controlling the distance to the edges and the curvature of the [Y , G, Y , Y, Y ] where the colour of the first, second, third, fourth
inferred contour. The Active Contour Model has several limitations. and fifth compartment is yellow, green, yellow, yellow and yellow,
It is not effective if we only know the object shape without any respectively. This corresponds to a most probable number value of
indication of its position in the image [26]. Typical shape extraction 1.2 according to Table 1.
algorithms can leverage template matching. For instance, they can We want to approximate a function F (·), where F (X) = c̃ is the
use Brute-Force Matcher [3] or Orientated FAST Rotated BRIEF predicted result for the colours of the compartments in a given
descriptor [30]. photograph. Therefore, we can cast the problem into an image
If the tasks is formulated as a classification or regression task, classification problem.
many candidate algorithms can be found in the zoo of statistical Figure 1 illustrates several relevant features of the bag that facil-
machine learning algorithms including support vector machines [6, itate or hinder the inference of the test result. Each compartment
8], decision tree [34, 42] and neural networks [9, 13, 16–18, 46]. We has a specific shape and position. The first, fourth and fifth com-
focus on some representative neural network algorithms. partments have a rectangular shape while the second and third
Convolutional neural network use convolution to extract the compartments have both an L-shape. The relative position of each
spatial correlation within images [9]. They are supervised models compartment is fixed. However, the actual shapes of the compart-
and the parameters are trained automatically using a large amount ments are often distorted because the plasticity of the the bag. There
of labelled images. could be several interfering objects and light reflections, such as
The authors of [45] show that the convolution layer in a convo- clips, in the foreground and background of the bag. The colours
lution neural network is neither random nor uninterpreted. Instead, of these objects might be green or yellow and the shape might
it identifies discriminant structures in the images, such background, be rectangle and L-shape. Finally, the colours of the fourth and
foreground, contours and texture. The authors of [5] propose and fifth compartments are very light due to the small volume of the
evaluate an implementation of a convolutional neural network on compartment.
iiWAS2019, December 2–4, 2019, Munich, Germany Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure Sioné, Ismail Khalil, and Stéphane Bressan

Predicted Predicted
preprocessing classification outcome mapping MPN

original processed
photograph image

Figure 2: The framework of our method. MPN stands for the most probable number of E. coli.

Table 1: Most probable number of E. coli table the region that contains the bag and resize the photograph from
M × N pixels into a square image of dimension 512 × 512 pixels. The
Compartment MPN former is achieved by only keeping the pixels with a color within
1 2 3 4 5 Category value the range from (8, 50, 10) to (100,255,242)1 . Isolated pixels or small
10mL 30mL 56mL 3mL 1mL /100mL groups of pixels are further eliminated using the erosion operation
0 0.0 with appropriate settings. This method assumes that there is no
2 1.0 significant yellow or green regions in the background.
1 1.0 Throughout the paper, an image refers to the output of the pre-
16 1.1 processed photographs of the bag. Figure 3 shows the image after
8 1.2 the preprocessing of the photograph in Figure 1.
4 1.5
3 2.0
18 2.1
17 2.1
10 2.4
9 2.4
24 2.6
19 3.2
11 3.7
5 3.1
6 3.2
20 3.4 Figure 3: Preprocessed image for the photograph of a bag
25 3.9 shown in Figure 1.
26 4.0
12 4.7
7 5.2 3.3 Classification
27 5.4
The initial idea that we considered has three steps: identifying the
21 5.6
compartments in the image, evaluating their colour and decoding
22 5.8
the outcome of the test. There is however no immediate method to
23 8.4 identify the five compartments in the image. Instead, we can try
13 9.1 and identify region and map them to the compartments.
14 9.6 The second idea, therefore, consists in four steps: finding regions
28 13.6 of the image that are candidate that correspond to the compart-
15 17.1 ments, mapping the regions to the compartments, evaluating their
29 32.6 colour and decoding the outcome of the bag. We consider finding
30 48.3 the regions is done using a contour detection method. We use an
31 >100 active contour method called the Snake [12]. It is however unfortu-
nately the case that a region may correspond in reality to several
compartments or that a compartment may correspond to several
3.2 Preprocessing regions.
The photographs of bags are taken from a variety of points of 1 The processing is in the hue, saturation, value colour space. The hue, saturation,
view with different camera settings and at different resolutions. value colour space is widely used in computer vision as it directly provides useful
The first step attempts to normalise the photographs by cropping information and can be easily obtained from the red, green and blue space.
Microbiological Water Quality Test Results Extraction from Mobile Photographs iiWAS2019, December 2–4, 2019, Munich, Germany

We therefore explore a third idea that abandons the objective of Table 2: Boolean Test Vectors
directly identifying the compartments and their respective colour.
Rather it focuses on finding the monochrome, i.e. green and yel- Notation Definition
low, regions after projection on the x- and y-axis of the image, ( ⌈W /3⌉)
respectively. It computes four lists of Boolean values indicating the Tx1 1W
( ⌈2W /3⌉)
presence of green and yellow pixels in the columns and rows of the Tx2 1W
image, respectively. ( ⌈W /2⌉) (W )
Tx3 ¬1W ∧ 1W
For example, the four lists indicating the green and yellow re- ( ⌈W /3⌉) (W )
Tx4 ¬1W ∧ 1W
gions on the columns and rows of the picture in Figure 4, where ( ⌈2H /3⌉)
the background is transparent, are [1, 0, 1, 1], [0, 1, 0, 1], [1, 0, 1, 1] Ty1 1H
and [0, 1, 0, 1], respectively. Note that the pixels of an image is read
from left to right horizontally and from top to bottom vertically.
Before designing the rules, we make the following assumptions.
We assume that there is exactly one bag in the image. We as-
sume that the region of that the first compartment does not exceed
left half part and top 2/3 part of the image, namely {(x, y) : 0 ≤
x ≤ W /3, 0 ≤ y ≤ 2H /3}. We assume that the region of second
compartment does not exceed the left 2/3 part of the image, namely
{(x, y) : 0 ≤ x ≤ 2W /3}. We assume that the region of third
compartment is within the right 2/3 part of the image, namely
{(x, y) : W /3 ≤ x ≤ W }. We assume that the region of fourth
compartment does not exceed between the horizontal range of 1/2
Figure 4: A 4 × 4 image where each pixel is either yellow, and 9/10 and the top 2/3 part of the image, namely {(x, y) : W /2 ≤
green or transparent. x ≤ 9W /10, 0 ≤ y ≤ 2H /3}. We assume that the region of fifth
compartment does not exceed the right 1/3 part and the top 2/3
part of the image, namely {(x, y) : 2W /3 ≤ x ≤ W , 0 ≤ y ≤ 2H /3}.
This representation is different but reminiscent of the Japanese
nonograms [38].
We refer to the width and height of the image as W and H ,
respectively and for the sake of generality, although W = H since
we consider a square image (after the preprocessing mentioned in
Subsection 3.2).
д
We now define the image Boolean image vectors. Let R x be a
Boolean vector in which a bit with value 1 denotes the presence of
д
green pixels in column in the corresponding column. Let Ry be a
Boolean vector in which a bit with value 1 denotes the presence
y
of green pixels in column in the corresponding row. Let R x be a
Boolean vector in which a bit with value 1 denotes the presence of
y
yellow pixels in column in the corresponding column. Let Ry be a
Boolean vector in which a bit with value 1 denotes the presence of Figure 5: Illustration of the assumptions on a practical case.
yellow pixels in column in the corresponding row.
Testing that a Boolean image vector A contains at least one bit These assumptions are reasonable for practical cases as illus-
with value 1 is testing that the disjunction of all bits, A is 1.
Ô
trated in Figure 5.
Testing that a Boolean image vector A consists of bits with value We now present the design of the rules. The rules are actually
1 is testing that the conjunction of all bits, A is 1.
Ó
arranged into a sequence [Ri ] to be checked in order. The corre-
Testing that a Boolean image vector A is a subset of a Boolean sponding possible sets of possible outcomes are noted Ci . At each
vector B, namely that if there is a bit, Bi with value 1 in position i step of the checking, if a rule Ri is satisfied, then we infer that the
in A, then there is a bit Ai with value 1 in the same position in B possible outcome c belongs to Ci and the algorithm terminates, oth-
for every positions i, is testing that, for every position i, Bi → Ai , erwise, Ci is eliminated from the possible cases and the algorithm
or, in short B → A. This is similarly tested as ¬B ∨ A. continues the checking.
In addition, we define a set of Boolean test vectors that we are In the case where the inferred Ci is a set of several possible
going to use to define the tests associated with the rules of the outcomes, then we select the best outcome using the number of
(k )
algorithm. These Boolean test vectors are given in Table 2. 1l consecutive regions and the length of these regions according to
l д д y y
denotes a Boolean test vector x ∈ {0, 1} where x i = 1 for 1 ≤ i ≤ k. R x , Ry , R x , Ry .
д д y y
We design a set of rules using R x , Ry , R x , Ry to infer the possible The pseudo-code of the algorithm is given in Algorithm 1.
outcomes of the test. Each rule is associated with a set of possible Each rule infers a set of possible outcomes based on the informa-
д д y y
outcomes. tion summarised in R x , Ry , R x , Ry . The possible outcomes for each
iiWAS2019, December 2–4, 2019, Munich, Germany Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure Sioné, Ismail Khalil, and Stéphane Bressan

Algorithm 1: Rule-based Inference Table 3: Rules

д д y y
Data: {C1, . . . , C14 }, {R 1, . . . , R 14 }, R x , Ry , R x , Ry
Result: Ci , i ∈ {1, . . . , 14} Rules
Ô д
for i = 1; i ≤ 14; i = i + 1 do R 1 = ¬( R x )
if Ri is satisfied then Ô y
R = ¬( R x )
Select best c ∈ Ci ; Ô д Ó д 2 Ô д Ó д
R 3 = (( (Ry )) ∧ ( (Ry → Ty1 ))) ∧ (( (R x )) ∧ ( (R x → Tx1 )))
return c Ô y Ó y y Ó y
R 4 = (( (R x )) ∧ ( (R x → Tx1 ))) ∧ (( (Ry )) ∧ ( (Ry → Ty1 )))
Ô
end if Ô д Ó д
R 5 = ( (R x )) ∧ ( (R x → Tx2 ))
end for Ô y Ó y
R 6 = ( (R x )) ∧ ( (R x → Tx2 ))
Ô д Ó д
R 7 = ( (R x )) ∧ ( (R x → Tx3 ))
Ô y Ó y
R 8 = ( (R x )) ∧ ( (R x → Tx3 ))
Ô д Ó д
д д y y R 9 = ( (Ry )) ∧ ( (Ry → Ty1 ))
rule are given in Table 4. Since the image vectors R x , Ry , R x , Ry are Ô y Ó y
R 10 = ( (Ry )) ∧ ( (Ry → Ty1 ))
Boolean vectors, the condition of each rule is a Boolean expression. Ô д Ó д
R 11 = ( (R x )) ∧ ( (R x → Tx4 ))
For instance, the first rule infers that the outcome is a bag with Ô y Ó y
R 12 = ( (R x )) ∧ ( (R x → Tx4 ))
all five compartments yellow because the Boolean image vector Ô д Ó д
R 13 = ( (R x )) ∧ ( (R x → (Tx1 ∧ Tx3 )))
indicating the presence of green pixels (for instance on the x-axis) Ô y Ó y
R 14 = ( (R x )) ∧ ( (R x → (Tx1 ∧ Tx3 )))
is all 0s. The is the negation of the disjunction of the bits in the
vector.
For instance, the third rule infers that the outcome is a bag with
the first compartment green and all four other compartments yellow Table 4: Output Sets
because the Boolean image vector indicating the presence of green
pixels on the y-axis in the region characterised by the Boolean Notation Definition
test vector Ty1 and the presence of green pixels on the x-axis in
C1 {[Y, Y, Y, Y , Y ]}
region characterised by the Boolean test vector Ty1 . This is tested C2 {[G, G, G, G, G]}
д
by verifying that Ry indicates some green pixels and is a subset of C3 {[G, Y, Y , Y, Y ]}
д
Ty1 as well as R x indicates some green pixels and is a subset of Tx1 . C4 {[Y, G, G, G, G]}
The subset testing is encoded as an implication with the Boolean C5 {[G, G, Y, Y, Y ], [Y, G, Y, Y , Y ]}
д Ó д
{[Y, Y, G, G, G], [G, Y, G, G, G]}
vectors. For instance, Ry ⊂ Tx1 is tested as ( (Ry → Ty1 ))(note that C6
д д
(Ry → Ty1 )) is (¬Ry ∨ Ty1 )). C7 {[Y, Y, Y, G, G], [Y, Y, Y, Y, G], [Y, Y, Y, G, Y ]}
For instance, the fifth rule infers that the outcome is a bag with C8 {[G, G, G, Y, Y ], [G, G, G, G, Y )], [G, G, G, Y, G]}
the first and second compartments green and all other compart- C9 {[G, Y, Y , G, G], [G, Y, Y, Y, G], [G, Y, Y, G, Y ]}
ments yellow or a bag with the second compartment green and C10 {[Y, G, G, Y, Y ], [Y, G, G, G, Y ], [Y, G, G, Y, G]}
all other compartments yellow because the Boolean image vec- C11 {[Y, Y, G, Y, G], [Y, Y, G, G, Y ], [Y, Y, G, Y, Y ]}
tor indicating the presence of green pixels on the x-axis in the C12 {[G, G, Y, Y, G], [G, G, Y, G, Y ], [G, G, Y, G, G]}
region characterised by the Boolean test vector Tx2 . This is tested by C13 {[Y, G, Y, Y, G], [Y, G, Y, G, Y ], [Y, G, Y, G, G]}
д
verifying that R x indicates some green pixels and is a subset of Tx2 C14 {[G, Y, G, G, Y ], [G, Y, G, Y, G], [G, Y, G, Y, Y ]}
The rules given in Table 3.
For instance, the outcome of the image in Figure 3 is [Y, G, Y, Y, Y ].
д (L ) (R ) д
The image vectors of the bag are R x = ¬1512д ∧ 1512д , Ry = in two different architecture designs that differ only on the number
(T ) (B ) y (512) y (512)
¬1512д ∧ 1512д , R x = 1512 , Ry = 1512 , where lд and tд are of nodes in the output layer.
the smallest indexes of the 1-value bits in the corresponding im- In the first design, we directly classify the image into 32 classes
age vectors and rд and bд are the largest indexes of the 1-value corresponding to all possible outcomes. Therefore, there are 32
Ô д nodes in the output layer. The output nodes corresponds to the
bits in the corresponding image vectors. As ¬( R x ) = False and
Ô y probability of the 32 possible outcomes from the most probable
¬( R x ) = False, the algorithm continues the checking the follow-
ing rules R 3 and R 4 . From Figure 5, we can see that bд is larger that number table. We refer to this model as the CNN-32 model.
y In the second design, we train the network to learn five binary
2H /3, which violate R 3 , and Ry is with all one bits, which violate
variables corresponding to the outcome. Therefore, there are five
R 4 . From Figure 5, we can see that Rд is smaller than ⌈W /2⌉. The
nodes in the output layer. Each node represents each compartment
output of the bag belongs to C3 .
in the kit, respectively. We apply the sigmoid function on the output
node. Therefore, each node outputs a number between 0 and 1 that
3.4 Classification with Neural Networks corresponds to the probability of the corresponding compartment
A radically different yet modern idea is to train a convolutional to be yellow. If the probability is more than or equal to 0.5, then
neural network to classify the cropped square images into 32 cate- we determine that the corresponding compartment is yellow and
gories corresponding to the 32 possible outcome. This can be done vice versa. We refer to this model as the CNN-5 model.
Microbiological Water Quality Test Results Extraction from Mobile Photographs iiWAS2019, December 2–4, 2019, Munich, Germany

4 PERFORMANCE EVALUATION The key point is to map the candidate regions to the corresponding
In Subsection 4.1, we describe the data set. In Subsection 4.2, we compartments of the bags. Since the general shape of a compart-
present a qualitative evaluation of the active contour method that ment is the same for every bag, it seems reasonable to apply a
we use. In Subsection 4.3, we present a comparative quantitative methods which can utilise an initial contour of the bag and the
evaluation of the rule-based method and neural network methods. compartments.
In Subsection 4.4, we continue this evaluation focusing on the We consider the active contour method. The active contour
performance for previously unseen configurations. method can be easily implemented by the existing libraries skimage [39].
The input of the active contour method is the image, an initial con-
4.1 Data Set tour and two important other parameters, which control contracting
speed and contour smoothness.
We have collected a data set of 300 photographs of bags. The test
Figure 7 shows two examples of initial and detected contours.
were conducted in several locations in Jakarta, Indonesia. The pho-
The initial contours are represented as red dash lines. The detected
tographs have been taken with a mobile phone.
contours are represented as blue solid lines. We use the Snake to
Figure 6 summarises the distribution of the possible colour con-
detect the contour of the second compartment. Note that the initial
figurations in the data set. The categories (numbers) identifying
contour and hyperparameters are the same for all images, as it is
the configurations are given in Table 1. From Figure 6a, we observe
meaningless and time consuming to manually tune the hyperparam-
that the data set is imbalanced.
eters for each image. This example illustrates two common errors
Most tests outcomes consist of five yellow compartments indi-
in the results of active contours: multiple and partial covering.
cating safe water quality. The numbers of outcomes of the other
types are less than ten. There are only twelve types of outcome
out of a total of 32. Figure 6b shows the number of outcomes for
each compartment. The number of yellow compartments is much
smaller than that of the green ones. The data set is imbalanced.

300

251
250

200
(a) The detected contour covers (b) The detected contour partially
150 multiple compartments. covers a compartment.

100
Figure 7: Results of active contour model for the second com-
50 partment with fine-tuned hyperparameters α = 0.01 and
2 5 9 5 2 6 3 6 3 5 3 β = 0.01.
0
0 2 4 8 12 13 16 24 28 29 30 31
category Figure 7a shows an erroneous situation in which the detected
contour for the second compartment covers multiple compartments.
(a) Histogram of outcomes in the data set.
Figure 7b shows an erroneous situation in which the detected con-
350
yellow green tour for the second compartment only covers part of the second
290 292
300 274 274 271 compartment. Such errors commonly occur in compartment identi-
250 fication by active contour method. Based on the qualitative eval-
200
uation of preliminary experiments that we have conducted using
and the active contour method and the many drawbacks of this
150
approach in such situations as illustrated above, we conclude that
100 the method alone is not viable for the task at hand, namely the
50 26 26 29
detection of individual compartments.
10 8
We conclude that we should not try to first identify the compart-
0
1st compart. 2nd compart. 3rd compart. 4th compart. 5th compart. ment and then estimate their colour but rather use a global method
that does not care about identifying individual compartments such
(b) Histogram of outcomes for each compartment.
as a neural network approach or such as the method that we de-
vise and present here, which partitions the image horizontally and
Figure 6: Statistical information of data set vertically into green and yellow areas and uses rules to decode the
test results without explicitly identifying individual compartments.

4.2 Qualitative Evaluation of the Active 4.3 Quantitative Evaluation of the Rule-based
Contour Method Method and the Neural Network Methods
As we discussed in Section 3, one of the natural ways to extract We use four evaluation metrics to comparatively evaluate the per-
the the water quality test results is with the active contour method. formance of the different methods. We measure the accuracy of the
iiWAS2019, December 2–4, 2019, Munich, Germany Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure Sioné, Ismail Khalil, and Stéphane Bressan

predicted outcome Aw calculated as follows.

number of correctly predicted outcome
Aw = 100% ×
size of the data set
We measure the performance of our method to detect the colour
of each compartment. Therefore, we measure the accuracy of the
predicted colour of each compartment Ai calculated as follows.
number of correctly predicted compartment i
Ai = 100% ×
size of the data set
We measure the average accuracy of the predicted colour of com-
partment calculated as follows. (a) Accuracy versus number of epochs for CNN-32
5 model.
1 Õ
As = Ai
5 i=1
We map the outcome to the most probable number based on the
Table 1. We measure the root mean square error (RMSE) of the most
probable number of E. coli calculated as follows.
rÍ
i (predicted mpn of bag i − true mpn of bag i)
2
RMSE =
size of data set
Note that if the rule-based method returns the predicted outcome
as “not found”, then we treat the outcome as [Y , Y, Y, Y, Y ] when
we calculate the root mean square error. (b) Accuracy versus number of epochs for CNN-5
For this experiment, we use the LeNet architecture [18] for the model.
neural network model. Specifically, there are two convolution layers
with 3 × 3 and 6 × 6 convolution with rectified linear unit activation Figure 8: Accuracy versus number of epochs for different
functions. After the convolution layers, there are three fully con- CNN models.
nected layers. The two hidden layers consist of 120 and 84 nodes
correspondingly. The number of the nodes in the output layer is 5
and 32 for the CNN-5 and CNN-32 model, respectively, as explained However, the performance of the rule-based method on the test
in Subsection 3.4. data set and the training data set are quite different in terms of root
We randomly split the data set into training and test data set. mean square error of the most probable number. The small size of
The size of the training data set is 240 images and the size of the test the data set causes a large variance of the randomly sampled test
data set is 60 images. We use the multi-label hinge loss function to data set.
train the CNN-5 model and the binary cross entropy loss function For the training data set, CNN-5 model reaches 100% accuracy
for CNN-32 model according to the properties of the output of the and zero root mean square error. This means that CNN-5 model
neural network models. overfits the training data set. The CNN-32, as illustrated in the
Figure 8 shows the accuracy versus the number of training Figure 8a, always predicts the outcomes as [Y, Y, Y, Y, Y ]. Note that
epochs. From Figure 8a, we observe that the performance of the although the accuracy of the predicted outcome of the rule-based
CNN-32 model never improves. This is influenced by the imbalance method is smaller than CNN-32, the root mean square error of the
of the data set even after tuning the hyperparameters. We observe rule-based method is smaller than CNN-32. This means that the
that the predicted outcome of the trained CNN-32 model is always predicted most probable number of the rule-based method, although
[Y, Y, Y, Y, Y ] for every images in the training and test data set. The incorrect, is closer to the correct most probable number than CNN-5
CNN-32 model only learns that the major outcomes of the training and CNN-32 model.
data set is [Y, Y, Y, Y, Y ]. The CNN-32 model does not perform well
for this imbalanced data set. 4.4 Unseen Data
From Figure 8b, we observe that the training of the of CNN-5 The performance of the neural networks is dependent on the quality
model converges after ±22 epochs. Contrary to the CNN-32 model, of the data set. They need to learn from the available data set.
the accuracy on the test data set increases as the accuracy on the Consequently, if the data set does not contain all cases, the neural
training data set increases. This indicates that the CNN-5 model networks cannot learn these unseen cases. Figure 6a shows that
learns something about the other outcomes than all yellow. our data set only consists of 12 possible outcome cases from the
Table 5 presents the performance of the three methods, i.e., rule- total 32 cases. To test the performance of the different method on
based method, CNN-5 and CNN-32, based on the four performance unseen data, we remove the images of the category 8, i.e., outcome
metrics. [Y , G, Y , Y, Y ], from the data set to construct a new data set. We
For the test data set, CNN-5 outperforms the other two methods choose the outcome 8 as the number of instances of outcome since
for all the performance metrics. The rule-based method does not it is the second largest outcome in the data set, which can provide
work better than CNN-5 on the randomly sampled test data set. more stable evaluation. We retrain the neural network models on
Microbiological Water Quality Test Results Extraction from Mobile Photographs iiWAS2019, December 2–4, 2019, Munich, Germany

Table 5: Performance of Each Method for Data Set

Data Set Method A1 A2 A3 A4 A5 As Aw RMSE

Rule-based 93.3% 96.6% 91.6% 88.3% 95.0% 93.0% 83.3% 13.2
test Data Set CNN-5 98.3% 100.0% 95.0% 96.7% 98.3% 97.7% 90.0% 11.16
CNN-32 88.3% 88.3% 86.6% 95.0% 95.0% 90.6% 80.0% 15.64
Rule-based 85.0% 94.5% 94.5% 92.5% 90.8% 87.5% 78.8% 8.41
Training Data Set CNN-5 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 100.0% 0
CNN-32 92.1% 87.9% 91.3% 97.1% 97.9% 93.3% 84.5% 11.47
Whole Data Set Rule-based 86.6% 95.0% 94.0% 91.7% 89.0% 91.3% 79.7% 9.57

this new data set. The size of the new data set is 291 images. We Table 6: Predicted Outcome of Each Method for Unseen
randomly split the new data set into the new training data set and Cases
the new test data set. We remove the images belonging to category
8 from the new data sets. There are nine such images. The size of Methods Rule-based CNN-5 CNN-32
the new training data set is 232 images and the size of the new test [Y, Y, Y, Y , Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
data set is 69 images. not found [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
We train CNN-5 and CNN-32 based on the this new data set. [Y, G, Y, Y, Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
Figure 9 shows the accuracy versus the number of training epochs [G, Y, Y, Y, Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
with the new data set. The pattern is similar to the one shown in predicted
[Y, Y, Y, Y , Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
Figure 8. outcome
[Y, Y, Y, Y , Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
[Y, G, Y, Y, Y ] [Y, Y, G, Y, Y ] [Y, Y, Y , Y, Y ]
[Y, G, Y, Y, Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
[Y, G, Y, Y, Y ] [Y, Y, Y, Y, Y ] [Y, Y, Y , Y, Y ]
As 84.4% 77.8% 80.0%
Aw 44.4% 0% 0%
RMSE 0.801 1.135 1.200

square error for the unseen data. We observe that both the CNN-5
and CNN-32 model cannot predict the right outcome for the unseen
data. The rule-based method predict four correct outcomes out of
nine images.
(a) Accuracy versus number of epochs for CNN-32 model.
The results reveal that the performance of the neural network
methods depend on the completeness and balance of the data set.
When the training data set is sparse, the rule-based method can
satisfactorily work for cases that are not seen in the training data
set.

5 CONCLUSION
We have presented several algorithms for the automatic extrac-
tion of results from photographs of microbiological water quality
tests. In particular, we devised a rule-based approach. We have
qualitatively and quantitatively evaluated the performance of the
different algorithm. It appears that traditional image processing
algorithms performing on region extraction, such as the Snake, are
not sufficiently robust for the task at hand. It also appears that,
(b) Accuracy versus number of epochs for CNN-5 model. while they are methods of choice when a comprehensive training
data set is available, neural networks suffer from small and imbal-
Figure 9: Accuracy versus number of epochs for different anced data sets and their architecture should be designed carefully.
CNN classifiers on the new data set. Fortunately, the knowledge and rule-based method that we devised
and presented performs satisfactorily.
Table 6 presents the predicted outcomes for the unseen data of This work is a modest building block in the development of
the three methods and the corresponding As , Aw and root mean practical tools to facilitate the interpretation of water quality data
iiWAS2019, December 2–4, 2019, Munich, Germany Jifang Xing, Zhang Ruixi, Remmy Zen, Ngurah Agus Sanjaya ER, Laure Sioné, Ismail Khalil, and Stéphane Bressan

collected through citizen science methods, in alignment with sus- [24] Cornell Lab of Ornithology. 2016. What is Citizen Science and PPSR? Available
tainable water management practices. at http://www.birds.cornell.edu/citscitoolkit/about/defining-citizen-science/.
[25] World Health Organization et al. 2011. Technical notes on drinking water, sani-
tation and hygiene in emergencies.
ACKNOWLEDGMENTS [26] Simon JD Prince. 2012. Computer vision: models, learning, and inference. Cam-
bridge University Press.
Research reported in this publication was funded by Singapore [27] Waseem Rawat and Zenghui Wang. 2017. Deep convolutional neural networks
for image classification: A comprehensive review. Neural computation 29, 9 (2017),
Institute for Data Science under the WATCHA project, Imperial 2352–2449.
College London P/L016826/1 EPSRC Centre for Doctoral Training [28] Todd R Reed and JM Hans Dubuf. 1993. A review of recent texture segmentation
in Sustainable Civil Engineering, the Austrian Federal Ministry and feature extraction techniques. CVGIP: Image understanding 57, 3 (1993),
359–372.
of Education, Science and Research and jointly supported by the [29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional
ASEAN-European Academic University Network (ASEA-UNINET) networks for biomedical image segmentation. In International Conference on
and the Austrian Agency for International Cooperation in Educa- Medical image computing and computer-assisted intervention. Springer, 234–241.
[30] Ethan Rublee, Vincent Rabaud, Kurt Konolige, and Gary R Bradski. 2011. ORB:
tion and Research (OeAD-GmbH) project ASEA 2019/Uni Linz/2. An efficient alternative to SIFT or SURF.. In ICCV, Vol. 11. Citeseer, 2.
[31] Martine Rutten, Ellen Minkman, and Maarten van der Sanden. 2017. How to get
and keep citizens involved in mobile crowd sensing for water management? A
REFERENCES review of key success factors and motivational aspects. Wiley Interdisciplinary
[1] Akvo. 2008. Akvo Caddisfly. https://akvo.org/flow-caddisfly-lumen/. Reviews: Water 4, 4 (2017), e1218.
[2] Aquagenx. 2012. Water Quality Test Ki. https://www.aquagenx.com/. [32] Miss Meenu Sahu, Amit Saxena, and Manish Manoria. 2015. Application of
[3] Roberto Brunelli. 2009. Template matching techniques in computer vision: theory feature extraction technique: A review. IJCSIT 4 (2015), 3014–3016.
and practice. John Wiley & Sons. [33] Amadu Salifu, Helen MK Essandoh, Afsatou Ndama Traore, and Natasha Potgieter.
[4] John Canny. 1987. A computational approach to edge detection. Elsevier. 184–203 2019. Water source quality in Ahenema Kokoben, Ghana. Journal of Water,
pages. Sanitation and Hygiene for Development 9, 3 (2019), 450–459.
[5] Dan C Cireşan, Ueli Meier, Jonathan Masci, Luca M Gambardella, and Jürgen [34] Ishwar K Sethi, Ioana L Coman, and Daniela Stan. 2001. Mining association rules
Schmidhuber. 2011. Flexible, high performance convolutional neural networks between low-level image features and high-level concepts. In Data Mining and
for image classification. In Proceedings of the Twenty-Second international joint Knowledge Discovery: Theory, Tools, and Technology III, Vol. 4384. International
conference on Artificial Intelligence-Volume Volume Two. AAAI, 1237–1242. Society for Optics and Photonics, SPIE, 279–290.
[6] Claudio CUSANO, Gianluigi CIOCCA, and Raimondo SCHETTINI. 2004. Im- [35] Wei Shen, Xinggang Wang, Yan Wang, Xiang Bai, and Zhijiang Zhang. 2015.
age annotation using SVM. In SPIE proceedings series. Society of Photo-Optical Deepcontour: A deep convolutional feature learned by positive-sharing loss for
Instrumentation Engineers, SPIE proceedings series, 330–338. contour detection. In Proceedings of the IEEE conference on computer vision and
[7] Economic and Social Council. 2019. Special edition: progress towards the Sustain- pattern recognition. IEEE, 3982–3991.
able Development Goals. United Nations. Available at https://undocs.org/E/2019/ [36] Laure Sione, Michael R. Templeton, and Christian Onof. 2019. Citizen science to
68. monitor water supply intermittency and quality in developing countries.
[8] K-S Goh, Edward Y Chang, and Beitao Li. 2005. Using one-class and two-class [37] Jack Sklansky. 1978. Image segmentation and feature extraction. IEEE Transactions
SVMs for multiclass image annotation. IEEE Transactions on Knowledge and Data on Systems, Man, and Cybernetics 8, 4 (1978), 237–247.
Engineering 17, 10 (2005), 1333–1346. [38] Nobuhisa Ueda and Tadaaki Nagao. 1996. NP-completeness results for NONOGRAM
[9] Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep learning. MIT via parsimonious reductions. Technical Report. Tokyo Institute of Technology.
press. [39] Stefan Van der Walt, Johannes L Schönberger, Juan Nunez-Iglesias, François
[10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017. Mask Boulogne, Joshua D Warner, Neil Yager, Emmanuelle Gouillart, and Tony Yu.
r-cnn. In Proceedings of the IEEE international conference on computer vision. IEEE, 2014. scikit-image: image processing in Python. PeerJ 2 (2014), e453.
2961–2969. [40] William F Vásquez, Pallab Mozumder, Jesus Hernandez-Arce, and Robert P
[11] Andreja Jonoski, Leonardo Alfonso, Adrian Almoradie, Ioana Popescu, Schalk Jan Berrens. 2009. Willingness to pay for safe drinking water: Evidence from Parral,
van Andel, and Zoran Vojinovic. 2012. Mobile phone applications in the water Mexico. Journal of environmental management 90, 11 (2009), 3391–3400.
domain. [41] Networking water. 2019. Networking water data project. Available at https:
[12] Michael Kass, Andrew Witkin, and Demetri Terzopoulos. 1988. Snakes: Active //www.networkingwater.com.
contour models. International journal of computer vision 1, 4 (1988), 321–331. [42] Roger CF Wong and Clement HC Leung. 2008. Automatic semantic annotation
[13] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. Imagenet classifica- of real-world web images. IEEE Transactions on Pattern Analysis and Machine
tion with deep convolutional neural networks. In Advances in neural information Intelligence 30, 11 (2008), 1933–1944.
processing systems. NIPS Proceedings, 1097–1105. [43] Saining Xie and Zhuowen Tu. 2015. Holistically-nested edge detection. In Pro-
[14] Gaurav Kumar and Pradeep Kumar Bhatia. 2014. A detailed review of feature ceedings of the IEEE international conference on computer vision. IEEE, 1395–1403.
extraction in image processing systems. In 2014 Fourth international conference [44] Mingqiang Yang, Kidiyo Kpalma, and Joseph Ronsin. 2008. A survey of shape
on advanced computing & communication technologies. IEEE, 5–12. feature extraction techniques.
[15] Emily Kumpel and Kara L Nelson. 2016. Intermittent water supply: prevalence, [45] Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolu-
practice, and microbial water quality. Environmental science & technology 50, 2 tional networks. In European conference on computer vision. Springer, 818–833.
(2016), 542–553. [46] Dengsheng Zhang, Md Monirul Islam, and Guojun Lu. 2012. A review on auto-
[16] Yann Le Cun, Lionel D Jackel, Brian Boser, John S Denker, Henry P Graf, Isabelle matic image annotation techniques. Pattern Recognition 45, 1 (2012), 346–362.
Guyon, Don Henderson, Richard E Howard, and William Hubbard. 1989. Hand-
written digit recognition: Applications of neural network chips and automatic
learning. IEEE Communications Magazine 27, 11 (1989), 41–46.
[17] Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature
521, 7553 (2015), 436.
[18] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. 1998. Gradient-
based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278–
2324.
[19] Thomas Leeuw. 2014. Crowdsourcing water quality data using the iPhone camera.
Master’s thesis. University of Maine.
[20] Martin D Levine. 1969. Feature extraction: A survey. Proc. IEEE 57, 8 (1969),
1391–1407.
[21] Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015. Fully convolutional
networks for semantic segmentation. In Proceedings of the IEEE conference on
computer vision and pattern recognition. IEEE, 3431–3440.
[22] mWater. 2012. mWater Surveyor Mobile App. https://www.mwater.co/surveyor.
html.
[23] Mark Nixon and Alberto S Aguado. 2012. Feature extraction and image processing
for computer vision. Academic Press.