This article has been accepted for inclusion in a future issue of this journal.

Content is final as presented, with the exception of pagination.

IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS 1

Improving the Accuracy and Hardware Efficiency

of Neural Networks Using Approximate Multipliers
Mohammad Saeed Ansari , Student Member, IEEE, Vojtech Mrazek , Member, IEEE,
Bruce F. Cockburn , Member, IEEE, Lukas Sekanina , Senior Member, IEEE,
Zdenek Vasicek , and Jie Han , Senior Member, IEEE

Abstract— Improving the accuracy of a neural network (NN) data mining, and neural network (NN)-based classifiers are
usually requires using larger hardware that consumes more especially required for computational resources. Approximate
energy. However, the error tolerance of NNs and their appli- computing is an emerging design paradigm that can reduce
cations allow approximate computing techniques to be applied
to reduce implementation costs. Given that multiplication is the the system cost without reducing the system effectiveness.
most resource-intensive and power-hungry operation in NNs, It leverages the inherent error tolerance of many applications,
more economical approximate multipliers (AMs) can significantly such as machine learning, multimedia processing, pattern
reduce hardware costs. In this article, we show that using recognition, and computer vision, to allow some accuracy to be
AMs can also improve the NN accuracy by introducing noise. traded off to save hardware cost [1]. NNs are now recognized
We consider two categories of AMs: 1) deliberately designed
and 2) Cartesian genetic programing (CGP)-based AMs. The as providing the most effective solutions to many challenging
exact multipliers in two representative NNs, a multilayer percep- pattern recognition and machine learning tasks such as image
tron (MLP) and a convolutional NN (CNN), are replaced with classification [2]. Due to their intrinsic error tolerance char-
approximate designs to evaluate their effect on the classification acteristics and high computation and implementation costs,
accuracy of the Mixed National Institute of Standards and there is increasing interest in using approximation in NNs.
Technology (MNIST) and Street View House Numbers (SVHN)
data sets, respectively. Interestingly, up to 0.63% improvement in Approximation in the memories, where the synaptic weights
the classification accuracy is achieved with reductions of 71.45% are stored [3], approximation in the computation, such as using
and 61.55% in the energy consumption and area, respectively. approximate multipliers (AMs) [4], [5] and approximation
Finally, the features in an AM are identified that tend to make in neurons [6], [7], are all strategies that have already been
one design outperform others with respect to NN accuracy. Those reported in the literature.
features are then used to train a predictor that indicates how well
an AM is likely to work in an NN. Given that multipliers are the main bottleneck of NNs
[8]–[10], this article focuses on the use of AMs in NNs.
Index Terms— Approximate multipliers (AMs), Cartesian The work in [11] showed that using approximate adders
genetic programing (CGP), convolutional NN (CNN), multi-layer
perceptron (MLP), neural networks (NNs). (with reasonable area and power savings) has an unacceptable
negative impact on the performance of NNs, so only exact
adders are used in this article.
I. I NTRODUCTION Several AMs have been proposed in the literature that
decrease the hardware cost, while maintaining acceptably
T HE increasing energy consumption of computer systems
still remains a serious challenge in spite of advances in
energy-efficient design techniques. Today’s computing systems
high accuracy. We divide the AMs into two main categories:
1) deliberately designed multipliers, which include designs that
are increasingly used to process huge amounts of data, and are obtained by making some changes in the truth table of
they are also expected to present computationally demanding the exact designs [12] and 2) Cartesian genetic programing
natural human interfaces. For example, pattern recognition, (CGP)-based multipliers, which are designs that are generated
automatically using the CGP heuristic algorithm [13]. Note
Manuscript received June 2, 2019; revised August 4, 2019; accepted that there are other classes of AMs that are based on analog
September 3, 2019. This work was supported in part by the Natural Sci-
ences and Engineering Research Council of Canada (NSERC) under Project mixed-signal processing [14], [15]. However, they are not
RES0018685 and Project RES0025211; and in part by the INTER-COST considered in this article since our focus is on digital design
under project LTC18053. (Corresponding author: Mohammad Saeed Ansari.) that is more flexible in implementation than analog-/mixed-
M. S. Ansari, B. F. Cockburn, and J. Han are with the Department
of Electrical and Computer Engineering, University of Alberta, Edmonton, signal-based designs.
AB T6G 1H9, Canada (e-mail: [email protected]; [email protected]; There is a tradeoff between the accuracy and the hardware
[email protected]). cost, and there is no single best design for all applications.
V. Mrazek, L. Sekanina, and Z. Vasicek are with the IT4Innovations
Centre of Excellence, Faculty of Information Technology, Brno University Thus, selecting the appropriate AM for any specific appli-
of Technology, 612 66 Brno, Czech Republic (e-mail: [email protected]; cation is a complex question that typically requires careful
[email protected]; [email protected]). consideration of multiple alternative designs. In this article,
Color versions of one or more of the figures in this article are available
online at http://ieeexplore.ieee.org. the objective is to find the AMs that improve the performance
Digital Object Identifier 10.1109/TVLSI.2019.2940943 of an NN, i.e., by reducing the hardware cost while preserving
1063-8210 c 2019 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

2 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

an acceptable output accuracy. To the best of our knowledge, where N denotes the number of neurons in layer l − 1 and
this article is the first that attempts to find the critical features wi j,l−1 denotes the connection weight between the neuron i
of an AM that make it superior to others for use in an NN. in layer l − 1 and the neuron j in layer l [2].
Our benchmark multipliers, including 500 CGP-based AMs SVHN is a data set of house digit images taken from Google
and 100 variants of deliberately designed multipliers, are eval- Street View images [18]. The data set contains 73 257 images
uated for two standard NNs: a multi-layer perceptron (MLP) for training and 26 032 images for testing. Each digit is
that classifies the MNIST data set [16] and a convolutional NN represented as a pair of a 32 × 32 RGB image and its label.
(CNN), LeNet-5 [17], that classifies the SVHN data set [18]. We used LeNet-5 [17] to classify this data set. This CNN
After each network is trained while using double-precision consists of two sets of convolutional and average pooling
floating-point exact multipliers, the accurate multipliers are layers, followed by a third convolutional layer, and then a
replaced with one approximate design (selected from the set fully-connected layer. It also uses ReLU AF, which simply
of benchmark multipliers), and then five steps of retraining are implements max(0, x). The convolutional and fully connected
performed. This process is repeated for each of the benchmark layers account for 98% of all the multiplications [13], therefore
multipliers, resulting in 600 variants for each of the two con- approximation is applied only to these layers. In order to
sidered NNs. The retraining is done for each AM only once. reduce the complexity, we converted the original 32 × 32 RGB
Then, the inference is performed to evaluate the accuracy. images to 32 × 32 grayscale images using the standard “luma”
Since the simulations always start from the same point, i.e., we mapping [13]
run the retraining steps on the pre-trained network (with exact
multipliers), there is no randomness, and therefore the results Y = 0.299 × R + 0.587 × G + 0.114 × B (2)
will be consistent if the simulation is repeated. where R, G, and B denote the intensities of red, green, and
The rest of this article is organized as follows. Section II blue additive primaries, respectively.
specifies the considered networks and different types of To train an NN, the synaptic weights are initialized to
AMs. Section III evaluates the considered multipliers from random values. Then, the network is trained by using the
two perspectives: 1) application-independent metrics and standard backpropagation-based supervised learning method.
2) application-dependent metrics, and discusses the implica- During the training process, the weights are adjusted to
tions of the results. Section IV is devoted to feature selection reduce the error. Instead of starting the training with ran-
and describes how the most critical features in an AM can dom initial weights, one can use the weights of a previ-
be identified. Section V discusses the error and hardware ously trained network. Initializing the weights in this way
characteristics of the AMs and recommends the five best AMs. is referred to as using a pre-trained network [2]. Note that
For further performance analysis, these five multipliers are a pretrained network can be retrained and used to perform
then used to implement an artificial neuron. Finally, Section VI a different task on a different data set. Usually, only a few
summarizes and concludes this article. steps of retraining are required to fine-tune the pre-trained
network.
II. P RELIMINARIES
This section provides background information on the two B. Approximate Multipliers
benchmark NNs and describes the considered AMs.
Through comprehensive simulations, we confirmed that
8-bit multipliers are just wide enough to provide reasonable
A. Employed Neural Networks and Data Sets
accuracies in NNs [10], [20]. Therefore, only 8-bit versions
MNIST (Mixed National Institute of Standards and Tech- of the approximate multipliers were evaluated in this article.
nology) is a data set of handwritten numbers that consists 1) Deliberately Designed Approximate Multipliers: Delib-
of a training set of 60 000 and a test set of 10 000 28 × 28 erately designed AMs are obtained by making carefully chosen
images and their labels [16]. We used an MLP network with simplifying changes in the truth table of the exact multiplier.
784 input neurons (one for each pixel of the monochrome In general, there are three ways of generating AMs [12], [21]:
image), 300 neurons in the hidden layer, and ten output 1) approximation in generating the partial products, such as
neurons, whose outputs are interpreted as the probability the under-designed multiplier (UDM) [22]; 2) approximation
of each of the classification into ten target classes (digits in the partial product tree, such as the broken-array multiplier
0 to 9) [16]. This MLP uses the sigmoid activation function (BAM) [23] and the error-tolerant multiplier (ETM) [24];
(AF). An AF introduces nonlinearity into the neuron’s output and 3) approximation in the accumulation of partial products,
and maps the resulting values onto either the interval [−1, 1] such as the inaccurate multiplier (ICM) [25], the approximate
or [0, 1] [19]. Using the sigmoid AF, the neuron j in layer l, compressor-based multiplier (ACM) [26], the AM [27], and
where 0 < l ≤ lmax , computes an AF of the weighted sum of the truncated AM (TAM) [28]. The other type of deliberately
its inputs, x j,l , as given by designed AM that is considered in this article is the recently
1 proposed alphabet set multiplier (ASM) [10].
x j,l =
1 + e−sum j,l Here, we briefly review the design of the deliberately
N designed AMs.
sum j,l = x i,l−1 × wi j,l−1 (1) The UDM [22] is designed based on an approximate 2 × 2
i=1 multiplier. This approximate 2 × 2 multiplier produces 1112,
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

ANSARI et al.: IMPROVING THE ACCURACY AND HARDWARE EFFICIENCY OF NNs USING APPROXIMATE MULTIPLIERS 3

instead of 10012 to save one output bit when both of the inputs TABLE I
are 112 . C ONSIDERED F EATURES OF THE E RROR F UNCTION
The BAM [23] omits the carry-save adders for the least
significant bits (LSBs) in an array multiplier in both the
horizontal and vertical directions. In other words, it truncates
the LSBs of the inputs to permit a smaller multiplier to be
used for the remaining bits.
The ETM [24] divides the inputs into separate MSB and
LSB parts that do not necessarily have equal widths. Every
bit position in the LSB part is checked from left to right and
if at least one of the two operands is 1, checking is stopped
and all of the remaining bits from that position onward are set
to 1. On the other hand, normal multiplication is performed
for the MSB part.
The ICM [25] uses an approximate (4:2) counter to build
III. E VALUATION OF A PPROXIMATE M ULTIPLIERS IN
AMs. The approximate 4-bit multiplier is then used to con-
N EURAL N ETWORKS
struct larger multipliers.
The ACM [26] is designed by using approximate 4:2 com- This section considers both application-dependent and
pressors. The two proposed approximate 4:2 compressors application-independent metrics to evaluate the effects of AMs
(AC1 and AC2) are used in a Dadda multiplier with four in NNs.
different schemes.
The AM [27] uses a novel approximate adder that generates A. Application-Independent Metrics
a sum bit and an error bit. The error of the multiplier is then Application-independent metrics measure the design fea-
alleviated by using the error bits. The truncated version of the tures that do not change from one application to another. Given
AM multiplier is called the TAM [28]. that AMs are digital circuits, these metrics can be either error
The ASM [10] decomposes the multiplicand into short bit or hardware metrics. Error function metrics are required for
sequences (alphabets) that are multiplied by the multiplier. the feature selection analysis.
Instead of multiplying the multiplier with the multiplicand, The main four error metrics are the error rate (ER), the error
some lower-order multiples of the multiplier are first calcu- distance (ED), the absolute ED (AED), and the relative ED
lated (by shift and add operations) and then some of those (RED). We evaluated all 600 multiplier designs using the
multiples are added in the output stage of the ASM [10]. nine features extracted from these four main metrics, as given
It should be noted that the ASM design was optimized for in Table I. All of the considered multipliers were implemented
use in NNs, and so it is not directly comparable to the in MATLAB and simulated over their entire input space,
other AMs considered in this article when used in other i.e., for all 256 × 256 = 65536 combinations.
applications. The definitions for most of these features are given in
Based on these main designs, variants were obtained by
changing the configurable parameter in each design, forming ED = E−A
A
a set of 100 deliberately designed approximate multipliers. For RED = 1 −
example, removing different carry-save adders from the BAM E
AED = |E − A|
multiplier results in different designs; also, the widths of the  

MSB and LSB parts in the ETM multiplier can be varied to  1 N
RMSED =  × (Ai − E i ) 2 (3)
yield different multipliers. N
i=1
2) CGP-Based Approximate Multipliers: Unlike the delib-  2
erately designed AMs, the CGP-based designs are generated 1 N
1 N
automatically using CGP [13]. Although several heuristic VarED = × EDi − × EDi .
N N
approaches have been proposed in the literature for approx- i=1 i=1
imating a digital circuit, we used CGP, since it is intrinsically Those that are not given in (3) are evident from the
multi-objective and has been successfully used to generate description. Note that E and A in (3) refer to the exact and
other high-quality approximate circuits [29]. approximate multiplication results, respectively. Also, note that
A candidate circuit in CGP is modeled as a 2-D array of the mean-/ variance-related features in Table I are measured
programable nodes. The nodes in this problem are the 2-input over the entire output domain of multipliers (N = 65536),
Boolean functions, i.e., AND, OR, XOR, and others. The initial i.e., 256 × 256 = 65536 cases for the employed eight-bit
population P of CGP circuits includes several designs of exact multipliers.
multipliers and a few circuits that are generated by performing Note that the variance and the root mean square (RMS) are
mutations on accurate designs. Single mutations (by randomly distinct metrics, as specified in (3). Specifically, the variance
modifying the gate function, gate input connection, and/or pri- measures the spread of the data around the mean, while the
mary output connections) are used to generate more candidate RMS measures the spread of the data around the best fit. In the
solutions. More details are provided in [13] and [29]. case of error metrics, the best possible fit is zero.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

4 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

steps are considered (we performed five retraining steps), 4-bit

designs can be used with only 2% degradation in classification
accuracy compared to 8-bit designs. Note that the 8-bit designs
were found to be only 0.04% less accurate than the 12-bit
designs.
Interestingly, we observed that almost all of the AMs result
in similar classification accuracies for the MNIST data set,
regardless of the circuit design. This was expected, since
MNIST is a relatively easy data set to classify. This bodes
well for the use of cheaper, AM designs. The SVHN data set,
however, shows a drop in classification accuracy more clearly
than the MNIST data set when reduced-width multipliers are
used. This might be due to the fact that SVHN data are harder
to classify than the MNIST data.

Fig. 1. Effects of multiplier size on classification accuracy. C. Overfitting

An interesting finding from this article is the observation
that a few AMs have slightly improved the classification
We found that the majority of the 100 deliberately designed accuracy over the exact multipliers. This is a potentially
AMs either always overestimate or always underestimate the significant result, since it means we can use less hardware
accurate product of the multiplication. This can be expected and yet get better results. We believe that overfitting in NNs
to cause problems when these multipliers are used in repet- may be the main reason for this interesting result.
itive or iterative operations, such as matrix multiplications. Overfitting happens when the network is trained so much
In those cases, the errors do not cancel out and are instead that it produces overly complex and unrealistic class bound-
accumulated. On the other hand, most of the CGP-generated aries when deciding whether to classify a data point into one
AMs sometimes overestimate and sometimes also underes- class or another [31]. An overfitted network performs well on
timate the product. This leads to some error cancellation the training data, since it effectively memorizes the training
and tends to make these multipliers better suited for use examples, but it performs poorly on test data because it has
in NNs. not learned to generalize to a larger population of data values.
All of the multipliers were implemented in VHSIC Hard- Several solutions have been proposed in the literature to avoid
ware Description Language (VHDL) and/or Verilog and syn- overfitting such as dropout [31], weight decay [32], early
thesized using the Synopsys Design Compiler (DC) for the stopping [33], and learning with noise [34]–[39].
STMicroelectronics CMOS 28-nm process to obtain the most Dropout techniques help to avoid overfitting by omitting
important hardware metrics: the power dissipation, the circuit neurons from an NN. More specifically, for each training case,
area, and the critical path delay. These hardware metrics are a few neurons are selected and removed from the network,
useful for identifying the most hardware-efficient multiplier along with all their input and output connections [31]. Weight
among those with similar error characteristics. decay is another strategy to handle overfitting in which a
We also generated 500 AMs using the CGP algorithm. The weight-decay term is added to the objective function. This
Verilog, C, and MATLAB codes for all the designs and their term reduces the magnitude of the trained weights and makes
error and hardware characteristics can be found in [30]. the network’s output function smoother, and consequently
improves the generalization (i.e., a well-generalized NN can
B. Application-Dependent Metrics more accurately classify unseen data from the same population
The classification accuracies of the MLP and LeNet-5 net- as the learning data) and reduces the overfitting [32]. Early
works were evaluated over the MNIST and SVHN data sets, stopping approaches stop the training process as soon as a
respectively. All 600 of the AM designs (100 deliberately pre-defined threshold value for classification accuracy has been
designed and 500 CGP-based AMs) were employed in both achieved [33].
NNs, and their classification accuracy was calculated. Last but not least, the addition of noise to the synaptic
The effect of multiplier size on the classification accuracy weights of NNs has been found to be a low-overhead technique
is shown in Fig. 1, where different-sized exact multipliers, for improving the performance of an NN [35]. Murray and
ranging in width from 4 to 12 bits (including the sign bit), are Edwards [37] report up to an 8% improvement in the clas-
shown. Note that the multiplication is performed on integer sification accuracy by injecting stochastic noise into synaptic
numbers. The original values in the range [−1, 1] are mapped weights during the training phase. The noise injected into the
and rounded to the closest integers, with 1 being mapped to synaptic weights in NNs can be modeled as either additive or
the maximum representable value, as determined by the size multiplicative noise [38], [39], as defined in
of the multiplier. Additive noise : Wi∗j = Wi j + δi j
The results show that without performing the retraining
Multiplicative noise : Wi∗j = Wi j δi j (4)
steps, the 6-bit multiplier is the smallest design that is able to
provide acceptable results. On the other hand, when retraining and both have been found to be beneficial.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

ANSARI et al.: IMPROVING THE ACCURACY AND HARDWARE EFFICIENCY OF NNs USING APPROXIMATE MULTIPLIERS 5

In (4), δi j denotes the injected noise and Wi j denotes the

noisy synaptic weight between the i th neuron in layer L and
the j th neuron in layer L + 1. The input of neuron j in layer
L + 1, denoted n j , is calculated as

NL
nj = x i × wi j (5)
i=1
where N L is the number of neurons in layer L and x i and
wi j denote a neuron’s output and its connection weight to
neuron j , respectively. If the exact multiplication in (5) is
replaced with an approximate one, the approximate product
for multiplicand a and multiplier b is given by
M(a, b) = a × b + (a, b) (6)
where the dither (error function) (a, b) is the function
Fig. 2. MNIST classification accuracy, training, and testing with additive
that expresses the difference between the output of an exact Gaussian noise.
multiplier and an AM. By combining (5) and (6), we obtain

NL 
NL
will degrade the classification accuracy. Note that the classifi-
nj = x i × wi j = M(x i , wi j )
cation accuracies in Fig. 2 are normalized to the classification
i=1 i=1
accuracy obtained by using exact multipliers.

NL
Additionally, we injected Gaussian noise with positive and
≈ approximate multipliers M  (x i , wi j )
−−−−−−−−−−−−−−−−→ i=1 negative offsets in our accuracy analysis in Fig. 2 to show the
NL 
negative effect of biased noise on the classification accuracy.
 For the biased noise, the errors are more likely to accumulate,
= (x i × wi j ) + (x i , wi j )
and therefore the accuracy drops. The mean is changed to
i=1
NL   1.1 × EMP and 0.9 × EMP to model the positive and negative
 (x i , wi j ) 
NL
offsets, respectively.
= x i × wi j + = x i × wi∗j .
xi
i=1 i=1
IV. C RITICAL F EATURES OF M ULTIPLIERS
(7)
FOR N EURAL N ETWORKS
Note that the noise term (x i , wi j ) in (7) depends on the In Section III, we showed that adding noise to the multi-
multiplier x i , and is a different function for each individual pliers can improve the accuracy of an NN. We also modeled
design. Hence, we cannot compare the result in (7) to the def- the difference between an exact multiplier and an approximate
initions given in (4), since (x i , wi j ) is an unknown function one using the error function (x i , wi j ) of the AM; see (7).
that changes for different multipliers. However, we hypothe- In this section, we consider different multipliers to investigate
size that the same argument that adding noise to the synaptic what properties of the error function might make one design
weights, as we did in (7), can sometimes help to avoid superior to others when employed in an NN.
overfitting in NNs. As previously mentioned, the error function depends on
To provide experimental support for this hypothesis, the multiplier and is a different function for each individual
we built an analytical AM, which is defined as design. An exact analysis of the error functions for different
M  (a, b) = a × b +  (8) multipliers is impractical, and so instead we sought the rele-
vant features of the error functions. Nine seemingly relevant
where  denotes the injected noise. We added Gaussian features of the error function were identified, and are listed
noise, since it is the most common choice in the literature in Table I. In order to determine the most discriminative
[34]–[36]. We used this noise-corrupted exact multiplier in an features of the error functions, i.e., the features that contribute
MLP (784-300-10) and tested it over the MNIST data set. the most to the performance of an AM in an NN, the nine
Fig. 2 shows how the accuracy is affected by increasing the features in Table I were applied to several statistical feature
noise levels. Note that the noise’s mean and standard deviation selection tools (as described next).
in the noise-corrupted multiplier are the exact multiplication To be able to run feature selection algorithms, the mul-
product (EMP) and a percentage of the EMP, respectively. This tipliers were classified into two categories based on their
percentage is given by the term noise level in Fig. 2. performance in NNs. We defined a threshold accuracy, Ath ,
Since the added Gaussian noise is stochastic, we ran the and classified the multipliers that produce higher accuracies
simulations ten times and report the average results. The than Ath into class 1, while the others into class 0. Since in
results in Fig. 2 confirmed the results in [34] and [39]: adding the NN accuracy analysis some AMs produce slightly higher
small amounts of noise can indeed improve the classification classification accuracies than exact multipliers when employed
accuracy. However, as shown in Fig. 2, adding too much noise in NNs, it was convenient to choose Ath = ACCExact , which
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

6 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

is the NN classification accuracy that is obtained when exact TABLE II

multipliers are employed in the network’s structure. Note that R ANKING OF E RROR F UNCTION F EATURES
the average noise level for class 1 AMs is 2.61%, which is
close to the obtained noise level range in Fig. 2.

A. Feature Selection
Feature selection is a statistical way of removing less
relevant features that are not as important to achieving accurate
classification performance. There are many potential benefits
to feature selection including facilitating data understanding
and space dimensionality reduction [40], [41]. In this article,
feature selection algorithms are used to select a subset of mul-
tipliers’ error function features that are most useful for building
a good predictor. This predictor anticipates the behavior of an
AM in an NN. to be a necessary, but not a sufficient condition for better
Scikit-learn is a free machine learning tool that is widely accuracy.
used for feature selection [42]. It accepts an input data array Given that class 1 AMs tend to have smaller Var-ED and
and their corresponding labels to build an estimator that imple- RMS-ED values and the observation that double-sided errors
ments a fitting method. We used the three classifiers, recursive are necessary for a good AM, the difference in the error
feature elimination (RFE) [43], mutual information (MI) [44], magnitude should be small to meet the RMS-ED requirement
and Extra-Tree [45]. i.e., having small RMS-ED values. Moreover, since the error
The RFE classifier iteratively prunes the least important should be double-sided to have a small variance, these errors
features from the current set of features until the desired should be distributed around zero.
number of features is reached. The i th output of the RFE
corresponds to the ranking position of the i th feature, such that B. Training the Classifier
the selected (i.e., the estimated best) features are assigned a Now, having found the most important features of the error
rank of 1. Note that in RFE, the nested feature subsets contain function of an AM, we can use them to predict how well a
complementary features and are not necessarily individually given AM would work in an NN. In this section, we explain
the most relevant features [43]. MI is another useful fea- how to build a classifier that has the error features of an AM
ture selection technique that relies on nonparametric methods as inputs and predicts if it belongs to class 1 or class 0.
based on entropy estimation from the K -nearest neighbor 1) NN-Based Classifier: The error features of 500 randomly
distances, as described in [44]. Each feature is assigned a selected multipliers were used to train the NN-based classifier
score, where higher scores indicate more important features. and those of the 100 remaining multipliers were used as the
Finally, the tree-based estimators can also be used to compute test samples to obtain the classification accuracy of the trained
feature importance to discard less relevant features. Extra-Tree, model. We designed a three-layer MLP with 20 neurons in the
an extremely randomized tree classifier, is a practical classifier hidden layer and two neurons in the output layer (since we
that is widely used for feature selection [45]. Similar to MI, have two classes of multipliers). The number of neurons in the
the i th output of this classifier identifies the importance of the input layer equals the number of features that are considered
i th feature, such that the higher the output score, the more for classification. The number of considered multiplier error
important the feature is. features that were used as inputs to the NN-based classifier was
The results of each of the three aforementioned feature varied from 1 up to 9 (for nine features, in total, see Table I).
selection algorithms are provided in Table II. The results The resulting classification accuracies, plotted in Fig. 3, reflect
in Table II show that Var-ED is the most important feature how well the classifier classifies AMs into class 1 or class 0.
according to all three classifiers. RMS-ED is another important Note that when fewer than nine features are selected,
metric, i.e., the most important metric according to RFE, the combination of features giving the highest accuracy is
the second-most critical feature in MI, and the third-most sig- reported in Fig. 3. The combination of features is selected
nificant metric in Extra-Tree classifier. Our simulation results according to the results in Table II and is given in Table III.
show that the average value of the Var-ED and RMS-ED To choose two features, for example, the candidate features
features for class 0 multipliers are 20.21× and 6.42× greater are selected from the top-ranked ones in Table II: 1) Var-ED
than those of the class 1 AMs, respectively. and Mean-AED (by Extra-Tree); 2) Var-ED and RMS-ED
Other important features that have a good ranking in the (by MI); and 3) Mean-ED, Var-ED, and RMS-ED (by RFE).
three classifiers are MEAN-AED and VAR-AED. We also For these four features (i.e., Mean-ED, Var-ED, RMS-ED,
observed that the multipliers that produced better accuracies and Mean-AED), we consider all six possible combinations
in an NN than the exact multiplier (class 1 multipliers) all and report the results for the combination that gives the
have double-sided error functions. Thus, they overestimate the highest accuracy. Using the same process as in this example,
actual multiplication product for some input combinations and the feature combinations for which the accuracy is maximized
underestimate it for others. Having double-sided EDs seems were found, and are provided in Table III.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

ANSARI et al.: IMPROVING THE ACCURACY AND HARDWARE EFFICIENCY OF NNs USING APPROXIMATE MULTIPLIERS 7

TABLE III
F EATURE C OMBINATIONS T HAT G IVE THE H IGHEST M ULTIPLIER
C LASSIFICATION A CCURACY

Fig. 3. Effect of the number of selected features on AM classifier accuracy.

achieved. In this section, the SVM classifier is used to predict

the performance of 14 representative AMs in a different
benchmark NN. The SVM classifier is selected since it shows
As shown in Fig. 3, the highest classification accuracy is the best performance compared to other classifiers, see Fig. 3.
achieved when two features are used as inputs to the NN-based Ideally, we would want to verify the classifier using all
classifier, namely Var-ED and RMS-ED. Also, Fig. 3 shows 600 AMs. However, the large number of multipliers in a
that using more than two features does not necessarily result deep NN benchmark and the large number of images in the
in a higher accuracy. data set would make the exhaustive experiment prohibitively
2) MATLAB Classification Learner Application: The time consuming. Therefore, in addition to the 100 previously
MATLAB software environment provides a wide variety considered multipliers, five multipliers were randomly selected
of specialized applications [46]. In particular, the classifier from each class of AMs, plus the two multipliers that provided
learner application, available in the apps gallery, allows us to the best accuracy when used in an NN to classify the SVHN
train a model (classifier) that predicts if a multiplier falls into and MNIST data sets, and the two multipliers that had the
class 0 or class 1 when applied to an NN. This application worst accuracy for those same data sets. The SVM classifier
provides the option of choosing a model type, i.e., decision was used to predict the behavior of each of these multipliers
trees, K -nearest neighbors, support vector machines (SVMs), in a given NN benchmark. Then, these multipliers were used
and logistic classifiers among others. We considered all of in the NN to verify the classifier’s accuracy.
these model types (with their default settings) to find the model AlexNet is considered as the benchmark NN and is trained
that most accurately fits the classification problem. Similarly, to classify the ImageNet data set [47]. AlexNet is a CNN
500 randomly selected multipliers were used to train the model with nine layers: an input layer, five convolution layers, and
and the 100 remaining multipliers were used as test samples three fully connected layers [48]. Note that training a deep
to obtain the classification accuracy of the trained model. CNN over a big data set, such as ImageNet, would be very
Fig. 3 also shows the effect of the number of selected time consuming. Hence, we used the MATLAB pre-trained
features on the accuracy of each of the three considered classi- model and performed ten retraining steps (using the AMs) as
fiers. Note that the SVM- and KNN-based classifiers achieve an alternative to train the network from scratch.
higher accuracies than the decision tree-based classifier. All Table IV shows how the SVM classifier anticipates the
three classifiers achieve better accuracies than the NN-based performance of each of the 14 multipliers (i.e., the five
classifier. randomly selected multipliers from each class of AMs and the
Similar to the NN-based classifier, the classifier’s accuracy four multipliers that provided the best and the worst accuracies
for the combination of features that gives the highest accu- when used in an NN to classify the SVHN and MNIST data
racy is shown in Fig. 3 when fewer than nine features are sets) in AlexNet.
selected. The highest classification accuracy for the SVM- and As shown in Fig. 3, none of the classifiers is 100% accurate.
KNN-based classifiers is achieved when only two features are For instance, AlexNet implemented with the AM M1 has
used as inputs to the classifier: i.e., Var-ED and RMS-ED. a worse accuracy than Ath (i.e., the accuracy of AlexNet
However, the decision tree-based classifier has the highest implemented with exact multipliers) even though the multi-
accuracy when only one feature, Var-ED, is considered. plier is classified into class 1 (see Table IV). However, this
misclassified multiplier produces an accuracy close to Ath and
C. Verifying the Classifiers the difference (0.41%) is small.
The trained SVM classifier was verified in Section III-B While some multipliers might perform well for one data set,
by using 100 AMs, where an accuracy of almost 86% was they might not work well for other data sets. In other words,
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

8 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

TABLE IV data set. Note that a multiplier is considered to be PDP-Pareto

C LASSIFICATION A CCURACY OF A LEX N ET ON THE optimal if there does not exist any other multiplier which
I MAGE N ET LSVRC-2010 D ATA S ET
improves the classification accuracy with the same PDP. It is
clear from Fig. 4 that the Pareto-optimal designs for the two
data sets are different.

V. E RROR AND H ARDWARE A NALYSES OF

A PPROXIMATE M ULTIPLIERS
This section analyzes the error and hardware characteristics
of AMs. Based on this analysis, a few designs that have a
superior performance in both considered data sets are identi-
fied and recommended.

A. Error Analysis
Fig. 5 compares class 0 and class 1 multipliers with respect
to four important error features: Var-ED, RMS-ED, Mean-
AED, and Var-AED. This plot shows how the class 1 and
class 0 multipliers measure differently for the considered
features. As shown in Fig. 5, class 1 multipliers generally have
smaller Mean-AED, Var-ED, Var-AED, and RMS-ED values,
when compared to class 0 multipliers. It also shows, in the
zoomed-in insets, that some class 0 multipliers having smaller
Var-AED, RMS-ED, Mean-AED, and/or Var-ED values than
some class 1 multipliers is the reason why some multipliers
are misclassified by the classifiers.

B. Hardware Analysis
To further understand the quality of AMs, we performed a
hardware analysis. The main hardware metrics of a multiplier,
i.e., power consumption, area, and critical path delay, and PDP,
are considered in this analysis. Note that all of the considered
multipliers in this article are pure combinational circuits for
which the throughput is inversely proportional to the critical
path delay.
Fig. 6 shows two scatter plots that best distinguish the two
classes of AMs, which are area versus delay (see Fig. 6(a))
and power consumption versus delay (see Fig. 6(b)). Note that
only the results for the SVHN data set are shown as the results
for the MNIST are almost the same.
As the results in Fig. 6 show, unlike for the error metrics,
there is no clear general trend in the hardware metrics.
However, the designs with small delay and power consumption
are preferred for NN applications, as discussed next.
As AMs are obtained by simplifying the design of an exact
multiplier, more aggressive approximations can be used to
further reduce the hardware cost and energy consumption.
Fig. 4. NN accuracy using the same AMs for different data sets.
(a) Pareto-optimal design in PDP for the SVHN. (b) Behavior of SVHN As previously discussed, some multipliers have almost similar
Pareto-optimal multipliers for the MNIST. accuracies, while as shown in Fig. 4, they have different
hardware measures. The main reasons are as follows: 1) the
hardware cost of a digital circuit totally depends on how it is
the performance of a multiplier is application dependent. implemented in hardware; e.g., array and Wallace multipliers
To illustrate this claim, we have plotted the Pareto-optimal are both exact designs, and therefore they have the same
designs in power-delay product (PDP) for the SVHN data set classification accuracy. However, they have different hardware
using all 600 AMs in Fig. 4(a). costs and 2) the classification accuracy of NNs is application
Fig. 4(b) shows the performance of the Pareto-optimal dependent and it depends on the network type, the data set,
multipliers in PDP for the SVHN data set for the MNIST the learning algorithm, and the number of training iterations.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

ANSARI et al.: IMPROVING THE ACCURACY AND HARDWARE EFFICIENCY OF NNs USING APPROXIMATE MULTIPLIERS 9

Fig. 6. Hardware comparison between class 0 and class 1 AMs. (a) Area
versus delay for class 1 and class 0 AMs. (b) Power versus delay for class 1
and class 0 AMs.

C. Recommended Approximate Multipliers

This section identifies a few AMs that exhibit superior
performance for both considered data sets. We chose the five
best AMs that produce better accuracies than exact multipliers
when used in the two considered NNs: the MLP for the
MNIST data set and LeNet-5 for the SVHN data set. Note
that these five designs were selected and sorted based on their
low PDP values.
Table V lists and Fig. 6 shows these multipliers. Their
Verilog, C, and MATLAB descriptions can be found online
from [30]. Table V also reports the main hardware charac-
teristics of these designs, i.e., the area, power consumption,
delay, and PDP. The results in Table V indicate that all five
chosen AMs (which are all CGP-based AMs) consume less
power (at least 73%) than the exact multiplier, while providing
slightly higher accuracies (up to 0.18% or more) when they
are used in NNs. Comparing the average area and the PDP
Fig. 5. Classification of class 0 and class 1 multipliers based on the
shows significant savings in hardware cost (i.e., 65.20% and
most important features. (a) Var-ED versus mean-AED. (b)Var-ED versus 81.74% less area and PDP, respectively) by replacing the exact
log10 (Var-AED). (c) Var-ED versus RMS-ED. multipliers with the approximate ones.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

10 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

TABLE V VI. C ONCLUSION

H ARDWARE C HARACTERISTICS OF THE F IVE B EST AM S
This article described the evaluation of a large pool of AMs,
which contained 100 deliberately designed and 500 CGP-
based multipliers, for application in NNs. The exact multipliers
in two benchmark networks, i.e., one MLP and one CNN
(LeNet-5), were replaced after training with AMs to see
how the classification accuracy is affected. The MLP and the
CNN were employed to classify the MNIST and SVHN data
sets, respectively. The classification accuracy was obtained
TABLE VI experimentally for both data sets for all 600 AMs.
E RROR C HARACTERISTICS OF THE F IVE B EST AM S The features in an AM that tend to make it superior to others
with respect to NN accuracy were identified and then used to
build a predictor that forecasts how well an multiplier is likely
to work in an NN. This predictor was verified by classifying
114 AMs based on their performance in LeNet-5 and AlexNet
CNN for the SVHN and ImageNet data sets, respectively.
The major findings of this article are as follows.
1) Unlike most of the CGP-generated AMs, the majority
of the 100 deliberately designed AMs either always
overestimate or always underestimate the actual value of
TABLE VII the multiplication. Hence, the errors in CGP-generated
H ARDWARE C HARACTERISTICS OF AN A RTIFICIAL N EURON multipliers are more likely to cancel out, and therefore
I MPLEMENTED U SING R ECOMMENDED AM S these multipliers are better suited for use in NNs.
2) It is not only possible, but can also be practical and more
economical, to use AMs in the structure of NNs instead
of exact multipliers.
3) NNs that use appropriate AMs can provide higher
accuracies compared to NNs that use the same num-
ber of exact multipliers. This is a significant result
since it shows that a better NN performance can be
The accuracies of the five recommended multipliers when obtained with significantly lower hardware cost while
employed in the two NN workloads are reported in Table VI. using approximation.
Although not an important error feature, the ER is shown 4) It appears that using AMs adds small inaccuracies (i.e.,
in Table VI, together with VAR-ED and RMS-ED, which are approximation noise) to the synaptic weights and this
two critical error features for the performance of an AM in noise helps to mitigate the overfitting problem, and thus
NNs. The results show that the five recommended multipliers improves the NN accuracy.
all have small VAR-ED and RMS-ED values. 5) The most important features that make a design superior
Hardware descriptions (in Verilog) of all of the CGP-based to others are the variance of the ED (Var-ED) and the
AMs can be found online in [30]. By using the Verilog code, RMS of the ED (RMS-ED).
one can easily obtain the truth table and/or the logic circuit Although the statistically most relevant and critical features
for each design. of AMs are identified in this article, a statistically accurate
VAR-ED and RMS-ED, as the two most critical error predictor based on those features cannot guarantee that the
features for the performance of an AM in NNs, are also given best approximate design will be identified: ensuring the best
in Table VI. The results show that the five recommended choice of AM requires application-dependent experimentation.
multipliers all have small VAR-ED and RMS-ED values,
which is consistent with the results in Fig. 5. R EFERENCES
An artificial neuron was also implemented using the five
recommended AMs to replace the exact ones. The imple- [1] J. Han and M. Orshansky, “Approximate computing: An emerging
paradigm for energy-efficient design,” in Proc. 18th IEEE Eur. Test
mented neuron has three inputs and an adder tree composed Symp. (ETS), May 2013, pp. 1–6.
of two adders to accumulate the three multiplication products. [2] J. Schmidhuber, “Deep learning in neural networks: An overview,”
This is a widely used technique for the performance analysis Neural Netw., vol. 61, pp. 85–117, Jan. 2015.
[3] G. Srinivasan, P. Wijesinghe, S. S. Sarwar, A. Jaiswal, and K. Roy,
of multipliers in NNs [10]. “Significance driven hybrid 8T-6T SRAM for energy-efficient synaptic
The hardware characteristics of the implemented neuron storage in artificial neural networks,” in Proc. Design, Autom. Test Eur.
are given in Table VII. The results show that the neurons Conf. Exhib. (DATE), 2016, pp. 151–156.
[4] T. Na and S. Mukhopadhyay, “Speeding up convolutional neural net-
constructed using the recommended multipliers can be up to work training with dynamic precision scaling and flexible multiplier-
71.45% more energy-efficient than the neuron that uses the accumulator,” in Proc. Int. Symp. Low Power Electron. Design, 2016,
exact multiplier while being 61.55% smaller than it. pp. 58–63.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

ANSARI et al.: IMPROVING THE ACCURACY AND HARDWARE EFFICIENCY OF NNs USING APPROXIMATE MULTIPLIERS 11

[5] M. Courbariaux, Y. Bengio, and J.-P. David, “Training deep neural [29] Z. Vasicek and L. Sekanina, “Evolutionary approach to approximate
networks with low precision multiplications,” 2014, arXiv:1412.7024. digital circuits design,” IEEE Trans. Evol. Comput., vol. 19, no. 3,
[Online]. Available: https://arxiv.org/abs/1412.7024 pp. 432–444, Jun. 2015.
[6] S. Venkataramani, A. Ranjan, K. Roy, and A. Raghunathan, “AxNN: [30] (2016). EvoApprox8b—Approximate Adders and Multipliers Library.
Energy-efficient neuromorphic systems using approximate computing,” [Online]. Available: http://www.fit.vutbr.cz/research/groups/ehw/
in Proc. Int. Symp. Low Power Electron. Design, 2014, pp. 27–32. approxlib/
[7] Q. Zhang, T. Wang, Y. Tian, F. Yuan, and Q. Xu, “ApproxANN: An [31] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
approximate computing framework for artificial neural network,” in R. Salakhutdinov, “Dropout: A simple way to prevent neural networks
Proc. Design, Autom. Test Eur. Conf. Exhib., 2015, pp. 701–706. from overfitting,” J. Mach. Learn. Res., vol. 15, no. 1, pp. 1929–1958,
[8] M. Marchesi, G. Orlandi, F. Piazza, and A. Uncini, “Fast neural networks 2014.
without multipliers,” IEEE Trans. Neural Netw., vol. 4, no. 1, pp. 53–62, [32] C. S. Leung, H.-J. Wang, and J. Sum, “On the selection of weight decay
Jan. 1993. parameter for faulty networks,” IEEE Trans. Neural Netw., vol. 21, no. 8,
[9] Z. Lin, M. Courbariaux, R. Memisevic, and Y. Bengio, “Neural networks pp. 1232–1244, Aug. 2010.
with few multiplications,” 2015, arXiv:1510.03009. [Online]. Available: [33] Y. Shao, G. N. Taff, and S. J. Walsh, “Comparison of early stopping
https://arxiv.org/abs/1510.03009 criteria for neural-network-based subpixel classification,” IEEE Geosci.
[10] S. S. Sarwar, S. Venkataramani, A. Ankit, A. Raghunathan, and K. Roy, Remote Sens. Lett., vol. 8, no. 1, pp. 113–117, Jan. 2011.
“Energy-efficient neural computing with approximate multipliers,” ACM [34] Y. Luo and F. Yang. (2014). Deep Learning With Noise. [Online].
J. Emerg. Technol. Comput. Syst., vol. 14, no. 2, 2018, Art. no. 16. Available: hp://www.andrew.cmu.edu/user/fanyang1/deep-learning-with-
[11] H. R. Mahdiani, M. H. S. Javadi, and S. M. Fakhraie, “Efficient utiliza- noise.pdf
tion of imprecise computational blocks for hardware implementation of [35] N. Nagabushan, N. Satish, and S. Raghuram, “Effect of injected noise
imprecision tolerant applications,” Microelectron. J., vol. 61, pp. 57–66, in deep neural networks,” in Proc. Int. Conf. Comput. Intell. Comput.
Mar. 2017. Res., 2016, pp. 1–5.
[12] H. Jiang, C. Liu, L. Liu, F. Lombardi, and J. Han, “A review, classifi- [36] T. He, Y. Zhang, J. Droppo, and K. Yu, “On training bi-directional
cation, and comparative evaluation of approximate arithmetic circuits,” neural network language model with noise contrastive estimation,” in
ACM J. Emerg. Technol. Comput. Syst., vol. 13, no. 4, p. 60, Aug. 2017. Proc. 10th Int. Symp. Chin. Spoken Lang. Process., 2016, pp. 1–5.
[13] V. Mrazek, S. S. Sarwar, L. Sekanina, Z. Vasicek, and K. Roy, “Design [37] A. F. Murray and P. J. Edwards, “Enhanced MLP performance and fault
of power-efficient approximate multipliers for approximate artificial tolerance resulting from synaptic weight noise during training,” IEEE
neural networks,” in Proc. 35th Int. Conf. Comput.-Aided Design, 2016, Trans. Neural Netw., vol. 5, no. 5, pp. 792–802, Sep. 1994.
pp. 1–7. [38] J. Sum, C.-S. Leung, and K. Ho, “Convergence analyses on on-line
[14] E. H. Lee and S. S. Wong, “Analysis and design of a passive switched- weight noise injection-based training algorithms for MLPs,” IEEE Trans.
capacitor matrix multiplier for approximate computing,” IEEE J. Solid- Neural Netw. Learn. Syst., vol. 23, no. 11, pp. 1827–1840, Nov. 2012.
State Circuits, vol. 52, no. 1, pp. 261–271, Jan. 2017. [39] K. Ho, C.-S. Leung, and J. Sum, “Objective functions of online weight
[15] S. Gopal et al., “A spatial multi-bit sub-1-V time-domain matrix mul- noise injection training algorithms for MLPs,” IEEE Trans. Neural
tiplier interface for approximate computing in 65-nm CMOS,” IEEE J. Netw., vol. 22, no. 2, pp. 317–323, Feb. 2011.
Emerg. Sel. Topics Circuits Syst., vol. 8, no. 3, pp. 506–518, Sep. 2018. [40] I. Guyon, S. Gunn, A. Ben-Hur, and G. Dror, “Result analysis of
the NIPS 2003 feature selection challenge,” in Proc. Adv. Neural Inf.
[16] Y. LeCun, C. Cortes, and C. Burges. (2010). MNIST handwritten
Process. Syst., 2005, pp. 545–552.
digit database. AT&T Labs. [Online]. Available: http://yann.lecun.com/
[41] I. Guyon and A. Elisseeff, “An introduction to variable and feature
exdb/mnist
selection,” J. Mach. Learn. Res., vol. 3, pp. 1157–1182, Jan. 2003.
[17] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based
[42] F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” J. Mach.
learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11,
Learn. Res., vol. 12, pp. 2825–2830, Oct. 2011.
pp. 2278–2324, Nov. 1998.
[43] I. Guyon, J. Weston, S. Barnhill, and V. Vapnik, “Gene selection for
[18] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, cancer classification using support vector machines,” Mach. Learn.,
“Reading digits in natural images with unsupervised feature learning,” in vol. 46, nos. 1–3, pp. 389–422, 2002.
Proc. NIPS Workshop Deep Learn. Unsupervised Feature Learn., 2011, [44] A. Kraskov, H. Stögbauer, and P. Grassberger, “Estimating mutual
p. 5. information,” Phys. Rev. E, Stat. Phys. Plasmas Fluids Relat. Interdiscip.
[19] R. J. Schalkoff, Artificial Neural Networks, vol. 1. New York, NY, USA: Top., vol. 69, no. 6, 2004, Art. no. 066138.
McGraw-Hill, 1997. [45] P. Geurts, D. Ernst, and L. Wehenkel, “Extremely randomized trees,”
[20] N. P. Jouppi et al., “In-datacenter performance analysis of a tensor Mach. Learn., vol. 63, no. 1, pp. 3–42, 2006.
processing unit,” in Proc. ACM/IEEE 44th Annu. Int. Symp. Comput. [46] MathWorks. MATLAB Classification Learner App. Accessed: Oct. 1,
Archit. (ISCA), 2017, pp. 1–12. 2019. [Online]. Available: https://www.mathworks.com/help/stats/
[21] M. S. Ansari, H. Jiang, B. F. Cockburn, and J. Han, “Low-power classificationlearner-app.html
approximate multipliers using encoded partial products and approximate [47] (2015). ImageNet Large Scale Visual Recognition Challenge (ILSVRC).
compressors,” IEEE J. Emerg. Sel. Topics Circuits Syst., vol. 8, no. 3, [Online]. Available: http://www.image-net.org/challenges/LSVRC/
pp. 404–416, Sep. 2018. [48] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classification
[22] P. Kulkarni, P. Gupta, and M. Ercegovac, “Trading accuracy for power with deep convolutional neural networks,” in Proc. Adv. Neural Inf.
with an underdesigned multiplier architecture,” in Proc. 24th Int. Conf. Process. Syst., 2012, pp. 1097–1105.
VLSI Design, 2011, pp. 346–351.
[23] H. R. Mahdiani, A. Ahmadi, S. M. Fakhraie, and C. Lucas, “Bio-inspired
imprecise computational blocks for efficient VLSI implementation of
soft-computing applications,” IEEE Trans. Circuits Syst. I, Reg. Papers,
vol. 57, no. 4, pp. 850–862, Apr. 2010.
[24] K. Y. Kyaw, W. L. Goh, and K. S. Yeo, “Low-power high-speed
multiplier for error-tolerant application,” in Proc. Int. Conf. Electron
Devices Solid-State Circuits, 2010, pp. 1–4.
[25] C.-H. Lin and I.-C. Lin, “High accuracy approximate multiplier with Mohammad Saeed Ansari (S’16) received the
error correction,” in Proc. 31st Int. Conf. Comput. Design, Oct. 2013, B.Sc. and M.Sc. degrees in electrical and electronic
pp. 33–38. engineering from Iran University of Science and
[26] A. Momeni, J. Han, P. Montuschi, and F. Lombardi, “Design and Technology, Tehran, Iran, in 2013 and 2015, respec-
analysis of approximate compressors for multiplication,” IEEE Trans. tively. He is currently working toward the Ph.D.
Comput., vol. 64, no. 4, pp. 984–994, Apr. 2015. degree in electrical and computer engineering at the
[27] C. Liu, J. Han, and F. Lombardi, “A low-power, high-performance University of Alberta, Edmonton, AB, Canada.
approximate multiplier with configurable partial error recovery,” in Proc. His current research interests include approxi-
Design, Autom. Test Eur. Conf. Exhib., 2014, pp. 1–4. mate computing, design of computing hardware for
[28] H. Jiang, J. Han, F. Qiao, and F. Lombardi, “Approximate radix-8 booth emerging machine learning applications, multilayer
multipliers for low-power and high-performance operation,” IEEE Trans. perceptrons (MLPs), convolutional NNs (CNNs) in
Comput., vol. 65, no. 8, pp. 2638–2644, Aug. 2016. particular, and reliability and fault tolerance.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.

12 IEEE TRANSACTIONS ON VERY LARGE SCALE INTEGRATION (VLSI) SYSTEMS

Vojtech Mrazek (M’18) received the Ing. and Ph.D. Zdenek Vasicek received the Ing. and Ph.D. degrees
degrees in information technology from the Faculty in electrical engineering and computer science
of Information Technology, Brno University of Tech- from the Faculty of Information Technology, Brno
nology, Brno, Czech Republic, in 2014 and 2018, University of Technology, Brno, Czech Republic,
respectively. in 2006 and 2012, respectively.
He is currently a Researcher with the Evolv- He is currently an Associate Professor with the
able Hardware Group, Faculty of Information Faculty of Information Technology, Brno University
Technology, Brno University of Technology. He of Technology. His current research interests include
is also a Visiting Postdoctoral Researcher with evolutionary design and optimization of complex
the Department of Informatics, Institute of Com- digital circuits and systems.
puter Engineering, Technische Universität Wien Dr. Vasicek received the Silver and Gold medals
(TU Wien), Vienna, Austria. He has authored or coauthored over 30 con- at HUMIES, in 2011 and 2015, respectively.
ference/journal papers focused on approximate computing and evolvable
hardware. His current research interests include approximate computing,
genetic programming, and machine learning.
Dr. Mrazek received several awards for his research in approximate com-
puting, including the Joseph Fourier Award for research in computer science
and engineering in 2018.

Bruce F. Cockburn (S’86–M’90) received the B.Sc.

degree in engineering physics from Queen’s Uni-
versity, Kingston, ON, Canada, in 1981, and the
M.Math. and Ph.D. degrees in computer science
from the University of Waterloo, Waterloo, ON,
Canada, in 1985 and 1990, respectively.
From 1981 to 1983, he was a Test Engineer and a
Software Designer with Mitel Corporation, Kanata,
ON, Canada. He was a Sabbatical Visitor with
Agilent Technologies, Inc., Santa Clara, CA, USA,
and The University of British Columbia, Vancouver,
BC, Canada, in 2001 and from 2014 to 2015, respectively. He is currently
a Professor with the Department of Electrical and Computer Engineering,
University of Alberta, Edmonton, AB, Canada. His current research interests Jie Han (S’02–M’05–SM’16) received the B.Sc.
include the testing and verification of integrated circuits, FPGA-based hard- degree in electronic engineering from Tsinghua Uni-
ware accelerators, parallel computing, stochastic and approximate computing, versity, Beijing, China, in 1999, and the Ph.D. degree
and bioinformatics. from Delft University of Technology, Delft, The
Netherlands, in 2004.
Lukas Sekanina (M’02–SM’12) received the Ing. He is currently an Associate Professor with the
and Ph.D. degrees from Brno University of Tech- Department of Electrical and Computer Engineering,
nology, Brno, Czech Republic, in 1999 and 2002, University of Alberta, Edmonton, AB, Canada. His
respectively. current research interests include approximate com-
He was a Visiting Professor with Pennsylvania puting, stochastic computing, reliability and fault
State University, Erie, PA, USA, in 2001, and the tolerance, nanoelectronic circuits and systems, and
Centro de Eléctronica Industrial (CEI), Universi- novel computational models for nanoscale and biological applications.
dad Politécnia de Madrid (UPM), Madrid, Spain, Dr. Han was a recipient of the Best Paper Award at the International
in 2012, and a Visiting Researcher with the Depart- Symposium on Nanoscale Architectures (NanoArch 2015) and Best Paper
ment of Informatics, University of Oslo, Oslo, Nor- Nominations at the 25th Great Lakes Symposium on VLSI (GLSVLSI 2015),
way, in 2001. He is currently a Full Professor and NanoArch 2016, and the 19th International Symposium on Quality Electronic
the Head of the Department of Computer Systems, Faculty of Information Design (ISQED 2018). He served as the General Chair for GLSVLSI 2017 and
Technology, Brno University of Technology. the IEEE International Symposium on Defect and Fault Tolerance in VLSI
Dr. Sekanina received the Fulbright Scholarship to work with the NASA Jet and Nanotechnology Systems (DFT 2013). He served as the Technical
Propulsion Laboratory, Caltech, in 2004. He has served as an Associate Editor Program Committee Chair for GLSVLSI 2016 and DFT 2012. He is currently
for the IEEE T RANSACTIONS ON E VOLUTIONARY C OMPUTATION from an Associate Editor of the IEEE T RANSACTIONS ON E MERGING T OPICS IN
2011 to 2014, the Genetic Programming and Evolvable Machines Journal, C OMPUTING (TETC), the IEEE T RANSACTIONS ON N ANOTECHNOLOGY,
and the International Journal of Innovative Computing and Applications. and Microelectronics Reliability (Elsevier Journal).