Available online at www.sciencedirect.

com
Available online at www.sciencedirect.com

ScienceDirect
ScienceDirect
Available online at www.sciencedirect.com

Procedia Manufacturing 00 (2019) 000–000

ScienceDirect
Procedia Manufacturing 00 (2019) 000–000 www.elsevier.com/locate/procedia
www.elsevier.com/locate/procedia

Procedia Manufacturing 42 (2020) 173–180

International Conference on Industry 4.0 and Smart Manufacturing (ISM 2019)

International Conference on Industry 4.0 and Smart Manufacturing (ISM 2019)
Shipbuilding
Shipbuilding Supply
Supply Chain
Chain Framework
Framework and
and Digital
Digital Transformation:
Transformation: A
A
Project Portfolios Risk Evaluation
Project Portfolios Risk Evaluation
Rafael Diaz(a) *, Katherine Smith(b), Rafael Landaeta(c), Antonio Padovano(d)
Rafael Diaz(a)*, Katherine Smith(b), Rafael Landaeta(c), Antonio Padovano(d)
ab
Virginia Modeling Analysis & Simulation Center, Old Dominion University, Suffolk, VA 23435, USA
c abVirginia Modeling Analysis & Simulation Center, Old Dominion University, Suffolk, VA 23435, USA
Frank Batten College of Engineering and Technology, Old Dominion University, Norfolk, VA 23529, USA
c
Frank Batten College
d of Engineering
DIMEG, and
University of Technology, Old Dominion
Calabria, Arcavacata University,
di Rende, 87036,Norfolk,
Italy VA 23529, USA
d
DIMEG, University of Calabria, Arcavacata di Rende, 87036, Italy
* Corresponding author. E-mail address: [email protected]
* Corresponding author. E-mail address: [email protected]

Abstract
Abstract
Program portfolio managers in digital transformation programs have a need for knowledge that can guide decisions related to the alignment of
Program portfolio managers
program investments with theinsustainability
digital transformation programs
and strategic haveofa the
objectives need for knowledge
organization. Thethat can guide
purpose of thisdecisions
research related to the alignment
is to illustrate of
the utility of
program
a frameworkinvestments
capable with the sustainability
of clarifying and strategic
the cost-benefit objectives
tradeoffs of the
stemming organization.
from assessing The purpose ofprogram
digitalization this research is to illustrate
investment risks in the
the utility of
military
ashipbuilding
framework sector.
capableOurof clarifying the cost-benefit
approach uses tradeoffs
Artificial Neural stemming
Network from assessing
to quantify digitalization
benefits and program
risks per project investment
while employing risks in the analysis
scenario military
shipbuilding
to quantify thesector. Our
effects approach uses
of operational ArtificialANeural
constraints. MonteNetwork to quantify
Carlo model is used benefits anddata
to generate riskssamples
per project while employing
that support the executionscenario
of theanalysis
Neural
to quantifyThis
Network. the effects
enablesof operational
the use of constraints. A Monte Carlo
Portfolio Management modelprinciples
Theory is used to generate dataand
to organize samples that support
estimate measuresthe of
execution of the Neural
performance of the
Network. Thisproject
digitalization enablesportfolio.
the use Weof Portfolio Management
demonstrate the utilityTheory
of the principles
frameworktobyorganize
means ofanda estimate measures
theoretical of performance
case study of the
presenting several
digitalization project investment
digitalization project portfolio. We demonstrate
scenarios. the utility
We conclude of framework
that the the framework makesbya means of a and
contribution theoretical
call for case study work
additional presenting
to extendseveral
this
digitalization
framework to project investment
formalize scenarios.
the portfolio We conclude
assessment activitythat theincluding
while framework makes a contribution
acceptable and constrain
risk ranges that call for additional
the final work to extend
fractions this
of budget
framework
allocations. to formalize the portfolio assessment activity while including acceptable risk ranges that constrain the final fractions of budget
allocations.
© 2020
© 2020The TheAuthors.
Authors. Published
Published by Elsevier
by Elsevier B.V.B.V.
© 2020
This
This The
isisan Authors.
anopen
open Published
access
access article
article by Elsevier
under
under B.V.BY-NC-ND
the BY-NC-ND
the CC CC licenselicense (http://creativecommons.org/licenses/by-nc-nd/4.0/)
(http://creativecommons.org/licenses/by-nc-nd/4.0/)
This is an open
Peer-review
Peer-review access
under
under article under
responsibility
responsibility thethe
ofof
the CC BY-NC-ND
scientific
scientific license
committee
committee (http://creativecommons.org/licenses/by-nc-nd/4.0/)
of
of the the International
International Conference
Conference on Industry
on Industry 4.0Smart
4.0 and and Smart Manufacturing.
Manufacturing.
Peer-review under responsibility of the scientific committee of the International Conference on Industry 4.0 and Smart Manufacturing.
Keywords: Project Management; Portfolio Management; Neural Networks; Cost Analysis; Training
Keywords: Project Management; Portfolio Management; Neural Networks; Cost Analysis; Training

1. Introduction executives assert that strategy implementation is critical to

1. Introduction executives
their firms, assert
only 61that strategy
percent implementation
recognize is critical to
that their organizations
Operational and financial pressures in tandem with the their firms, only 61 percent recognize that their
frequently have struggled in bridging the gap between organizations
Operational
increasing and financial
demands pressures
of customers in tandem stress
have intensified with the
for frequently have
strategic goals andstruggled in bridging
their everyday operationthe
[1]. gap between
Furthermore,
increasing
competent demands of The
operations. customers haveofintensified
evolution industrialstress for
systems strategic goals and
in the digital their everydayera,
transformation operation [1]. Furthermore,
a smaller fraction of
competent operations. The evolution of industrial
operating within intricate competitive environments has systems in the digital
executives transformation
confesses to truly era, a smaller
engaging fraction of
in digitalization
operating within intricate
placed substantial competitive
constraints environments and
on the configuration has executives confesses to truly engaging in
projects as they might be considered a risky endeavor [2].digitalization
placed substantial
performance constraints
of agile, efficient, on
andthe configuration
flexible business andand projects
This gapassuggests
they might
a lackbeofconsidered a risky
understanding thatendeavor [2].
projects and
performance of agile, efficient,
institutional operations. Successfuland flexible
firms business
recognize that and
the This gap suggests a lack of understanding that
programs are catalysts in strategic change [3] as they projects and
institutional operations. Successful firms recognize
continual enhancement of existing operations is essential that the
to programs versatile
represent are catalysts in strategic
structures that enablechange [3] as they
the realization of
continual
gaining and enhancement of existing
maintaining operations isedge.
a competitive essential
Theseto represent
distinctiveversatile structures
and innovative that enable
organizational the realization
offerings [4]. of
gaining and maintaining a competitive edge.
organizations engage in strategic management initiatives by These distinctive and innovative organizational offerings [4].
Aligning digitalization projects or programs with strategic
organizations engage in project,
employing effective strategic management
program, and initiatives by
portfolio Aligninghas
objectives digitalization
the greatest projects or programs
potential to add withvaluestrategic
to an
employing
management effective
techniquesproject, program,their
to improve and operational
portfolio objectives
organizationhas[5,6].
the Although,
greatest potential
average toorganizations
add value report
to an
management techniques to However,
efficiency and responsiveness. improve while
their 88operational
percent of organization
that three of [5,6]. Although,
five projects and average
programsorganizations
are not aligned report
to
efficiency and responsiveness. However, while 88 percent of that three of five projects and programs are not aligned to
2351-9789 © 2020 The Authors. Published by Elsevier B.V.
2351-9789 © 2020 The Authors. Published by Elsevier B.V.
This is an©open
2351-9789 2020
This is an open
access
The
access
article
Authors.
article
under by
Published
under
theElsevier
the CC BY-NC-ND
CC BY-NC-ND
B.V. license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review
This is an open
Peer-review under
access
under responsibility
article under
responsibility the of
of the CCthe scientific
BY-NC-ND
scientific committee
license
committee of the International
Conference on Conference onSmart
Industry
(http://creativecommons.org/licenses/by-nc-nd/4.0/)
of the International Industry 4.0 and 4.0 and Smart Manufacturing.
Manufacturing.
10.1016/j.promfg.2020.02.067
Peer-review under responsibility of the scientific committee of the International Conference on Industry 4.0 and Smart Manufacturing.
174 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180
2 Author name / Procedia Manufacturing 00 (2019) 000–000

strategy, those that are aligned to an organization’s strategy management of either a portfolio containing several projects,
are successfully completed (71%) more frequently than or a specific program encompassing a set of projects, entails
projects that are misaligned (48%) [3]. Problems caused by an understanding of the mission of the organization and
misalignment include: confusion; waste of time, money and decision-making that will realize benefits that are of strategic
opportunity; diminished productivity; de-motivation of importance [17,18]. Since the pursuit of programs and the
individuals and teams; internal conflicts and power struggles continuation of projects require expenditure of finite
and ultimately project failure [7]. In a shipyard, resources that organizational resources, the identification of digitalization
have been initially allocated to specific shipbuilding projects projects that are most aligned with strategic priorities is
may be pulled away to temporarily serve other projects (e.g., critical for the success of the organization [19,20].
maintenance, emergency repairs) [8]. For example, in the US Digital portfolio and project managers must necessarily
shipbuilding industry, important delays caused by a high determine the variations in the amount of support among the
number of interactions from liberal concurrent engineering projects in an effort to maximize the utility of the mix in
policies has led to an 18 year maintenance backlog [8,9]. advancing the organization towards its digital transformation
The alignment of projects and programs to strategic goals strategic goals. This prioritization process frequently involves
requires stakeholders to develop a consistent means of evaluating the relative risks, benefits, and costs among the
identifying, prioritizing, and defining metrics/outcomes and projects [21]. Maximizing benefit-cost relationships while
their aligned outputs [10]. In this alignment process, a minimizing risks are common tasks performed in the planning
program or project may be seen as a line of organized and controlling stages of project portfolio management.
activities that is intended to advance a product, practice, Project portfolio managers seek to select or support
procedure, or service towards meeting a need within the projects within the project’s portfolio that traditionally
government or marketplace, contribute to some type of balance benefits (mean) and risks (variance). Risk, however,
benefits (e.g., readiness, revenue), and contribute to the may be measured using other forms, e.g., conditional value at
sustainability of the organization. In an intensive capital risk (CVaR), Tail value at risk (TVaR), also known as tail
industry sector such as the military shipbuilding, this conditional expectation (TCE), or conditional tail expectation
organizational configuration by project/program makes sense (CTE). In this paper, the minimum variance opportunity set
given the large number of non-repetitive and complex indicates combinations of risks and benefits that yield the
activities and tasks associated to it [11]. Firms and institutions minimum variance for a particular benefit. Hence, an efficient
that use this managerial perspective, generally group projects digitalization project portfolio frontier may identify
and programs that have the potential to advance a common investments in a set of projects where it is not possible to
organizational goal into portfolios that require continuous obtain less risky project combinations relative a desired
control and evaluation [12]. Thus, an organization may have benefit level.
several concurrent programs and each program may The following sections of this paper presents a background
encompass one or more projects or programs that address the section that positions this work within the industrial context,
development of a product or capability that is understood to presents an innovative digital shipbuilding supply chain
contribute to longer-term institutional sustainability. framework; outlines a neural network analysis to examine
The shipbuilding industry that is largely characterized by digital projects using a digital shipbuilding supply chain
using a project-based approach to manufacture ships, perspective; presents a brief description of stochastic portfolio
submarines, and engage in ship repair and overhaul [9] would analysis; and introduces a framework integrating these
largely benefit from the implementation of novel approaches. This is followed by a brief analysis of a case
technological and operational means that seek to enhance its study in the development research industry illustrating the
questionable performance. Digital transformation programs application of the framework. The results stemming from the
emerge as a means that seek to convert slow-, outdated-, and analysis of theoretical scenarios are presented and discussed.
ill-defined operations into data-driven agile and at the same We then revisit and summarize the contributions of this work.
time lean operations based on the extensive use of Industry
4.0 technology [13–15]. 2. The Shipbuilding and Shipyard Supply Chain
Mostly organized with a project-based configuration as Management Context
well, resources are temporarily assigned to carry on digital
transformation assignments that seek to increase industrial In general, supply chain refers to all parties involved,
capabilities and become more efficient and largely flexible directly or indirectly in fulfilling customer requests [22]. In
while increasing process and product integrity. These the shipbuilding supply chain management context, activities
digitally-oriented initiatives are mostly based on data sharing upstream include procuring material and components from
and processing resulting in a paradigm shift in the way that numerous, complex suppliers to managing the deactivation of
organizations, such as shipbuilding, engage in decision- ships and submarines downstream once they have reached
making processes toward a data-driven approach [16]. Thus, their end-of-life. The basic elements of a shipbuilding supply
a set of digital transformation projects may be further chain management framework that seek to guide an
represented within a larger project or program portfolio, understanding of a value chain of these processes is presented
supervised by a digital transformation portfolio manager. The in Figure 1.
 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180 175
Author name / Procedia Manufacturing 00 (2019) 000–000 3

Fig. 1. Shipbuilding Supply Chain Framework

scenario analysis is performed to gain knowledge regarding
3. Digital Project Portfolio: Balancing Risk and Returns impacts of new constraints per project. This knowledge is
used to determine portfolio management measures of
A balance among risks and returns is required for each performance that include overall benefits and variances. In
combination of projects that compose the portfolio frontier. In addition, project portfolio optimization can be potentially
an industry sector such as military shipbuilding, monitoring used to minimize the overall portfolio variance while
risk is an important endeavour as delays, labour availability, maintaining risk-aversion and budget-constrained range
and large expenditures may drastically affect the readiness restrictions. This framework adequately captures the
and sustainment of the fleet. The selection of these complexities associated with determining risks and returns,
opportunity sets may be determined by the indifference curve relevant cost and benefit considerations guided by a proven
map that describes the project strategic preferences and assignment method. ANN assists by providing the critical
policies. Accordingly, an optimal risk management portfolio elements that guide the resource assignment process. This
maximizes the average benefit while minimizing variance. framework consists of three stages summarized as follows.
The risk associated with a project portfolio containing
multiple projects may be different from the risk associated 4.1. Quantifying risks and benefits of digital transformation
with an individual project. The different associated risks and projects
potential relationships among projects make project portfolio
management a complex subject. Risk in projects is commonly defined in terms of exposure
The risk-benefit relationships associated with digital to explicit factors that present a threat to attaining the
transformation projects in a shipbuilding project-based expected results. For example, the probability-weighted
institution further increases the complexity as different types impact of an event on a project usually defines risk in
of risk exert compounding effects on the already multifaceted software development projects [23]. The notion of risk,
nature of the shipbuilding process. A proficient mechanism to usually measured in dollars or time, may be characterized by
assess this compounding effect is important for assessing and R = P x I , where R represents the risk exposure that is related
prioritizing digital transformation programs. to an explicit risk issue while P is the probability the adverse
We assert that Artificial Neural Network (ANN) and event will be realized, and I is the effect of the loss if the
Monte Carlo simulation may be combined to offer a capable undesired event materializes [24]. Often digital transformation
framework that assists digital transformation portfolio development risks are managed by lists which are ranked by
managers in performing an effective portfolio evaluation. The subjective qualitative measures resulting in excessive
objective of this research is to describe this framework and its expenditure of risk management resources. Risk identification
capacity to clarify the cost-benefit and risk trade-offs is the most critical step in risk management, yet often is
stemming from modifying project investments. The poorly done [25]. As fundamental aspects of digitalization
framework uses a hybrid between ANN and Monte Carlo projects resemble those factors in software and hardware
capabilities to determine the benefits and risks per development, the uncertainties considered in this paper are
digitalization project considering the relationships that exist guided by [26]. These uncertainties include time uncertainty,
among multiple relevant projects’ attributes, e.g., time, which reflects uncertainties about the occurrence of expected
control, and information. This Monte Carlo simulation and unexpected events and the ability to react to them; control
approach generates complementary samples that assist in uncertainty that concerns with inadequate authority to make
training the network model to increase its accuracy. or influence decisions or inconsistency on processes; and
information uncertainty, which concerns with inadequate or
4. Digital Transformation Risk Evaluation Framework inaccurate information on which to base decisions. These
authors also suggest an extensive list of potential project
A framework based on ANN to analyse project portfolios development risk concerns that includes but is not limited to
is presented in this section and illustrated in Figure 2. The creeping user requirements, large size and complexity of
approach uses ANN to determine the benefits and risk projects, complex applications, unnecessary features, and lack
associated with projects within a portfolio of projects. A of user support.
 Available online at www.sciencedirect.com

ScienceDirect
Procedia Manufacturing 00 (2019) 000–000
176 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180www.elsevier.com/locate/procedia

Fig. 2. Digital Transformation Risk Evaluation Framework

Project portfolio evaluations require reasonable estimations (supervised training) is then executed with new information
of the risks and associated benefits per project. The complex about the assessed project portfolio, and thus the overall risk
and stochastic nature of project risks makes the analysis of per project can be obtained (scoring process).
investment choices intended to mitigate risks, a candidate for
ANN analysis. ANN models have been recognized in the IT 4.2. Sampling Enhancement via Monte Carlo
development context as competent tools to reasonably
estimate these values accurately [27]. Empirically generated continuous or discrete probability
ANN models employ the concept of hidden layers which distribution per risk component can be elicited from historical
are composed of a number of neurons (hidden neurons). data, subject matter expert (SME) information, or a
These models use mathematical functions to map inputs to combination of both. These probability distributions per
outputs and may be conceived as a massively parallel adaptive project risk level can be used to generate samples through a
network of simple nonlinear computing neurons, which are Monte Carlo simulation, and thus, to train the data that is used
intended to characterize and mimic some of the functionality to calculate overall risk to projects. The historical data reflects
of the human nervous system in an effort to partially capture a set composed of individual project risk factors and overall
some of its computational strengths [28]. ANN can be risk scores per project information. Additional Monte Carlo
described as a complex nonlinear characterization where the samples can be used to extend the historical and SME input
task of variable transformation, composite variable data set to an acceptable number of observations to improve
transformation, and model estimation are done simultaneously the accuracy of the ANN analysis. In this machine learning
in such a way that a specified error function is minimized process, the sample data is partitioned into three major groups
[29]. An appropriate number of hidden units depend on a that include: learning (40%), training (30%), and testing
relationship between the number of input and output units, the (30%). The Levenberg-Marquardt optimization technique (see
number of training cases, the type of hidden unit activation, [31] for details) is used to train the data.
and the training algorithm [30].
As a rule of thumb, it is well known that 30 observations 4.3. Neural Network Scoring
are required in the input layer per neuron in the hidden layer.
The ANN model cycles through an iterative process until a The knowledge created during the machine learning
termination criterion is reached. training and testing process may be described as the
In the studied context, the response model may reflect the behavioral pattern of the effects resulting from different factor
risk related to the likelihood of the occurrence of undesirable risk arrangements. Given a selection criterion, such as
events. Thus, an overall project risk factor that relates minimizing a set of error functions, the neural network model
individual risk elements per project can be defined as a target is generated. A second data source that corresponds to the risk
output. The ANN model obtained with historical data scores of the assessed projects is obtained by applying the

2351-9789 © 2020 The Authors. Published by Elsevier B.V.

This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the International Conference on Industry 4.0 and Smart Manufacturing.
 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180 177
Author name / Procedia Manufacturing 00 (2019) 000–000 5

neural network model to calculate the overall risk per project 5. Case Study
as a target layer. To ensure the best neural network model is
obtained for computing overall risk per project, different types A Research and Development (R&D) institution that
of neural architecture can be used to train the data. However, investigates and develops digital solutions in the military
workers are required to carefully assess these opportunities as domains is engaged in three types of digitalization
each one might contain unacceptable risk levels (e.g., development programs categorized by three risk levels (low,
additional cyber risks) that can jeopardize the operational medium, and high) with equal possibility as exhibited in
integrity. Table 1.

Table 1. Risk Factor Probability per Project Risk Level

Low Risk Level Medium Risk Level High Risk Level

1 2 3 4 5 1 2 3 4 5 1 2 3 4 5
Creeping user (CR) Probability .50 .25 .15 .06 .04 .10 .25 .40 .10 .15 .10 .15 .20 .35 .20
requirements
CR Risk .10 .30 .45 .50 .60 .20 .30 .45 .50 .70 .30 .35 .45 .55 .60
Large and Complex Probability .50 .20 .15 .10 .05 .10 .30 .35 .20 .05 .01 .05 .34 .40 .20
Projects (LCP)
LCP Risk .05 .30 .40 .50 .60 .20 .30 .40 .50 .60 .30 .20 .35 .50 .60
Delay (D) Probability .65 .15 .10 0.06 0.04 .05 .30 .40 .20 .05 .05 .10 .25 .40 .20
D Risk .10 .25 .40 0.45 0.5 .15 .25 .40 .55 .60 .30 .30 .40 .65 .70
Control (CON) Probability .65 .15 .10 0.07 0.03 .05 .40 .35 .15 .05 .10 .20 .20 .30 .20
CON Risk .15 .25 .35 0.50 0.60 .20 .25 .35 .50 .60 .10 .25 .30 .50 .60
Complex User Probability .65 .25 .05 0.04 0.01 .05 .30 .35 .25 .05 .10 .20 .25 .30 .15
Requirements (CUR)
CUR Risk .10 .30 .40 0.60 0.80 .20 .30 .40 .60 .80 .30 .35 .40 .60 .80
Unnecessary Features Probability .70 .15 .08 0.05 0.02 .10 .25 .35 .25 .05 .05 .15 .20 .50 .10
(UF)
UF Risk .05 .10 .15 0.20 0.80 .05 .10 .15 .20 .80 .30 .10 .15 .20 .80
Lack of User Support Probability .50 .20 .15 0.10 0.05 .10 .30 .35 .20 .05 .01 .09 .20 .45 .25
(LUS) LUS Risk .10 .40 .50 0.60 0.90 .30 .40 .50 .60 .90 .30 .40 .50 .60 .90

Unlike most project studies that consider risk probabilities 5.1. Determine benefits and risks
as a continuous distribution (e.g., beta or normal
distributions), in this paper, we consider an empirically The portfolio manager computes the overall risk associated
discrete modeled distribution for characterizing these with each project in the portfolio during a risk analysis phase.
probabilities. Thus, each individual risk probability is This is done by assessing the importance, or weight, of the
partitioned into five sub-levels (1-5). Each level considers individual risk factor for each project. ANN is used to
both the probability of occurrence and associated risk degree generate a model that can be used to relate individual risk
represented nominally and measured from 0 to 1. For each factors and produce an overall risk score for each individual
factor, the probability mass function adds up to 1. The risk project. A simple ANN with 2 hidden layers with 8 neurons
factors considered in these projects include: 1. Creeping user each and a single output is used to produce these risk scores
requirements, 2. Large and project complexity, 3. Application (8-8-1) [32]. Eight was selected as this corresponds to each
complexity, 4. Unnecessary features, and 5. Lack of user the five project risks and three uncertainties. SAS Miner 7.1 is
support [26]. In addition, the manager ponders uncertainties employed to configure, execute, and compare the ANN
within three additional dimensions that include 60% in time models.
uncertainty, 10% in control uncertainty, and 30% in The risk probability distribution presented in Table 1 per
information uncertainty. Notice that as explained before, risk project risk level is used to generate samples through a Monte
value represents a convoluted measure that characterizes the Carlo simulation, and thus, simulate the training data that is
probability that a project will fail to meet its objectives. used to evaluate overall project risks of the projects presented
Table 2 presents the financial and individual risk factors in Table 2. Notice that some analysists and researchers might
for ten ongoing projects A through J which requires use the Monte Carlo technique to generate samples and then
evaluation. The costs of each project as well as its cost as a calculate averages in determining the overall project risks.
share of the entire portfolio (estimated by the portfolio However, in this paper the Monte Carlo simulation is limited
manager to be $124.7 million) are presented in the second and only to determine samples that will feed the neural network
third columns, while individual project risk information is model as other authors has considered. The machine learning
presented in columns four to eleven. approach used in this paper seeks to employ ANN as a better
approximation of the real risk behaviour exhibited by the
portfolio.
178 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180
6 Author name / Procedia Manufacturing 00 (2019) 000–000

The historical data is based on a (disguised) data set quantify benefits and risks per project while employing
comprising individual project risk factors and overall risk scenario analysis to quantify the effects of constraints per
scores per project for 30 projects. More specifically, the project. Monte Carlo simulation is used to obtain sample data
Monte Carlo samples obtained using the probability that facilitates the execution of the neural network. The results
distribution presented in Table 2 are used to extend the from these simulations could be used to enable the use of
historical input data set from 30 project observations to 300 Portfolio Management Theory principles to organize and
observations. The sample data is further partitioned into three estimate measures of performance and the utilization of
major groups that include: learning (40%), training (30%), nonlinear programming tools to minimize the overall portfolio
and testing (30%). variance while maintaining risk-aversion and budget-
As previously discusses, the knowledge created during the constrained parameters.
training and testing process are assessed using 1. classical The digital transformation project risk evaluation
multilayer perceptron (MLP), 2. ordinary radial basis function framework offered follows a logical approach. First, it uses
with equal widths (ORBFEQ), 3. ordinary radial basis ANN to capture individual risk factors and quantify the
function with unequal widths (ORBFUN), 4. normalized overall risk of each project. Cost-benefit analysis is then
radial basis with equal heights (NRBFEH), 5. normalized employed to provide the foundation for performing a scenario
radial basis with equal volumes (NRBFEV), 6. normalized analysis. An examination of the effects of selected scenarios
radial basis with equal widths (NRBFEW), 7. normalized can subsequently be performed. This examination is required
radial basis with equal widths and heights (NRBFEQ), 8. to understand variations in the potential resources assigned
normalized radial basis with unequal widths and heights per digitalization projects. Afterward, correlation and
(NRBFUN). As mentioned, the Average Square Error from covariance matrices can be determined. These correlations
the validating stage is used as the reference for model may represent elicited perceptions that indicate how
selection. Table 3 shows the statistical results of this individual projects are related. The covariance matrix can
comparison that suggests the normalized radial basis with relate the project correlations, the returns, and the variability
equal widths and heights (NRBFEQ) architecture outperforms or risk required. The overall digitalization project portfolio
the alternative assessed ANN architectures. Thus, the returns, and the portfolio variance can be then determined.
NRBFEQ architecture is employed to determine the risk data. Returns and variances can be subsequently recalculated given
Once the ANN model using the NRBFEQ architecture is additional constraints. Thus, portfolio managers are able to
selected to execute the ANN model, the target variable which optimize the distribution of available resources while
is overall risk per project is obtained as presented below in preserving portfolio integrity.
Table 4. It is important for training analysts and managers to
perform risk evaluations as the organization moves through
Table 2. Current Project Investment and Risk Level per Risk Factor digital transformation process while achieving strategic goals.
In particular, organizations engaged in becoming data-driven
managed and operated might consistently reveal vast
Information

opportunities to digitalize their operations. However, workers

millions)

are required to carefully assess these opportunities as each one

Project

Cost ($

CON

CUR

might contain unacceptable risk levels (e.g., additional cyber

LCP

LUS
CR

UF
D

risks) that can jeopardize the operational integrity. Digital

A 15.0 .20 .30 .25 .35 .20 .10 .30 .22 project portfolio analysts and managers are required to learn
B 20.1 .30 .40 .15 .20 .30 .10 .45 .31 the type of risks that their portfolio may contain. Likewise,
C 2.30 .45 .50 .25 .35 .30 .20 .45 .38 they should be trained in the use of competent tools to elicit
D 14.0 .10 .05 .10 .25 .50 .10 .50 .25 projects risks and determine desired risk thresholds for their
firm. This enables the anticipation of actions and plans that
E 9.0 .55 .50 .70 .30 .60 .30 .55 .50
makes the organization more resilient as vulnerability might
F 12.3 .50 .05 .10 .15 .05 .05 .10 .15 increase if risks are not properly elicited. Training modules
G 7.0 .10 .05 .10 .15 .05 .15 .10 .09 might include introduction to digital supply chain to provide
H 11.0 .10 .05 .25 .15 .05 .10 .10 .08 an overview of the shipbuilding context and risk
I 16.0 .30 .30 .15 .35 .30 .15 .50 .31
quantification methodologies as they apply to specific settings
of the shipbuilding process.
J 18.0 .30 .50 .10 .15 .60 .10 .10 .32
Future research endeavours include extending this
framework to consider calculating the overall portfolio
6. Summary and Managerial Implications performance, optimizing resource allocation, and determining
acceptable risk ranges per project that constrain the final
Organizations may benefit from applying investment fractions of budget allocations as well as including the
allocation approaches traditionally employed in financial examination of portfolio actions that balance resource
markets analyses. A framework that combines the principles allocations considering risks and benefits. This might trigger
of Artificial Neural Network and Monte Carlo Simulation to project removal decisions for those projects whose acceptable
analyze project portfolios may assist portfolio managers in risk ranges violate upper or lower boundaries.
their resource allocation activities. Our approach uses ANN to
 Author name / Procedia
Rafael Diaz Manufacturing 00 Manufacturing
et al. / Procedia (2019) 000–000 42 (2020) 173–180 7
179

Table 3. Neural Network Architectures: Statistical Evaluation (AIC: Akaike’s Information Criterion; SBC: Schwarz’s Bayesian Criterion; ASE: Average Squared
Error; MAE: Maximum Absolute Error; RASE: Root Average Squared Error; SSE: Sum of Squared Errors; FPE: Final Prediction Error; MSE: Mean Squared
Error; RFPE: Root Final Prediction Error; RMSE: Root Mean Squared Error; AEF: Average Error Function; EF: Error Function)

ANN Architecture Valid: ASE AIC SBC ASE MAE RASE SSE FPE MSE RFPE RMSE AEF EF
NRBFEQ 3.33E-06 -1372.62 -1169.13 3.19E-06 .0093 .0018 .0004 1.310E-05 8.15E-06 .0036 .0029 3.19E-06 .0004
NRFEW 4.36E-06 -1366.40 -1140.61 2.94E-06 .0057 .0017 .0004 1.518E-05 9.05E-06 .0040 .0030 2.94E-06 .0004
MLP 5.25E-06 -1323.98 -1098.19 4.19E-06 .0095 .0020 .0005 2.157E-05 1.29E-05 .0047 .0036 4.19E-06 .0005
NRBFUN 1.09E-05 -1198.70 -953.40 1.06E-05 .0107 .0033 .0013 6.888E-05 3.97E-05 .0083 .0063 1.06E-05 .0013
ORBFUN 1.38E-05 -1217.82 -992.03 1.01E-05 .0102 .0032 .0012 5.227E-05 3.12E-05 .0072 .0056 1.01E-05 .0012
NRBFEV 2.48E-05 -1311.71 -1088.71 4.72E-06 .0063 .0022 .0006 2.379E-05 1.42E-05 .0049 .0038 4.72E-06 .0006
ORBFEQ 3.39E-05 -1212.57 -1006.30 1.19E-05 .0086 .0035 .0014 5.024E-05 3.11E-05 .0071 .0056 1.19E-05 .0014
NRBFEH 1.31E-04 -1511.94 -1288.94 8.89E-07 .0037 .0009 .0001 4.446E-06 2.67E-06 .0021 .0016 8.89E-07 0.001

Table 4. Neural Network Architectures: Statistical Evaluation 370–387. https://doi.org/10.1108/14637150510609408.

[8] U.S.G.A. Office, Naval Shipyards - Actions Needed to Improve
Project Cost Percent of Scored Poor Conditions that Affect Operations, 2017.
Project (millions) Portfolio Cost Risk
https://www.gao.gov/assets/690/687105.pdf (accessed October 11,
A 15.00 .1203 .25 2019).
B 20.10 .1612 .20 [9] United States Government Accountability Office, Navy
C 2.30 .0184 .30 Shipbuilding - Past Performance Provides Valuable Lessons for
D 14.00 .1123 .16 Future Investments, 2018.
E 9.00 .0722 .60 https://www.gao.gov/assets/700/692331.pdf (accessed October 11,
F 12.30 .0986 .12 2019).
G 7.00 .0561 .10 [10] K. Nogeste, D.H.T. Walker, Development of a method to improve
H 11.0 .0882 .19 the definition and alignment of intangible project outcomes and
I 16.0 .1283 .22 tangible project outputs, Int J Manag Proj Bus. 1 (2008) 279–287.
J 18.0 .1443 .17 https://doi.org/10.1108/17538370810866377.
[11] W.J. Stevenson, Operations Management, McGraw-Hill Education,
Acknowledgements 2003.
[12] S.K. Sebastian Nokes, The definitive guide to project management:
This research has been partially supported by the Digital the fast track to getting the job done on time and on budget, Pearson
Shipbuilding Initiative - GoVirginia.org. Education, 2003.
[13] C. Matt, T. Hess, A. Benlian, Digital Transformation Strategies,
References Bus Inf Syst Eng. 57 (2015) 339–343.
https://doi.org/10.1007/s12599-015-0401-5.
[1] Economist Intelligence Unit, Why Good Strategies Fail: Lessons [14] M. Rüßmann, M. Lorenz, P. Gerbert, M.Waldner, J. Justus, P.
for the C-Suite, 2013. https://www.pmi.org/- Engel, M. Harnisch., Industry 4.0: the Future of Productivity and
/media/pmi/documents/public/pdf/learning/thought-leadership/why- Growth in Manufacturing Industries., 2015.
good-strategies-fail-report.pdf (accessed April 27, 2019). [15] A. Ustundag, E. Cevikcan, Industry 4.0: Managing The Digital
[2] S.J. Andriole, Five myths about digital transformation, MIT Sloan Transformation, 2018. https://doi.org/10.1007/978-3-319-57870-5.
Manag Rev. 58 (2017). [16] F. Provost, T. Fawcett, Data Science and its Relationship to Big
https://doi.org/10.7551/mitpress/11633.003.0005. Data and Data-Driven Decision Making, Big Data. 1 (2013) 51–59.
[3] Project Management Institute, The high cost of low performance, https://doi.org/10.1089/big.2013.1508.
Pulse Prof. (2014). [17] Office of Government Commerce, Managing Successful
[4] I.R. Bardhan, V. V. Krishnan, S. Lin, Project performance and the Programmes, The Stationery Office, 2007.
enabling role of information Technology: An exploratory study on [18] K. Artto, M. Martinsuo, H.G. Gemünden, J. Murtoaro, Foundations
the role of alignment, Manuf Serv Oper Manag. 9 (2007) 579–595. of program management: A bibliometric view, Int J Proj Manag. 27
https://doi.org/10.1287/msom.1070.0163. (2009) 1–18. https://doi.org/10.1016/j.ijproman.2007.10.007.
[5] S. Ransbotham, D. Kiron, Analytics as a Source of Business [19] T.S. Glickman, Program portfolio selection for reducing prioritized
Innovation, MIT Sloan Manag Rev. 58 (2017). security risks, Eur J Oper Res. 190 (2008) 268–276.
https://doi.org/10.1002/ece3.1283. https://doi.org/10.1016/j.ejor.2007.06.006.
[6] S. Ransbotham, D. Kiron, P. Gerbert, M. Reeves, Reshaping [20] Project Management Institute, A guide to the project management
business with artificial intelligence., MIT Sloan Mangement Rev. body of knowledge (PMBOK® guide), 2008.
59 (2017). https://doi.org/10.3934/dcdsb.2008.10.925. [21] G. Reiss, M. Anthony, J. Chapman, G. Leigh, A. Pyne, Gower
[7] S. Box, K. Platts, Business process management: Establishing and Handbook of Programme Management, Routledge, 2006.
maintaining project alignment, Bus Process Manag J. 11 (2005)
180 Rafael Diaz et al. / Procedia Manufacturing 42 (2020) 173–180
8 Author name / Procedia Manufacturing 00 (2019) 000–000

[22] S. Chopra, P. Meindl, Supply Chain Management. Strategy, model for software effort estimation, Expert Syst Appl. 35 (2008)
Planning & Operation, Prentice Hall, New Jersey, 2010. 929–937. https://doi.org/10.1016/j.eswa.2007.08.001.
https://doi.org/10.1007/978-3-8349-9320-5_22. [28] R. Lippmann, Book Review: “Neural Networks, A Comprehensive
[23] R.N. Charette, The mechanics of managing IT risk, J Inf Technol. Foundation”, by Simon Haykin, Int J Neural Syst. 5 (1994) 363–
11 (1996) 373–378. https://doi.org/10.1057/jit.1996.10. 364.
[24] P.L. Bannerman, Risk and risk management in software projects: A [29] S.K. Sarma, Predicting modeling with SAS practical solution for
reassessment, J Syst Softw. 81 (2008) 2118–2133. business application, (2007).
https://doi.org/10.1016/j.jss.2008.03.059. [30] E.B.R.D. Stuart Geman, Neural networks and the bias/variance
[25] Y.H. Kwak, J. Stoddard, Project risk management: Lessons learned dilemma, Neural Comput. 4 (1992) 1–58.
from software development environment, Technovation. 24 (2004) [31] M.I.A. Lourakis, A Brief Description of the Levenberg-Marquardt
915–920. https://doi.org/10.1016/S0166-4972(03)00033-6. Algorithm Implemened by levmar, Found Res Technol. 4 (2005) 1–
[26] D.X. Houston, G.T. Mackulak, J.S. Collofello, Stochastic 6. https://doi.org/10.1016/j.ijinfomgt.2009.10.001.
simulation of risk factor potential effects for software development [32] J. Singh, B. Sahoo, Software Effort Estimation with Different
risk management, J Syst Softw. 59 (2001) 247–257. Artificial Neural Network, (2012).
https://doi.org/10.1016/S0164-1212(01)00066-8.
[27] H. Park, S. Baek, An empirical validation of a neural network