Computer Methods and Programs in Biomedicine 158 (2018) 113–122

Contents lists available at ScienceDirect

Computer Methods and Programs in Biomedicine

journal homepage: www.elsevier.com/locate/cmpb

NiftyNet: a deep-learning platform for medical imaging

Eli Gibson a,b,1, Wenqi Li a,1,∗, Carole Sudre b, Lucas Fidon a, Dzhoshkun I. Shakir a,
Guotai Wang a, Zach Eaton-Rosen b, Robert Gray c,d, Tom Doel a, Yipeng Hu b, Tom Whyntie b,
Parashkev Nachev c,d, Marc Modat b, Dean C. Barratt a,b, Sébastien Ourselin a,
M. Jorge Cardoso b,2, Tom Vercauteren a,2
a
Wellcome / EPSRC Centre for Interventional and Surgical Sciences (WEISS), University College London, UK
b
Centre for Medical Image Computing (CMIC), Departments of Medical Physics & Biomedical Engineering and Computer Science, University College London,
UK
c
Institute of Neurology, University College London, UK
d
National Hospital for Neurology and Neurosurgery, London, UK

a r t i c l e i n f o a b s t r a c t

Article history: Background and objectives: Medical image analysis and computer-assisted intervention problems are in-
Received 2 October 2017 creasingly being addressed with deep-learning-based solutions. Established deep-learning platforms are
Revised 8 January 2018
flexible but do not provide specific functionality for medical image analysis and adapting them for this
Accepted 24 January 2018
domain of application requires substantial implementation effort. Consequently, there has been substan-
tial duplication of effort and incompatible infrastructure developed across many research groups. This
Keywords: work presents the open-source NiftyNet platform for deep learning in medical imaging. The ambition
Medical image analysis of NiftyNet is to accelerate and simplify the development of these solutions, and to provide a common
Deep learning mechanism for disseminating research outputs for the community to use, adapt and build upon.
Convolutional neural network
Segmentation Methods: The NiftyNet infrastructure provides a modular deep-learning pipeline for a range of medical
Image regression imaging applications including segmentation, regression, image generation and representation learning
Generative adversarial network applications. Components of the NiftyNet pipeline including data loading, data augmentation, network
architectures, loss functions and evaluation metrics are tailored to, and take advantage of, the idiosyn-
cracies of medical image analysis and computer-assisted intervention. NiftyNet is built on the TensorFlow
framework and supports features such as TensorBoard visualization of 2D and 3D images and computa-
tional graphs by default.
Results: We present three illustrative medical image analysis applications built using NiftyNet infrastruc-
ture: (1) segmentation of multiple abdominal organs from computed tomography; (2) image regression
to predict computed tomography attenuation maps from brain magnetic resonance images; and (3) gen-
eration of simulated ultrasound images for specified anatomical poses.
Conclusions: The NiftyNet infrastructure enables researchers to rapidly develop and distribute deep
learning solutions for segmentation, regression, image generation and representation learning applica-
tions, or extend the platform to new applications.
© 2018 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY license. (http://creativecommons.org/licenses/by/4.0/)

1. Introduction ing and diagnosis to treatment delivery and monitoring. This role
is poised to grow as analysis methods become more accurate and
Computer-aided analysis of medical images plays a critical role cost effective. In recent years, a key driver of such improvements
at many stages of the clinical workflow from population screen- has been the adoption of deep learning and convolutional neural
networks in many medical image analysis and computer-assisted
intervention tasks.
∗
Corresponding author. Deep learning refers to a deeply nested composition of many
E-mail address: [email protected] (W. Li). simple functions (principally linear combinations such as convolu-
1
Wenqi Li and Eli Gibson contributed equally to this work.
2
tions, scalar non-linearities and moment normalizations) parame-
M. Jorge Cardoso and Tom Vercauteren contributed equally to this work.

https://doi.org/10.1016/j.cmpb.2018.01.025
0169-2607/© 2018 The Authors. Published by Elsevier B.V. This is an open access article under the CC BY license. (http://creativecommons.org/licenses/by/4.0/)
114 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122

terized by variables. The particular composition of functions, called modeling (e.g. [29]) are available for use as part of image analysis
the architecture, defines a parametric function (typically with pipelines. Pipelines for specific research applications such as FSL
millions of parameters) that can be optimized to minimize an ob- [52] for functional MRI analysis and Freesurfer [14,19] for structural
jective, or ‘loss’, function, usually using some form of gradient de- neuroimaging have reached widespread use. More general toolkits
scent. offering standardized implementations of algorithms (VTK and ITK
Although the first use of neural networks for medical im- [44]) and application frameworks (NifTK [12], MITK [43] and 3D
age analysis dates back more than twenty years [35], their usage Slicer [44]) enable others to build their own pipelines. Common
has increased by orders of magnitude in the last five years. Re- software infrastructure has supported and accelerated medical im-
cent reviews [34,51] have highlighted that deep learning has been age analysis and computer-assisted intervention research across
applied to a wide range of medical image analysis tasks (segmen- hundreds of research groups. However, despite the wide availabil-
tation, classification, detection, registration, image reconstruction, ity of general purpose deep learning software tools, deep learning
enhancement, etc.) across a wide range of anatomical sites (brain, technology has limited support in current software infrastructure
heart, lung, abdomen, breast, prostate, musculature, etc.). Although for medical image analysis and computer-assisted intervention.
each of these applications have their own specificities, there is sub- Software infrastructure for general purpose deep learning is a
stantial overlap in software pipelines implemented by many re- recent development. Due to the high computational demands of
search groups. training deep learning models and the complexity of efficiently us-
Deep-learning pipelines for medical image analysis comprise ing modern hardware resources (general purpose graphics process-
many interconnected components. Many of these are common to ing units and distributed computing, in particular), numerous deep
all deep-learning pipelines: learning libraries and platforms have been developed and widely
adopted, including cuDNN [9], TensorFlow [1], Theano [4], Caffe
• separation of data into training, testing and validation sets; [28], Torch [13], CNTK [50], and MatConvNet [54].
• randomized sampling during training; These platforms facilitate the definition of complex deep learn-
• image data loading and sampling; ing networks as compositions of simple functions, hide the com-
• data augmentation; plexities of differentiating the objective function with respect to
• a network architecture defined as the composition of many trainable parameters during training, and execute efficient imple-
simple functions; mentations of performance-critical functions during training and
• a fast computational framework for optimization and inference; inference. These frameworks have been optimized for performance
• metrics for evaluating performance during training and infer- and flexibility, and using them directly can be challenging, inspir-
ence. ing the development of platforms that simplify the development
In medical image analysis, many of these components have do- process for common usage scenarios, such as Keras [10], and Ten-
main specific idiosyncrasies, detailed in Section 4. For example, sorLayer [17] for TensorFlow and Lasagne [15] for Theano. However,
medical images are typically stored in specialized formats that by avoiding assumptions about the application to remain general,
handle large 3D images with anisotropic voxels and encode addi- the platforms are unable to provide specific functionality for med-
tional spatial information and/or patient information, requiring dif- ical image analysis and adapting them for this domain of applica-
ferent data loading pipelines. Processing large volumetric images tion requires substantial implementation effort.
has high memory requirements and motivates domain-specific Developed concurrently with the NiftyNet platform, the Deep
memory-efficient networks or custom data sampling strategies. Im- Learning Toolkit4 aims to support fast prototyping and repro-
ages are often acquired in standard anatomical views and can ducibility by implementing deep learning methods and modules
represent physical properties quantitatively, motivating domain- for medical image analysis. While still in preliminary development,
specific data augmentation and model priors. Additionally, the clin- it appears to focus on deep learning building blocks rather than
ical implications of certain errors may warrant custom evaluation analysis pipelines. NifTK [12,24] and Slicer3D (via the DeepInfer
metrics. Independent reimplementation of all of this custom infras- [38] plugin) provide infrastructure for distribution of trained deep
tructure results in substantial duplication of effort, poses a barrier learning pipelines. Although this does not address the substantial
to dissemination of research tools and inhibits fair comparisons be- infrastructure needed for training deep learning pipelines, integra-
tween competing methods. tion with existing medical image analysis infrastructure and modu-
This work presents the open-source NiftyNet3 platform to 1) lar design makes these platforms promising routes for distributing
facilitate efficient deep learning research in medical image anal- deep-learning pipelines.
ysis and computer-assisted intervention; and 2) reduce duplication
of effort. The NiftyNet platform comprises an implementation of 3. Typical deep learning pipeline
the common infrastructure and common networks used in medi-
cal imaging, a database of pre-trained networks for specific appli- Deep learning adopts the typical machine learning pipeline con-
cations and tools to facilitate the adaptation of deep learning re- sisting of three phases: model selection (picking and fitting a
search to new clinical applications with a shallow learning curve. model on training data), model evaluation (measuring the model
performance on testing data), and model distribution (sharing the
2. Background model for use on a wider population). Within these simple phases
lies substantial complexity, illustrated in Fig. 1. The most obvious
The development of common software infrastructure for med- complexity is in implementing the network being studied. Deep
ical image analysis and computer-assisted intervention has a long neural networks generally use simple functions, but compose them
history. Early efforts included the development of medical imag- in complex hierarchies; researchers must implement the network
ing file formats (e.g. ACR-NEMA (1985), Analyze 7.5 (1986), DICOM being tested, as well as previous networks (often incompletely
(1992) MINC (1992), and NIfTI (2001)). Toolsets to solve common specified) for comparison. To train, evaluate and distribute these
challenges such as registration (e.g. NiftyReg [42], ANTs [2] and networks, however, requires further infrastructure. Data sets must
elastix [31]), segmentation (e.g. NiftySeg [8]), and biomechanical be correctly partitioned to avoid biassed evaluations, sometimes

3 4
Available at http://niftynet.io. https://dltk.github.io.
 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122 115

4.1. Data availability

Acquiring, annotating and distributing medical image data sets

have higher costs than in many computer vision tasks. For many
medical imaging modalities, generating an image is costly. Anno-
tating images for many applications requires high levels of exper-
tise from clinicians with limited time. Additionally, due to privacy
concerns, sharing data sets between institutions, let alone inter-
nationally, is logistically and legally challenging. Although recent
tools such as DeepIGeoS [56] for semi-automated annotation and
GIFT-Cloud [16] for data sharing are beginning to reduce these bar-
riers, typical data sets remain small. Using smaller data sets in-
creases the importance of data augmentation, regularization, and
cross-validation to prevent over-fitting. The additional cost of data
set annotation also places a greater emphasis on semi- and unsu-
pervised learning.

4.2. Data dimensionality and size

Data dimensionality encountered in medical image analysis and

computer-assisted intervention typically ranges from 2D to 5D.
Many medical images, including MRI, CT, PET and SPECT, capture
volumetric images. Longitudinal imaging (multiple images taken
over time) is typical in interventional settings as well as clinically
useful for measuring organ function (e.g. blood ejection fraction in
cardiac imaging) and disease progression (e.g. cortical thinning in
neurodegenerative diseases).
At the same time, capturing high-resolution data in multi-
ple dimensions is often necessary to detect small but clinically
important anatomy and pathology. The combination of these fac-
tors results in large data sizes for each sample, which impact com-
putational and memory costs. Deep learning in medical imaging
uses various strategies to account for this challenge. Many net-
works are designed to use partial images: 2D slices sampled along
one axis from 3D images [57], 3D subvolumes [33], anisotropic
convolution [55], or combinations of subvolumes along multiple
axes [48]. Other networks use multi-scale representations allowing
Fig. 1. Data flow implemented in typical deep learning projects. Boxes represent deeper and wider networks on lower-resolution representations
the software infrastructure to be developed and arrows represent the data flow. [30,40]. A third approach uses dense networks to reuse feature
representations multiple times in the network [23]. Smaller batch
sizes can reduce the memory cost, but rely on different weight
considering data correlations (e.g. images acquired at the same normalization functions such as batch renormalization [27], weight
hospital may be more similar to each other than to those from normalization [49] or layer normalization [3].
other hospitals). The data must be sampled, loaded and passed
to the network, in different ways depending on the phase of the
4.3. Data formatting
pipeline. Algorithms for tuning hyper-parameters within a family
of models and optimizing model parameters on the training data
Data sets in medical imaging are typically stored in differ-
are needed. Logging and visualization are needed to debug and dis-
ent formats than in many computer vision tasks. To support the
sect models during and after training. In applications with limited
higher-dimensional medical image data, specialized formats have
data, data sets must be augmented by perturbing the training data
been adopted (e.g. DICOM, NIfTI, Analyze). These formats fre-
in realistic ways to prevent over-fitting. In deep learning, it is com-
quently also store metadata that is critical to image interpre-
mon practice to adapt previous network architectures, trained or
tation, including spatial information (anatomical orientation and
untrained, in part or in full for similar or different tasks; this re-
voxel anisotropy), patient information (demographics and identi-
quires a community repository (popularly called a model zoo) stor-
fiers), and acquisition information (modality types and scanner
ing models and parameters in an adaptable format. Much of this
parameters). These medical imaging specific data formats are typ-
infrastructure is recreated by each researcher or research group un-
ically not supported by existing deep learning frameworks, requir-
dertaking a deep learning project, and much of it depends on the
ing custom infrastructure for loading images.
application domain being addressed.

4.4. Data properties

4. Design considerations for deep learning in medical imaging
The characteristic properties of medical image content pose
Medical image analysis differs from other domains where deep opportunities and challenges. Medical images are obtained under
learning is applied due to characteristics of the data itself, and the controlled conditions, allowing more predictable data distributions.
applications in which they are used. In this section, we present the In many modalities, images are calibrated such that spatial re-
domain-specific requirements driving the design of NiftyNet. lationships and image intensities map directly to physical quan-
116 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122

tities and are inherently normalized across subjects. For a given

clinical workflow, image content is typically consistent, potentially
enabling the characterization of plausible intensity and spatial vari-
ation for data augmentation. However, some clinical applications
introduce additional challenges. Because small image features can
have large clinical importance, and because some pathology is very
rare but life-threatening, medical image analysis must deal with
large class imbalances, motivating special loss functions [18,40,53].
Furthermore, different types of error may have very different clin-
ical impacts, motivating specialized loss functions and evaluation
metrics (e.g. spatially weighted segmentation metrics). Applica-
tions in computer-assisted intervention where analysis results are
used in real time (e.g. [21,24]) have additional constraints on anal-
ysis latency.

5. NiftyNet: a platform for deep learning in medical imaging

The NiftyNet platform aims to augment the current deep learn-

ing infrastructure to address the ideosyncracies of medical imag- Fig. 2. A brief overview of NiftyNet components.
ing described in Section 4, and lower the barrier to adopting this
technology in medical imaging applications. NiftyNet is built using
the TensorFlow library, which provides the tools for defining com-
Network includes sub-components representing individual net-
work blocks or larger conceptual units. These components are
putational pipelines and executing them efficiently on hardware
briefly depicted in Fig. 2 and detailed in the following sections.
resources, but does not provide any specific functionality for pro-
As a concrete illustration, one instantiation of the
cessing medical images, or high-level interfaces for common med-
ical image analysis tasks. NiftyNet provides a high-level deep
SegmentationApplication could use the following modules.
During training, it could use a UniformSampler to generate
learning pipeline with components optimized for medical imaging
small image patches and corresponding labels; a vnet Network
applications (data loading, sampling and augmentation, networks,
would process batches of images to generate segmentations; a
loss functions, evaluations, and a model zoo) and specific interfaces
for medical image segmentation, classification, regression, image
Dice LossFunction would compute the loss used for backprop-
agation using the Adam Optimizer. During inference, it could
generation and representation learning applications.
use a GridSampler to generate a set of non-overlapping patches
to cover the image to segment, the same network to generate
5.1. Design goals corresponding segmentations, and a GridSamplesAggregator
to aggregate the patches into a final segmentation.
The design of NiftyNet follows several core principles which
support a set of key requirements: 5.3. Component details: TensorFlow framework
• support a wide variety of application types in medical image
The TensorFlow framework defines the interface for and ex-
analysis and computer-assisted intervention;
ecutes the high performance computations used in deep learn-
• enable research in one aspect of the deep learning pipeline
ing. Briefly, TensorFlow provides a Python application program-
without the need for recreating the other parts;
ming interface to construct an abstract computation graph
• be simple to use for common use cases, but flexible enough for
comprising composable operations with support for automatic dif-
complex use cases;
ferentiation. The choice of the TensorFlow framework over the
• support built-in TensorFlow features (parallel processing, visu-
many deep learning frameworks described above reflects both en-
alization) by default;
gineering concerns – including cross-platform support, multi-GPU
• support best practices (data augmentation, data set separation)
support, built-in visualization tools, installation without compila-
by default;
tion, and semantic versioning – as well as pragmatic concerns,
• support model distribution and adaptation.
such as its larger number of users and support from industry.

5.2. System overview 5.4. Component details: ApplicationDriver class

The NiftyNet platform comprises several modular components. The NiftyNet ApplicationDriver defines the common
The TensorFlow framework defines the interface for and executes structure for all NiftyNet pipelines. It is responsible for instan-
the high performance computations used in deep learning. The tiating the data and Application objects and distributing the
NiftyNet ApplicationDriver defines the common structure workload across and recombining results from the computational
across all applications, and is responsible for instantiating the data resources (potentially including multiple CPUs and GPUs). It is
analysis pipeline and distributing the computation across the avail- also responsible for handling variable initialization, variable sav-
able computational resources. The NiftyNet Application classes ing and restoring, and logging. Implemented as a template design
encapsulate standard analysis pipelines for different medical image pattern [20], the ApplicationDriver delegates application-
analysis applications, by connecting four components: a Reader specific functionality to separate Application classes.
to load data from files, a Sampler to generate appropriate sam- The ApplicationDriver can be configured from the com-
ples for processing, a Network to process the inputs, and an out- mand line or programmatically using a human-readable con-
put handler (comprising the Loss and Optimizer during train- figuration file. This file contains the data set definitions and
ing and an Aggregator during inference and evaluation). The all the settings that deviate from the defaults. When the
Sampler includes sub-components for data augmentation. The ApplicationDriver saves its progress, the full configuration
 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122 117

(including default parameters) is also saved so that the analysis

pipeline can be recreated to continue training or carry out infer-
ence internally or with a distributed model.

5.5. Component details: application class

Medical image analysis encompasses a wide range of tasks for

different parts of the pre-clinical and clinical workflow: segmenta-
tion, classification, detection, registration, reconstruction, enhance-
ment, model representation and generation. Different applications
use different types of inputs and outputs, different networks, and
different evaluation metrics; however, there is common structure
and functionality among these applications supported by NiftyNet.
NiftyNet currently supports
• image segmentation,
• image regression,
• image model representation (via auto-encoder applications),
and
• image generation (via auto-encoder and generative adversarial
networks (GANs)),
and it is designed in a modular way to support the addition of new
application types, by encapsulating typical application workflows
in Application classes.
The Application class defines the required data interface for
the Network and Loss, facilitates the instantiation of appropriate
Sampler and output handler objects, connects them as needed for
the application, and specifies the training regimen. For example,
the SegmentationApplication specifies that networks accept
images (or patches thereof) and generate corresponding labels,
that losses accept generated and reference segmentations and an
optional weight map, and that the optimizer trains all trainable
variables in each iteration. In contrast, the GANApplication
specifies that networks accept a noise source, samples of real data
and an optional conditioning image, losses accept logits denoting
if a sample is real or generated, and the optimizer alternates be-
tween training the discriminator sub-network and the generator
sub-network.

5.6. Component details: networks and layers

The complex composition of simple functions that comprise a

deep learning architecture is simplified in typical networks by the
repeated reuse of conceptual blocks. In NiftyNet, these concep-
tual blocks are represented by encapsulated Layer classes, or in-
Fig. 3. TensorBoard visualization of a NiftyNet generative adversarial network. Ten-
line using TensorFlow’s scoping system. Composite layers, and even
sorBoard interactively shows the composition of conceptual blocks (rounded rectan-
entire networks, can be constructed as simple compositions of gles) and their interconnections (grey lines) and color-codes similar blocks. Above,
NiftyNet layers and TensorFlow operations. This supports the reuse the generator and discriminator blocks and one of the discriminator’s residual
of existing networks by clearly demarcating conceptual blocks of blocks are expanded. Font and block sizes were edited for readability.
code that can be reused and assigning names to corresponding
sets of variables that can be reused in other networks (detailed in
Section 5.11). This also enables automatic support for visualization cross-validation studies). Input and output of medical file formats
of the network graph as a hierarchy at different levels of detail are already supported in multiple existing Python libraries, al-
using the TensorBoard visualizer [37] as shown in Fig. 3. Follow- though each library supports different sets of formats. To facili-
ing the model used in Sonnet [46], Layer objects define a scope tate a wide range of formats, NiftyNet uses nibabel [6] as a core
upon instantiation, which can be reused repeatedly to allow com- dependency but can fall back on other libraries (e.g. SimpleITK
plex weight-sharing without breaking encapsulation. [36] if they are installed and a file format is not supported by
nibabel. A pipeline of image-wide preprocessing functions, de-
5.7. Component details: data loading scribed in Section 5.9, is applied to each image before samples are
taken.
The Reader class is responsible for loading corresponding im-
age files from medical file formats for a specified data set, and 5.8. Component details: samplers and output handlers
applying image-wide preprocessing. For simple use cases, NiftyNet
can automatically identify corresponding images in a data set by To handle the breadth of applications in medical image analy-
searching a specified file path and matching user-specified pat- sis and computer-assisted intervention, NiftyNet provides flexibil-
terns in file names, but it also allows explicitly tabulated comma- ity in mapping from an input data set into packets of data to be
separated value files for more complex data set structures (e.g. processed and from the processed data into useful outputs. The
118 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122

former is encapsulated in Sampler classes, and the latter is en- voxel-wise sensitivity, specificity and accuracy), boundary distances
capsulated in output handlers. Because the sampling and output (e.g. mean absolute distance and Hausdorff distances) and region-
handling are tightly coupled and depend on the action being per- wise errors (e.g. detection rate; region-wise sensitivity, specificity
formed (i.e. training, inference or evaluation), the instantiation of and accuracy).
matching Sampler objects and output handlers is delegated to
the Application class.
5.11. Component details: model zoo for network reusability
Sampler objects generate a sequence of packets of corre-
sponding data for processing. Each packet contains all the data
To support the reuse of network architectures and trained mod-
for one independent computation (e.g. one step of gradient de-
els, many deep learning platforms host a database of existing
scent during training), including images, labels, classifications,
trained and untrained networks in a standardized format, called
noise samples or other data needed for processing. During train-
a model zoo. Trained networks can be used directly (as part of a
ing, samples are taken randomly from the training data, while dur-
workflow or for performance comparisons), fine-tuned for differ-
ing inference and evaluation the samples are taken systematically
ent data distributions (e.g. a different hospital’s images), or used
to process the whole data set. To feed these samples to TensorFlow,
to initialize networks for other applications (i.e. transfer learning).
NiftyNet automatically takes advantage of TensorFlow’s data queue
Untrained networks or conceptual blocks can be used within new
support: data can be loaded and sampled in multiple CPU threads,
networks. NiftyNet provides several mechanisms to support the
combined into mini-batches and consumed by one or more GPUs.
distribution and reuse of networks and conceptual blocks.
NiftyNet includes Sampler classes for sampling image patches
Trained NiftyNet networks can be restored directly us-
(uniformly or based on specified criteria), sampling whole images
ing configuration options. Trained networks developed out-
rescaled to a fixed size and sampling noise; and it supports com-
side of NiftyNet can be adapted to NiftyNet by encapsu-
posing multiple Sampler objects for more complex inputs.
lating the network within a Network class derived from
Output handlers take different forms during training and infer-
TrainableLayer. Externally trained weights can be loaded
ence. During training, the output handler takes the network out-
within NiftyNet using a restore_initializer, adapted from
put, computes a loss and the gradient of the loss with respect to
Sonnet [46], for the complete network or individual concep-
the trainable variables, and uses an Optimizer to iteratively train
tual blocks. restore_initializer initializes the network
the model. During inference, the output handler generates useful
weights with those stored in a specified checkpoint, and supports
outputs by aggregating one or more network outputs and perform-
variable_scope renaming for checkpoints with incompatible
ing any necessary postprocessing (e.g. resizing the outputs to the
scope names. Smaller conceptual blocks, encapsulated in Layer
original image size). NiftyNet currently supports Aggregator ob-
classes, can be reused in the same way. Trained networks incor-
jects for combining image patches, resizing images, and computing
porating previous networks are saved in a self-contained form to
evaluation metrics.
minimize dependencies.
The NiftyNet model zoo contains both untrained networks (e.g.
5.9. Component details: data normalization and augmentation
unet [11] and vnet [40] for segmentation), as well as trained net-
works for some tasks (e.g. dense_vnet [22] for multi-organ ab-
Data normalization and augmentation are two approaches to
dominal CT segmentation, wnet [55] for brain tumor segmenta-
compensating for small training data sets in medical image anal-
tion and simulator_gan [26] for generating ultrasound images).
ysis, wherein the training data set is too sparse to represent the
Model zoo entries should follow a standard format comprising:
variability in the distribution of images. Data normalization re-
duces the variability in the data set by transforming inputs to have • Python source code defining any components not included in
specified invariant properties, such as fixed intensity histograms or NiftyNet (e.g. external Network classes, Loss functions);
moments (mean and variance). Data augmentation artificially in- • an example configuration file defining the default settings and
creases the variability of the training data set by introducing ran- the data ordering;
dom perturbations during training, for example applying random • documentation describing the network and assumptions on
spatial transformations or adding random image noise. In NiftyNet, the input data (e.g. dimensionality, shape constraints, intensity
data augmentation and normalization are implemented as Layer statistic assumptions).
classes applied in the Sampler, as plausible data transformations
will vary between applications. Some of these layers, such as his- For trained networks, it should also include:
togram normalization, are data dependent; these layers compute
parameters over the data set before training begins. NiftyNet cur- • a Tensorflow checkpoint containing the trained weights;
rently supports mean, variance and histogram intensity data nor- • documentation describing the data used to train the network
malization, and flip, rotation and scaling spatial data augmentation. and on which the trained network is expected to perform ade-
quately.
5.10. Component details: data evaluation
5.12. Platform processes
Summarizing and comparing the performance of image analy-
sis pipelines typically rely on standardized descriptive metrics and In addition to the implementation of common functionality,
error metrics as surrogates for performance. Because individual NiftyNet development has adopted good software development
metrics are sensitive to different aspects of performance, multiple processes to support the ease-of-use, robustness and longevity of
metrics are reported together. Reference implementations of these the platform as well as the creation of a vibrant community. The
metrics reduce the burden of implementation and prevent imple- platform supports easy installation via the pip installation tool5
mentation inconsistencies. NiftyNet currently supports the calcula- (i.e. pip install niftynet) and provides analysis pipelines
tion of descriptive and error metrics for segmentation. Descriptive that can be run as part of the command line interface. Examples
statistics include spatial metrics (e.g. volume, surface/volume ratio, demonstrating the platform in multiple use cases are included to
compactness) and intensity metrics (e.g. mean, quartiles, skewness
of intensity). Error metrics, computed with respect to a reference
segmentation, include overlap metrics (e.g. Dice and Jaccard scores; 5
https://pip.pypa.io.
 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122 119

Table 1
Median segmentation metrics for 8 organs aggregated over the 9-fold cross-
validation.

95th
Mean percentile
Relative absolute Hausdorff
Dice volume distance distance
score difference (voxels) (voxels)

Spleen 0.94 0.03 1.07 2.00

L. Kidney 0.93 0.04 1.06 3.00
Gallbladder 0.79 0.17 1.55 4.41
Esophagus 0.68 0.57 2.05 6.00
Liver 0.95 0.02 1.42 4.12
Stomach 0.87 0.09 2.06 8.88
Pancreas 0.75 0.19 1.93 7.62
Duodenum 0.62 0.24 3.05 12.47

reduce the learning curve. The NiftyNet repository uses contin-

uous integration incorporating system and unit tests for regres-
sion testing. To mitigate issues due to library version compati-
bility, NiftyNet releases will follow two policies: (1) the range of
compatible versions of NiftyNet dependencies will be encoded in
a requirements.txt file in the code repository enabling auto-
matic installation of compatible libraries for any NiftyNet version,
and (2) NiftyNet versions will follow the semantic versioning 2.0
standard [45] to ensure clear communication regarding backwards Fig. 4. Reference standard (left) and NiftyNet (right) multi-organ abdominal CT seg-
mentation for the subject with Dice scores closest to the median. Each segmenta-
compatibility.
tion is shown with a surface rendering view from the posterior direction and with
organ labels overlaid on a transverse CT slice.

6. Results: illustrative applications

6.1. Abdominal organ segmentation The pre-NiftyNet implementation used TensorFlow directly for
deep learning and used custom MATLAB code and third-party
Segmentations of anatomy and pathology on medical images MATLAB libraries for converting data from medical image formats,
can support image-guided interventional workflows by enabling pre-/post-processing and evaluating the inferred segmentations. In
the visualization of hidden anatomy and pathology during surgi- addition to python code implementing the novel aspects of the
cal navigation. Here we present an example, based on a simplified work (e.g. a new memory-efficient dropout implementation and a
version of [22], that illustrates the use of NiftyNet to train a Dense new network architecture), additional infrastructure was developed
V-network to segment organs on abdominal CT that are important to load data, separate the data for cross-validation, sample train-
to pancreatobiliary interventions: the gastrointestinal tract (esoph- ing and validation data, resample images for data augmentation,
agus, stomach and duodenum), the pancreas, and anatomical land- organise model snapshots, log intermediate losses on training and
mark organs (liver, left kidney, spleen and stomach). validation sets, coordinate each experiment, and compute inferred
The data used to train the network comprised 90 abdominal CT segmentations on the test set. The pre-NiftyNet implementation
with manual segmentations from two publicly available data sets was not conducive to distributing the code or the trained net-
[32,47], with additional manual segmentations performed at our work, and lacked visualizations for monitoring segmentation per-
centre. formance during training.
The network was trained and evaluated in a 9-fold cross- In contrast, the NiftyNet implementation was entirely Python-
validation, using the network implementation available in NiftyNet. based and required implementations of custom network, data
Briefly, the network, available as dense_vnet in NiftyNet, uses augmentation and loss functions specific to the new architecture,
a V-shaped structure (with downsampling, upsampling and skip including four conceptual blocks to improve code readability. The
connections) where each downsampling stage is a dense feature network was trained using images in their original NIfTI medical
stack (i.e. a sequence of convolution blocks where the inputs are image format and the resulting trained model was publicly de-
concatenated features from all preceding convolution blocks), up- ployed in the NiftyNet model zoo. Furthermore, now that the Den-
sampling is bilinear upsampling and skip connections are convolu- seVNet architecture is incorporated into NiftyNet, the network and
tions. The loss is a modified Dice loss (with additional hinge losses its conceptual blocks can be used in new segmentation problems
to mitigate class imbalance) implemented external to NiftyNet with no code development using the command line interface.
and included via a reference in the configuration file. The net-
work was trained for 30 0 0 iterations on whole images (using the
ResizeSampler) with random affine spatial augmentations. 6.2. Image regression
Segmentation metrics, computed using NiftyNet’s evaluation
action, and aggregated over all folds, are given in Table 1. The Image regression, more specifically, the ability to predict the
segmentation with Dice scores closest to the median is shown in content of an image given a different imaging modality of the same
Fig. 4. object, is of paramount importance in real-world clinical work-
Because this network was initially developed prior to NiftyNet flows. Image reconstruction and quantitative image analysis algo-
and later re-developed for inclusion in NiftyNet, a comparison of rithms commonly require a minimal set of inputs that are often
the two implementations illustrates the relative advantages of de- not be available for every patient due to the presence of imag-
veloping with NiftyNet. ing artefacts, limitations in patient workflow (e.g. long acquisition
120 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122

Table 2 trained ultrasound simulation network that was ported to NiftyNet

The Mean Absolute Error (MAE) and the Mean Er-
for inclusion in the NiftyNet model zoo.
ror (ME) between the ground truth and the pseu-
doCT in Hounsfield units, comparing the NiftyNet The network was originally trained outside of the NiftyNet plat-
method with pCT [7] and the UTE-based method form as described in [26]. Briefly, a conditional GAN network was
of the Siemens Biograph mMR. trained to generate ultrasound images of specified views of a fetal
NiftyNet pCT UTE phantom using 26,0 0 0 frames of optically tracked ultrasound. An
image can be sampled from the generative model based on a con-
MAE Average 88 121 203
ditioning image (denoting the pixel coordinates in 3D space) and a
S.D 7.5 17 24
ME Average 9.1 −7.3 −132 model parameter (sampled from a 100-D Gaussian distribution).
S.D. 12 23 34 The network was ported to NiftyNet for inclusion in the model
zoo. The network weights were transferred to the NiftyNet net-
work using NiftyNet’s restore_initializer, adapted from
Sonnet [46], which enables trained variables to be loaded from
networks with different architectures or naming schemes.
The network was evaluated multiple times using the
linear_interpolation inference in NiftyNet, wherein
samples are taken from the generative model based on one
conditioning image and a sequence of model parameters evenly
interpolated between two random samples. Two illustrative re-
sults are shown in Fig. 6. The first shows the same anatomy,
but a smooth transition between different levels of ultrasound
shadowing artifacts. The second shows a sharp transition in the
interpolation, suggesting the presence of mode collapse, a common
issue in GANs [25].

7. Discussion

7.1. Lessons learned

Fig. 5. The input T1 MRI image (left), the ground truth CT (centre) and the NiftyNet NiftyNet development was guided by several core principles
regression output (right).
that impacted the implementation. Maximizing simplicity for sim-
ple use cases motivated many implementation choices. We envi-
sioned three categories of users: novice users who are comfort-
time), image harmonization, or due to ionising radiation exposure
able with running applications, but not with writing new Python
minimization.
code, intermediate users who are comfortable with writing some
An example application of image regression is the process of
code, but not with modifying the NiftyNet libraries, and advanced
generating synthetic CT images from MRI data to enable the atten-
users who are comfortable with modifying the libraries. Support
uation correction of PET-MRI images [7]. This regression problem
for pip installation simplifies NiftyNet for novice and intermedi-
has been historically solved with patch-based or multi-atlas propa-
ate users. In this context, enabling experimental manipulation of
gation methods, a class of models that are very robust but compu-
individual pipeline components for intermediate users, and down-
tationally complex and dependent on image registration. The same
loadable model zoo entries with modified components for novice
process can now be solved using the deep learning architectures
users required a modular approach with plugin support for exter-
similar to the ones used in image segmentation.
nally defined components. Accordingly, plugins for networks, loss
As a demonstration of this application, a neural network was
functions and even application logic can be specified by Python
trained and evaluated in a 5-fold cross-validation setup using the
net_regress application in NiftyNet. Briefly, the network, avail- import paths directly in configuration files without modifying the
NiftyNet library. Intermediate users can customize pipeline com-
able as highresnet in NiftyNet, uses a stack of residual dilated
ponents by writing classes or functions in Python, and can embed
convolutions with increasingly large dilation factors [33]. The root
them into model zoo entries for distribution.
mean square error was used as the loss function and implemented
Although initially motivated by simplifying variable sharing
as part of NiftyNet as rmse. The network was trained for 15,0 0 0 it-
within networks, NiftyNet’s named conceptual blocks also simpli-
erations on patches of size 80 × 80 × 80, and using the iSampler
fied the adaptation of weights from pre-trained models and the
[5] for patch selection with random affine spatial augmentations.
TensorBoard-based hierarchical visualization of the computation
Regression metrics, computed using NiftyNet’s ‘evaluation‘ ac-
graphs. The scope of each conceptual blocks maps to a meaning-
tion, and aggregated over all folds, are given in Table 2. The 25th
ful subgraph of the computation graph and all associated variables,
and 75th percentile example result with regards to MAE is shown
meaning that all weights for a conceptual block can be loaded into
in Fig. 5.
a new model with a single scope reference. Furthermore, because
these conceptual blocks are constructed hierarchically through the
6.3. Ultrasound simulation using generative adversarial networks composition of Layer objects and scopes, they naturally encode a
hierarchical structure for TensorBoard visualization
Generating plausible images with specified image content can Supporting machine learning for a wide variety of applica-
support training for radiological or image-guided interventional tion types motivated the separation of the ApplicationDriver
tasks. Conditional GANs have shown promise for generating plausi- logic that is common to all applications from the Application
ble photographic images [41]. Recent work on spatially-conditioned logic that varies between applications. This facilitated the rapid
GANs [26] suggests that conditional GANs could enable software- development of new application types. The early inclusion of
based simulation in place of costly physical ultrasound phantoms both image segmentation/regression (mapping from images to im-
used for training. Here we present an example illustrating a pre- ages) and image generation (mapping from parameters to images)
 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122 121

Fig. 6. Interpolated images from the generative model space based on linearly interpolated model parameters. The top row shows a smooth variation between different
amounts of ultrasound shadow artefacts. The bottom row shows a sharp transition suggesting the presence of mode collapse in the generative model.

motivated a flexible specification for the number, type and seman- Conflict of interest
tic meaning of inputs and outputs, encapsulated in the Sampler
and Aggregator components. None.

7.2. Platform availability Acknowledgments

The NiftyNet platform is available from http://niftynet.io/. The The authors would like to acknowledge all of the contribu-
source code can be accessed from the Git repository6 or installed tors to the NiftyNet platform. This work was supported by the
as a Python library using pip install niftynet. NiftyNet is Wellcome/EPSRC [203145Z/16/Z, WT101957, NS/A0 0 0 027/1]; Well-
licensed under an open-source Apache 2.0 license7 . The NiftyNet come [106882/Z/15/Z, WT103709]; the Department of Health and
Consortium welcomes contributions to the platform and seeks in- Wellcome Trust [HICF-T4-275, WT 97914]; EPSRC [EP/M020533/1,
clusion of new community members to the consortium. EP/K503745/1, EP/L016478/1]; the National Institute for Health Re-
search University College London Hospitals Biomedical Research
7.3. Future direction Centre (NIHR BRC UCLH/UCL High Impact Initiative); Cancer Re-
search UK (CRUK) [C28070/A19985]; the Royal Society [RG160569];
The active NiftyNet development roadmap is focused on three a UCL Overseas Research Scholarship, and a UCL Graduate Re-
key areas: new application types, a larger model zoo and more search Scholarship. The authors would like to acknowledge that
advanced experimental design. NiftyNet currently supports image the work presented here made use of Emerald, a GPU-accelerated
segmentation, regression, generation and representation learning High Performance Computer, made available by the Science & En-
applications. Future applications under development include image gineering South Consortium operated in partnership with the STFC
classification, registration, and enhancement (e.g. super-resolution) Rutherford-Appleton Laboratory; and hardware donated by NVIDIA.
as well as pathology detection. The current NiftyNet model zoo
contains a small number of models as proof of concept; expand- References
ing the model zoo to include state-of-the-art models for common
[1] M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G.S. Corrado,
tasks and public challenges (e.g. brain tumor segmentation (BRaTS) A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving,
[39,55]); and models trained on large data sets for transfer learn- M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mane,
R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner,
ing will be critical to accelerating research with NiftyNet. Finally,
I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viegas,
NiftyNet currently supports a simplified machine learning pipeline O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, X. Zheng, TensorFlow:
that trains a single network, but relies on users for data partition- large-scale machine learning on heterogeneous distributed systems, White Pa-
ing and model selection (e.g. hyper-parameter tuning). Infrastruc- per, 2016. arXiv: 1603.04467v2.
[2] B.B. Avants, N.J. Tustison, G. Song, P.A. Cook, A. Klein, J.C. Gee, A reproducible
ture to facilitate more complex experiments, such as built-in sup- evaluation of ANTs similarity metric performance in brain image registration,
port for cross-validation and standardized hyper-parameter tuning Neuroimage 54 (3) (2011) 2033–2044.
will, in the future, reduce the implementation burden on users. [3] J.L. Ba, J.R. Kiros, G.E. Hinton, Layer normalization (2016). arXiv:1607.06450v1.
[4] F. Bastien, P. Lamblin, R. Pascanu, J. Bergstra, I.J. Goodfellow, A. Bergeron,
N. Bouchard, Y. Bengio, Theano: new features and speed improvements, in:
8. Summary of contributions and conclusions Proceedings of the Workshop on Deep Learning and Unsupervised Feature
Learning NIPS, 2012.
[5] L. Berger, E. Hyde, M.J. Cardoso, S. Ourselin, An adaptive sampling scheme
This work presents the open-source NiftyNet platform for deep to efficiently train fully convolutional networks for semantic segmentation,
learning in medical imaging. Our modular implementation of (2017). ArXiv e-prints arXiv:1709.02764.
the typical medical imaging machine learning pipeline allows re- [6] M. Brett, M. Hanke, B. Cipollini, M.-A. Côté, C. Markiewicz, S. Gerhard, E. Lar-
son, Nibabel, 2016, Online. doi:10.5281/zenodo.60808.
searchers to focus implementation effort on their specific inno- [7] N. Burgos, M. Cardoso, K. Thielemans, M. Modat, S. Pedemonte, J. Dickson,
vations, while leveraging the work of others for the remaining A. Barnes, R. Ahmed, J. Mahoney, J. Schott, J. Duncan, D. Atkinson, S. Arridge,
pipeline. The NiftyNet platform provides implementations for data B. Hutton, S. Ourselin, Attenuation correction synthesis for hybrid PET-MR
scanners: Application to brain studies, Med. Imaging IEEE Trans. 33 (12) (2014)
loading, data augmentation, network architectures, loss functions 2332–2341.
and evaluation metrics that are tailored for the idiosyncracies of [8] M. Cardoso, M. Clarkson, M. Modat, S. Ourselin, NiftySeg: open-source software
medical image analysis and computer-assisted intervention. This for medical image segmentation, label fusion and cortical thickness estimation,
in: Proceedings of the ISBI Workshop on Open Source Medical Image Analysis
infrastructure enables researchers to rapidly develop deep learning
Software, 2012.
solutions for segmentation, regression, image generation and rep- [9] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, E. Shel-
resentation learning applications, or extend the platform to new hamer, cuDNN: efficient primitives for deep learning, arXiv:1410.0759v3.
[10] F. Chollet, et al., Keras, 2015, https://github.com/fchollet/keras.
applications.
[11] Ö. Çiçek, A. Abdulkadir, S.S. Lienkamp, T. Brox, O. Ronneberger, 3D U-net:
learning dense volumetric segmentation from sparse annotation, in: Proceed-
ings of the MICCAI, Springer, 2016, pp. 424–432.
6
https://github.com/NifTK/NiftyNet. [12] M.J. Clarkson, G. Zombori, S. Thompson, J. Totz, Y. Song, M. Espak, S. Johnsen,
7
https://www.apache.org/licenses/LICENSE-2.0. D. Hawkes, S. Ourselin, The NifTK software platform for image-guided inter-
122 E. Gibson et al. / Computer Methods and Programs in Biomedicine 158 (2018) 113–122

ventions: platform overview and NiftyLink messaging, Int. J. Comput. Assist. [35] S.-C. Lo, S.-L. Lou, J.-S. Lin, M.T. Freedman, M.V. Chien, S.K. Mun, Artificial con-
Radiol. Surg. 10 (3) (2015) 301–316. volution neural network techniques and applications for lung nodule detection,
[13] R. Collobert, K. Kavukcuoglu, C. Farabet, Torch7: A MATLAB-like environment IEEE Trans. Med. Imaging 14 (4) (1995) 711–718.
for machine learning, in: Proceedings of the NIPS Workshop on Algorithms, [36] B.C. Lowekamp, D.T. Chen, L. Ibáñez, D. Blezek, The design of SimpleITK, Front.
Systems, and Tools for Learning at Scale (Big Learning), in: EPFL-CONF-192376, Neuroinf. 7 (2013).
2011. [37] D. Mané, et al., TensorBoard: TensorFlow’s visualization toolkit, 2015, https:
[14] A.M. Dale, B. Fischl, M.I. Sereno, Cortical surface-based analysis:I. Segmentation //github.com/tensorflow/tensorboard.
and surface reconstruction, Neuroimage 9 (2) (1999) 179–194. [38] A. Mehrtash, M. Pesteie, J. Hetherington, P.A. Behringer, T. Kapur, W.M. Wells
[15] S. Dieleman, J. Schlter, C. Raffel, E. Olson, S.K. Snderby, D. Nouri, D. Maturana, III, R. Rohling, A. Fedorov, P. Abolmaesumi, DeepInfer: Open-source deep learn-
M. Thoma, E. Battenberg, J. Kelly, J.D. Fauw, M. Heilman, D.M. de Almeida, B. ing deployment toolkit for image-guided therapy, in: Proceedings of the SPIE,
McFee, H. Weideman, G. TakÃ!‘cs, P. de Rivaz, J. Crall, G. Sanders, K. Rasul, C. Medical Imaging, 10135, NIH Public Access, 2017.
Liu, G. French, J. Degrave, Lasagne: first release, 2015, 10.5281/zenodo.27878 [39] B.H. Menze, A. Jakab, S. Bauer, J. Kalpathy-Cramer, K. Farahani, J. Kirby, Y. Bur-
[16] T. Doel, D.I. Shakir, R. Pratt, M. Aertsen, J. Moggridge, E. Bellon, A.L. David, ren, N. Porz, J. Slotboom, R. Wiest, L. Lanczi, E. Gerstner, M.A. Weber, T. Ar-
J. Deprest, T. Vercauteren, S. Ourselin, GIFT-Cloud: A data sharing and col- bel, B.B. Avants, N. Ayache, P. Buendia, D.L. Collins, N. Cordier, J.J. Corso,
laboration platform for medical imaging research, Comput. Methods Programs A. Criminisi, T. Das, H. Delingette, Ç. Demiralp, C.R. Durst, M. Dojat, S. Doyle,
Biomed. 139 (2017) 181–190. J. Festa, F. Forbes, E. Geremia, B. Glocker, P. Golland, X. Guo, A. Hamamci,
[17] H. Dong, A. Supratak, L. Mai, F. Liu, A. Oehmichen, S. Yu, Y. Guo, TensorLayer: a K.M. Iftekharuddin, R. Jena, N.M. John, E. Konukoglu, D. Lashkari, J.A. Ma-
versatile library for efficient deep learning development, ACM Multimed, 2017. riz, R. Meier, S. Pereira, D. Precup, S.J. Price, T.R. Raviv, S.M.S. Reza, M. Ryan,
[18] L. Fidon, W. Li, L.C. Garcia-Peraza-Herrera, J. Ekanayake, N. Kitchen, S. Ourselin, D. Sarikaya, L. Schwartz, H.C. Shin, J. Shotton, C.A. Silva, N. Sousa, N.K. Sub-
T. Vercauteren, Generalised Wasserstein Dice score for imbalanced multi-class banna, G. Szekely, T.J. Taylor, O.M. Thomas, N.J. Tustison, G. Unal, F. Vasseur,
segmentation using holistic convolutional networks, (2017). arXiv:1707.00478. M. Wintermark, D.H. Ye, L. Zhao, B. Zhao, D. Zikic, M. Prastawa, M. Reyes,
[19] B. Fischl, M.I. Sereno, A.M. Dale, Cortical surface-based analysis: II: inflation, K.V. Leemput, The multimodal brain tumor image segmentation benchmark
flattening, and a surface-based coordinate system, Neuroimage 9 (2) (1999) (BraTS), IEEE Trans. Med. Imaging 34 (10) (2015) 1993–2024.
195–207. [40] F. Milletari, N. Navab, S.-A. Ahmadi, V-Net: Fully convolutional neural networks
[20] E. Gamma, J. Vlissides, R. Johnson, R. Helm, Design Patterns: Elements of for volumetric medical image segmentation, in: Proceedings of the Fourth In-
Reusable Object-Oriented Software, Addison-Wesley, 1994. ternational Conference on 3D Vision (3DV), 2016, pp. 565–571.
[21] L.C. Garcia-Peraza-Herrera, W. Li, L. Fidon, C. Gruijthuijsen, A. Devreker, G. Atti- [41] M. Mirza, S. Osindero, Conditional generative adversarial nets. Proc. NIPS 2016
lakos, J. Deprest, E.V. Poorten, D. Stoyanov, T. Vercauteren, S. Ourselin, ToolNet: Workshop on Adversarial Training. 2016 (2014), arXiv:1411.1784.
holistically-nested real-time segmentation of robotic surgical tools, in: 2017 [42] M. Modat, G.R. Ridgway, Z.A. Taylor, M. Lehmann, J. Barnes, D.J. Hawkes,
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), N.C. Fox, S. Ourselin, Fast free-form deformation using graphics processing
Vancouver, BC, 2017, pp. 5717–5722, doi:10.1109/IROS.2017.8206462. units, Comput. Methods Progr. Biomed. 98 (3) (2010) 278–284.
[22] E. Gibson, F. Giganti, Y. Hu, E. Bonmati, S. Bandula, K. Gurusamy, B. David- [43] M. Nolden, S. Zelzer, A. Seitel, D. Wald, M. Müller, A.M. Franz, D. Maleike,
son, S.P. Pereira, M.J. Clarkson, D.C. Barratt, Automatic multi-organ segmenta- M. Fangerau, M. Baumhauer, L. Maier-Hein, K.H. Maier-Hein, H.-P. Meinzer,
tion on abdominal CT with dense v-networks, IEEE Trans. Med. Imaging (2017). I. Wolf, The medical imaging interaction toolkit: challenges and advances, Int.
in press J. Comput. Assist. Radiol. Surg. 8 (4) (2013) 607–620.
[23] E. Gibson, F. Giganti, Y. Hu, E. Bonmati, S. Bandula, K. Gurusamy, B.R. Davidson, [44] S. Pieper, B. Lorensen, W. Schroeder, R. Kikinis, The NA-MIC kit: ITK, VTK,
S.P. Pereira, M.J. Clarkson, D.C. Barratt, Towards image-guided pancreas and bil- pipelines, grids and 3D Slicer as an open platform for the medical image com-
iary endoscopy: automatic multi-organ segmentation on abdominal CT with puting community, in: Proceedings of the IEEE International Symposium on
dense dilated networks, in: Proceedings of the 20th International Conference Biomedical Imaging: From Nano to Macro (ISBI), 2006, pp. 698–701.
on Medical Image Computing and Computer Assisted Intervention (MICCAI), [45] T. Preston-Werner, Semantic versioning, Technical Report, T. Preston-Werner,
2017. 2015. URL: http://semver.org/.
[24] E. Gibson, M.R. Robu, S. Thompson, P.E. Edwards, C. Schneider, K. Gurusamy, [46] M. Reynolds, et al., Sonnet, 2017, https://github.com/deepmind/sonnet.
B. Davidson, D.J. Hawkes, D.C. Barratt, M.J. Clarkson, Deep residual networks [47] H.R. Roth, A. Farag, E.B. Turkbey, L. Lu, J. Liu, R.M. Summers, Data from
for automatic segmentation of laparoscopic videos of the liver, in: Proceed- TCIA pancreas-CT, The Cancer Imaging Archive, 1994, doi:10.7937/K9/TCIA.
ings of the SPIE, Medical Imaging, vol. 10135, 2017, doi:10.1117/12.2255975. 2016.tNB1kqBU.
101351M [48] H.R. Roth, L. Lu, A. Seff, K.M. Cherry, J. Hoffman, S. Wang, J. Liu, E. Turkbey,
[25] I. Goodfellow, NIPS 2016 tutorial: generative adversarial networks (2016). R.M. Summers, A new 2.5D representation for lymph node detection using ran-
arXiv:1701.00160v4. dom sets of deep convolutional neural network observations, in: Proceedings
[26] Y. Hu, E. Gibson, L.-L. Lee, W. Xie, D.C. Barratt, T. Vercauteren, J.A. Noble, Free- of the MICCAI, 2014, doi:10.1007/978- 3- 319- 10404-1_65.
hand ultrasound image simulation with spatially-conditioned generative ad- [49] T. Salimans, D.P. Kingma, Weight normalization: a simple reparameterization
versarial networks, in: Proceedings of MICCAI Workshop on Reconstruction to accelerate training of deep neural networks (2016). arXiv:1602.07868v3.
and Analysis of Moving Body Organs (RAMBO), 2017. [50] F. Seide, A. Agarwal, CNTK: Microsoft’s open-source deep-learning toolkit, in:
[27] S. Ioffe, Batch renormalization: towards reducing minibatch dependence in Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge
batch-normalized models, in: I. Guyon, U.V. Luxburg, S. Bengio, H. Wallach, Discovery and Data Mining, ACM, 2016, p. 2135.
R. Fergus, S. Vishwanathan, R. Garnett (Eds.), Advances in Neural Information [51] D. Shen, G. Wu, H.-I. Suk, Deep learning in medical image anal-
Processing Systems 30, 2017, pp. 1942–1950. ysis, Ann. Rev. Biomed. Eng. (19) (2017) 221–248, doi:10.1146/
[28] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, annurev- bioeng- 071516- 044442.
T. Darrell, Caffe: Convolutional architecture for fast feature embedding, in: Pro- [52] S.M. Smith, M. Jenkinson, M.W. Woolrich, C.F. Beckmann, T.E. Behrens, H. Jo-
ceedings of the 22nd ACM International Conference on Multimedia (ACMMM), hansen-Berg, P.R. Bannister, M. De Luca, I. Drobnjak, D.E. Flitney, R. Niazy,
ACM, 2014, pp. 675–678. J. Saunders, J. Vickers, Y. Zhang, N. De Stefano, J. Brady, P. Matthews, Advances
[29] S.F. Johnsen, Z.A. Taylor, M.J. Clarkson, J. Hipwell, M. Modat, B. Eiben, L. Han, in functional and structural MR image analysis and implementation as FSL,
Y. Hu, T. Mertzanidou, D.J. Hawkes, S. Ourselin, NiftySim: a GPU-based non- Neuroimage 23 (2004) S208–S219.
linear finite element package for simulation of soft tissue biomechanics, Int. J. [53] C.H. Sudre, W. Li, T. Vercauteren, S. Ourselin, M.J. Cardoso, Generalised Dice
Comput. Assist. Radiol. Surg. 10 (7) (2015) 1077–1095. overlap as a deep learning loss function for highly unbalanced segmentations,
[30] K. Kamnitsas, C. Ledig, V.F. Newcombe, J.P. Simpson, A.D. Kane, D.K. Menon, in: Proceedings of MICCAI Workshop on Deep Learning in Medical Image Anal-
D. Rueckert, B. Glocker, Efficient multi-scale 3D CNN with fully connected CRF ysis (DLMIA), 2017.
for accurate brain lesion segmentation, Med. Image Anal. 36 (2017) 61–78. [54] A. Vedaldi, K. Lenc, MatConvNet – convolutional neural networks for MATLAB,
[31] S. Klein, M. Staring, K. Murphy, M.A. Viergever, J.P. Pluim, Elastix: a toolbox for in: Proceedings of the ACMM, 2015.
intensity-based medical image registration, IEEE Trans. Med. Imaging 29 (1) [55] G. Wang, W. Li, S. Ourselin, T. Vercauteren, Automatic brain tumor segmen-
(2010) 196–205. tation using cascaded anisotropic convolutional neural networks, Proc. Multi-
[32] B. Landman, Z. Xu, J.E. Igelsias, M. Styner, T.R. Langerak, A. Klein, Multi- modal Brain Tumor Segmentation (BRATS) Challenge 2017 - MICCAI workshop.
atlas labeling beyond the cranial vault, 2015, URL: https://www.synapse.org/ 2016 (2017). arXiv:1709.00382.
#!Synapse:syn3193805, accessed July 2017. 10.7303/syn3193805. [56] G. Wang, M.A. Zuluaga, W. Li, R. Pratt, P.A. Patel, M. Aertsen, T. Doel, A.L. David,
[33] W. Li, G. Wang, L. Fidon, S. Ourselin, M.J. Cardoso, T. Vercauteren, On the com- J. Deprest, S. Ourselin, T. Vercauteren, DeepIGeoS: a deep interactive geodesic
pactness, efficiency, and representation of 3D convolutional networks: Brain framework for medical image segmentation, (2017). arXiv:1707.00652v1.
parcellation as a pretext task, in: Proceedings of Information Processing in [57] X. Zhou, T. Ito, R. Takayama, S. Wang, T. Hara, H. Fujita, Three-dimensional
Medical Imaging (IPMI), 2017, pp. 348–360. CT image segmentation by combining 2D fully convolutional network with 3D
[34] G. Litjens, T. Kooi, B.E. Bejnordi, A.A.A. Setio, F. Ciompi, M. Ghafoorian, majority voting, in: Proceedings of the LABELS, Springer, 2016, pp. 111–120,
J.A.W.M. van der Laak, B. van Ginneken, C.I. Sánchez, A survey on deep doi:10.1007/978- 3- 319- 46976- 8_12.
learning in medical image analysis, Medical Image Analysis 42 (2017) 60–88,
doi:10.1016/j.media.2017.07.005.