Computers & Geosciences 52 (2013) 218–226

Contents lists available at SciVerse ScienceDirect

Computers & Geosciences

journal homepage: www.elsevier.com/locate/cageo

Doing fieldwork on the seafloor: Photogrammetric techniques to

yield 3D visual models from ROV video
Tom Kwasnitschka n, Thor H. Hansteen, Colin W. Devey, Steffen Kutterolf
GEOMAR Helmholtz Centre for Ocean Research Kiel, Wischhofstr. 1-3, D-24148 Kiel, Germany

a r t i c l e i n f o a b s t r a c t

Article history: Remotely Operated Vehicles (ROVs) have proven to be highly effective in recovering well localized
Received 26 July 2012 samples and observations from the seafloor. In the course of ROV deployments, however, huge amounts
Received in revised form of video and photographic data are gathered which present tremendous potential for data mining.
10 October 2012
We present a new workflow based on industrial software to derive fundamental field geology
Accepted 11 October 2012
Available online 23 October 2012
information such as quantitative stratigraphy and tectonic structures from ROV-based photo and video
material. We demonstrate proof of principle tests for this workflow on video data collected during dives
Keywords: with the ROV Kiel 6000 on a new hot spot volcanic field that was recently identified southwest of the
Physical volcanology island of Santo Antao~ in the Cape Verdes. Our workflow allows us to derive three-dimensional models
Photogrammetry
of outcrops facilitating quantitative measurements of joint orientation, bedding structure, grain size
Microbathymetry
comparison and photo mosaicking within a georeferenced framework. The compiled data facilitate
Visualization
Grain size analysis volcanological and tectonic interpretations from hand specimen to outcrop scales based on the
Underwater navigation quantified optical data. The demonstrated procedure is readily replicable and opens up possibilities
for post-cruise ‘‘virtual fieldwork’’ on the seafloor.
& 2012 Elsevier Ltd. All rights reserved.

1. Introduction post-cruise. This motivated the development of the photogram-

metric mapping workflow described below, as available acoustic
The scientific use of Remotely Operated Vehicles (ROVs) has survey methods and video coverage did not allow adequate
traditionally concentrated on making visual observations, installing interpretation of the bedding structures and series of deposits
equipment precisely on the seafloor and the recovery of physical which we observed. The detailed geological interpretation and
samples. While the samples can later be localized with high resulting development of new concepts on eruptive mechanisms
precision, the sampling decision is often based upon very limited in the deep sea is subject of our ongoing research.
visual information relayed through video cameras. Without com- Our aim was to design a workflow that would allow examina-
prehensive prior surveys of the study area on scales at which the tion of the outcrops indirectly after the dive, based on a three-
ROV operates afterwards, researchers cannot be sure to be observing dimensional digital model (e.g. Kreylos et al., 2006). This model
or sampling in the scientifically most relevant places. should be accurate enough to allow quantitative measurements
During the RV Meteor 80/3 cruise to the Cape Verdes, we matching the precision of terrestrial surveys. At the same time,
surveyed a complex submarine cone field, called the Charles
the representation of geological features should be realistic and
Darwin Volcanic Field (CDVF), located at a depth of 3500 m off
unobstructed by effects of the water in order to allow the
the Island of Santo Anta~ o. Based on a ship-based bathymetric map
development of discussable qualitative field impressions. The
gridded at 25 m resolution (Fig. 1) and on a scattered set of
procedure should be stable, repeatable and reasonably fast
dredge samples, we identified targets for the detailed surveying of
several structures using the ROV Kiel6000 vehicle. It quickly while easy to use, allowing a first evaluation of new data while
became clear that the observational protocols usually used during still at sea. The workflow should be open, follow industry
ROV operations were failing to capture much geological informa- standards, and offer interfaces to further third party data treat-
tion (especially on the relationships between rock structures in ment procedures. We directed our attention towards existing
3D) which could greatly help interpretation of the samples industry-proven components that would allow integration on a
user level and be widely available to other users without
specialized programming knowledge (although a familiarity with
n
Corresponding author. Tel.: þ49 431 600 2136; fax: þ49 431 600 2924.
the features of the software we used (which can be acquired from
E-mail addresses: [email protected] (T. Kwasnitschka),
[email protected] (T.H. Hansteen), [email protected] (C.W. Devey), the software documentation and HowTo video tutorials) is
[email protected] (S. Kutterolf). assumed here).

0098-3004/$ - see front matter & 2012 Elsevier Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.cageo.2012.10.008
 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226 219

-27° -26° -25° -24° -23° -22° -21°

18°
a
17°

-4000 16°
16.75° b
15°
-3000

N
16.70° -3400
14°

-36
00
-38
00

16.65°

-25.6° -25.55° -25.5°

Fig. 1. Location of the working area. (a) Asterisk marks the location of CDVF on the southwestern slope of the Island of Santo Anta~ o. (b) Box marks the working area of ROV
dive 059 at Tambor Cone, part of the central group of CDVF (compare Fig. 4 for close up view).

2. Previous work seafloor geological interpretation, where dip angles are mostly
low and so features of interest are best viewed by looking
Photogrammetry has previously been successfully applied to a horizontally onto vertical walls.
variety of marine scientific disciplines, an overview of which is
given by Johnson-Roberson et al. (2010). Our own work was
inspired by recent applications in the field of aerial archeology
(Verhoeven, 2011) with which we share the same general 3. Hardware
requirements of georeferenced visualization of terrain features
on multiple scales. At the same time, photogrammetric methods The ROV KIEL 6000 is a 6000 m-rated work class ROV built by
have been developed to derive structural geology data from Schilling Robotics, Inc. Seven thrusters allow precise positioning;
outcrops in time-critical situations such as tunneling and quarry- an automatic station-keep mode is available for stationary opera-
ing (Gaich et al., 2003). Our own previous use of such packages tion. The lighting is centered to the front within a cone of
(Kwasnitschka, 2008) also informed the development of the approximately 1001, featuring a total of 990 W. Table 1 and
current project. Fig. 2 give an overview of the standard camera equipment
The underwater application of photogrammetric techniques available on the vehicle. In addition to this, an experimental
poses considerable problems since all camera equipment need to stereo camera rig consisting of two Ocean Imaging Systems
be housed in some form of water-tight (and, in our case, pressure- camera housings with Nikon D80 cameras and 20 mm optics
resistant) housing. The air–glass–water interface introduced by was tested during the test deployments described here. The
this camera housing acts as an additional optical element – in the cameras were mounted parallel at a separation of 20 cm between
case of pressure housings, the optical geometry will also change the optical axes. All housings are equipped with flat glass ports.
in a possibly non-linear manner as its multiple components Only the stereo rig was successfully calibrated for optical aberra-
deform under load. Various approaches have been taken to tions since the other cameras, mechanically and electrically fully
address these issues (e.g. Beall et al., 2010; Sedlazeck and Koch, integrated into the vehicle itself, were unavailable for such a
2011). Previous authors have investigated the optical aberrations procedure.
induced by the air–glass–water interface (e.g. Sedlazeck et al., The underwater navigation system installed on the vehicle
2009) providing critical constraints for positional reconstruction consists of three elements. An RDI Workhorse Navigator 1200
in the absence of reliable external navigation data. Doppler Velocity Logger (DVL) provides three-dimensional
Using established subaerial calibration methods, it is also differential motion tracking at sub-centimeter resolution. Abso-
possible to achieve locally well-defined reconstructions by using lute position determination is supported by a Posidonia 6000
rigid stereoscopic camera rigs (Johnson-Roberson et al., 2010) or Ultra Short Baseline Logger (USBL) pinger installed on the
monoscopic approaches (Pizarro and Eustice, 2009) and aligning vehicle and used in combination with a USBL antenna mounted
clusters of such local models using vehicle navigation data. Much on the surface vessel (in this case RV METEOR). A compass
of the previous work has been made using images from Auton- (Precision Navigation TCM2-50) provides information
omous Underwater Vehicles (AUVs) rather than ROVs since they on the vehicle heading. Depth and altitude readings were pro-
are generally able to cover larger areas due to their deployment vided by a Seabird Electronics FastCat SBE 49 CTD sensor, the DVL
scheme (Yoerger et al., 2007). For our scientific purposes, though, and by a Paroscientific Digiquartz pressure sensor. Orientation
AUVs have two major drawbacks: (a) they do not allow an data of the vehicle (heading, pitch and roll) were received from a
immediate reaction to new discoveries and so often have objects Crossbow VG700CA orientation sensor. Pan and tilt values of
of interest at the edges of images and (b) they generally view the navigation cameras were logged from the Schilling rotary
scenes from above, not the visualization angle most suited to actuators.
220 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226

Table 1
Relevant camera equipment installed on ROV Kiel 6000 (only HD and stereo cameras were used for reconstruction).

Brand Recording format FoV Type of mount Operation mode

Navigation cameras Kongsberg OE14–366 MKII PAL 631 Pan/tilt, logged Continuous
HD cameras Kongsberg OE14–500 HDV, 1080i 501 tilt, unlogged On demand
Rear navigation camera Oktopus 6000 PAL 751 Fixed Not recorded
Stereo camera Ocean imaging systems 10.2MP RAW 571 Fixed 4 s interval on demand
Still camera Kongsberg OE14–208 5MP JPEG 621 Pan/tilt, logged On demand

c
c

e
Top Nav Camera e f f
1.75 m 1.48 m
d
Inclination Sensor
Low Nav Camera Compass
1.10 m
d

HD Camera
0.46 m
0.40 m 0m
Left Stereo Camera DVL Sensor

DVL Sensor 0 m g

0m
1.30 m 0.90 m 0m 2.20 m Left Stereo Camera
0.80 m
HD Camera
Top/Low Nav Cameras

Fig. 2. Cameras, positional sensors and lighting of ROV Kiel 6000 seen from (a) front and (b) right views. Relevant lighting equipment is labeled as (c) Flash guns for the
stereo system (OE11–242), (d) Halogen lamps (Deep Multi-SeaLite) (e) HMI lamps (SeaArc 2), (f) HID lamps (SeaArc 5000). The stereo camera rig was mounted on a
retractable sled and is shown in retracted position, while during operation the lenses were positioned over the front edge of the vehicle. The DVL and orientation sensors
are obstructed. Approximate positions of the cameras and sensors are given relative to (g) lower front port corner of vehicle. Photos courtesy H. Huusmann and N.
Augustin.

4. Data -25.500° -25.495° -25.490° -25.485° Geogr.

Position
End
The outcrops which we used to test the 3D reconstruction a
workflow are situated on the Charles Darwin Seamounts. They 1400

De
16.720°

via
display abundant units with clearly visible bedding structures cut

tio
by several generations of joints. Our test data were acquired on a

n
1200
ROV dive 059 during Meteor cruise 80/3 which led across the
northwestern flank of a volcanic crater showing the morphologi- 1000
cal characteristics of a tuff ring (Fig. 3). The ring rises about 200 m ise
ec
above the surrounding terrain and its central depression mea-
USBL Raw
pr
m
,i
DVL Raw
800 te 16.715°
sures 1 km across featuring steep inner walls dropping step wise USBL Smoothed
a
Synthesized ur
between 20 m and 50 m at angles between 351 and 901. The acc
600 L: e
structure is composed of loose to strongly compacted volcani- B rat
US accu
clastics and lava flows, featuring a sparse to locally moderate e, in
population of vagile and sessile epibenthic fauna. cis
400
: pre
The sample film clip (059–47) was recorded during a descent D VL
16.710°
of the vehicle 28 m down a cliff of 651 inclination with sedimen- 200
tary talus at the foot of the cliff marking the transition to the Start
crater floor. Using the ROVs onboard sonar, a roughly constant 0
distance of 2.5 m to the wall could be maintained. Consequently, N
the width of the model ranges between 3.8 m and 8.5 m. The Distance
0 200 400 600 800 1000 1200 1400
[m]
duration of the sequence is just over 2 min, resulting in a set of
125 source images after pre-processing. All but the last seven Fig. 3. The Vehicle track of dive 059 used for alignment of the reconstructed
images (where the image quality deteriorated with increasing geometry data consists of the merged DVL bottom velocity and USBL data
distance to the wall) could be included in the reconstruction. This substituted by just the smoothed USBL data where DVL data was unavailable
(a). The gray arrow marks the accumulated offset (458 m) between the raw USBL
yielded a model of 200,000 polygons and a texture of 8192  8192 and uncorrected DVL signal by the end of the dive.
pixels, corresponding to an estimated geometrical resolution of
15 cm and a local maximum textural resolution of 2 cm. The
textures of the uppermost and lowermost 25% of the model in in the horizontal viewing angle, some laterally facing portions,
particular show signs of color absorption into bluish hues due to a especially features facing right, remained occluded from the
slightly increased object distance. Since there was little variation camera and could not be textured.
 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226 221

5. Data pre-processing 6. Photogrammetric reconstruction

The individual streams of video and still imagery were A number of different processing approaches can be grouped
synchronized in Adobe Premiere and merged into a master under the term photogrammetry, employing heterogeneous
stream for reference. At this point it became clear that we would image clusters, a single camera or stereo pairs. In order to
not be able to use the data from the experimental stereo mathematically reverse the projection of a camera and recon-
camera system as it showed synchronization faults and had struct three-dimensional information from two-dimensional
collected data only sporadically during the dive. Pre-selected images, the algorithms require the extrinsic camera parameters
video sequences covering outcrops were reformatted using Adobe (essentially, camera position relative to the object) and intrinsic
After Effects. Elements obstructing the view such as visible parts camera parameters (a description of the optical path of light
of the vehicle and non-static objects were masked out from the through the lens onto the sensor). A truly accurate reconstruction
video (as they are not part of the static seafloor which we wanted can only be achieved if there is precise information on the
to reconstruct and so would have led to erroneous results during intrinsic parameters and some minimal external information in
automated 3D reconstruction). No noise reduction filters were the form of camera position and orientation or reference points
applied as we found that the resulting loss of detail caused on the object in order to establish absolute scale, orientation and
reconstruction gaps on uniform surfaces. The clips were exported position within a world coordinate system.
at a rate of one frame per second as still sequences with the mask After tests using a variety of motion tracking packages
embedded as an alpha channel. The complete workflow is (PFtrack5.0, Boujou; Condell et al., 2006), open source and free
illustrated in Fig. 4. bundle adjustment software (Bundler, Microsoft Photosynth;
The USBL (absolute position with relatively large errors) and Triggs et al., 2000, Snavely et al., 2010), we now rely on Agisoft
DVL (relative position with smaller errors) navigational data Photoscan Professional, a commercial suite for aerial photogram-
underwent extensive correction as it formed the only frame of metry. This software offers an integrated workflow including a
reference to which the quality of reconstructed models could core of sturdy reconstruction and georeferencing tools along with
be gauged. Using a semi-automatic Matlab routine, we generated sufficient means to pre-process the input images and edit the
a hybrid vehicle path, stabilizing the two data sources with finished models for further export. Verhoeven (2010) gives a
respect to each other (Fig. 3): The x and y components of detailed overview of the software functionality and the basic
both signals were filtered for system-inherent outliers, after process of (1) model triangulation from image features matched
which the short-wavelength component of the DVL was copied across a cluster of overlapping images into (2) a sparse estimate of
onto the long-wavelength component of the USBL signal. We take the scene geometry from which (3) a dense model is derived,
this to be a best-practice approach with the given data quality, followed by (4) surface modeling, and (5) texture generation
with the drawback that, in passages where the DVL failed, the (Fig. 4). The range of features has since been expanded for
precision of the accepted position is diminished to an average of georeferencing based on landmarks visible on the model for
the USBL position and can only serve for rough georeferencing but which coordinates have been determined by GPS. Since a
not for quality control. Likewise, the vertical component of the high-resolution, georeferenced AUV map was unavailable, we
DVL signal was corrected for drift effects and failed passages registered our models to the camera poses (i.e. position and
using the merged records of the CTD and Digiquartz pressure orientation) derived from the ROV motion record. The software
sensors. first tries to georeference by a best-fit, rigid seven-parameter

Preparation Data Generation Interpretation

Still Photos 1
Stereo Photos Media Synchonized 4 Distributable
SD-Video Synchronization Media Stream Sparse Reconstruction Camera Positions Outcrop Visualizations
HD-Video Dense Reconstruction
2
Frame Extraction Raw Outcrop
Mesh Triangulation
Masking Geometry Real Time Data Visualization
Noise Reduction Manual Mesh Editing Virtual Globes6
Color Correction Georeferencing Interactive Viewers7
GIS Projects8
Nonlinear Optimization

1 Hz Still Frame Sequence Planar Mosaicking Image Mosaic 5

Texturization Georeferenced Model editing
Outcrop Model Quantitative Analysis
DVL 3 Export of Visual Interpretation
USBL Navigation
Compass Signal
CTD Synthesis
Depth 9
Preparation of 2D Graphics

Commercial Software Products employed in the Workflow:

Illustrations Measurement Data
1. Premiere Pro (Adobe Systems, Inc.) 6. Google Earth (Google, Inc.)
2. After Effects (Adobe Systems, Inc.) World Wide Telescope (Microsoft Research)
3. Matlab (The MathWorks, Inc.) 7. Acrobat (Adobe Systems, Inc.)
4. Photoscan Professional (Agisoft Llc.) 8. ArcGIS (Esri, Inc.),
5. 3dsMax (Autodesk, Inc.) Fledermaus (QPS, B.V.)
9. Illustrator (Adobe Systems, Inc.)

Fig. 4. Workflow scheme developed during this study employs a number of commercial software products which are linked using their own data interfaces as well as
custom import and export scripts for quantitative interpretation.
222 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226

transformation. In a second step, the reconstructed geometry and For immediate visualization purposes, the model resolution is
camera poses are subjected to a nonlinear optimization process to diminished to 20,000 polygons, and the texture is downscaled
the ROV vehicle track. For all poses, the degree of misfit is from 8192  8192 pixel to 2048  2048 pixel to be compatible to
reported, along with RMSE values. real time viewing applications. Another 3ds version is saved,
Afterwards, the photographic texture can be exported as an along with a georeferenced Collada model with a KML link, and a
orthophoto mosaic after re-projecting the images from the camera U3D file contained in a PDF.
poses back across the model surface. Various texture blending rules
can be chosen to treat overlapping areas. The most instructive results
can be acquired by always choosing the brightest available pixel 7. Model interpretation
from the set of overlapping pixels which could be projected onto a
given surface coordinate. This somewhat suppresses the effects of The goal of our project was to derive quantitative data of
light absorption through the water and brightness falloff towards the geological structures beyond mere size and distance measure-
outer parts of the cone of illumination. In the case of strongly varying ments. To achieve this, we edit the models in Autodesk 3dsMax,
object distances, more visually pleasing results can be achieved by which counts among the standard tools of the CGI industry. One
forming an average of overlapping pixels. A mathematically correct major drawback of the software is that geographic references are
treatment of this problem appears to be difficult as the distance of lost while scale and orientation are maintained. The 3D scenes
the objects to the light sources and the camera varies strongly build upon a cartesian coordinate system in meters but cannot
compared to a case of flat sea bottom and a monotonous vehicle deal with geographic positions due to limitations in computa-
track, where such corrections have been successfully demonstrated tional precision facing large numbers (Mach and Petschek, 2007).
(Sedlazeck et al., 2009; Johnson-Roberson et al., 2010). A workaround is to define a reference point at the center of the
The models can be exported to a variety of formats and working area and to arrange all data relative to that point.
geographic reference frames. The full-resolution model is To derive quantitative geological information from the model,
exported in the Autodesk 3ds format containing the textured we follow the same basic route (e.g. Jones et al., 2008): (1) Create
model and the camera positions to be used for further interpreta- an Autodesk 3dsMax helper object, (2) align and scale it to match
tion. The orthophoto mosaic is derived at a resolution of 5 cm per the geological feature to be measured, (3) read the respective
pixel, which matches the average resolution of the input material. property of the helper object. The additional benefit is a direct

Unit 4
Example Joint
Joint
Bedding Horizon
Unit Boundary
Erosional Unconformity
Unit 3b

Unit 3

Unit 3a

Unit 2b

Unit 2

Unit 2a

Unit 1b

Unit 1

Unit 1a

1m

Fig. 5. (a) Vertical profile, geometric features and stratigraphic units of outcrop 059–47 derived from (b) the outcrop model. Lateral and vertical extent match in
dimensions. The white dashed line marks the location of the profile. Linears on the model mark the intersection of the rock face and the joints and bedding planes. Note the
white semitransparent example joint plane with indicators for strike (line) and dip (arrow) directions. (c) shows a frame of the original video sequence at its approximate
position in the outcrop (grey arrow).
 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226 223

visualization of the measurements, which can later be refined to models of the local bathymetry, still and video footage, and any
produce a visually informative illustration. other 2D or 3D content that can be referenced within the scene’s
To measure the orientation of planar structures such as faults, coordinate system (e.g. Kwasnitschka, 2008). Thus, this is the only
joints or bedding planes, a planar Autodesk ‘‘section object‘‘ is software applied in our study allowing the simultaneous visuali-
placed on the model. The orientation of this ‘‘section object‘‘ is zation of all our data sets in four dimensions. At the same time, it
then adjusted until its intersection with the modeled seafloor is the only fully featured application able to actively manipulate
matches that of the geological structure (Fig. 5). The more surface the models in order to create new data products.
relief there is on the model, the more accurate the measurement We use a number of other visualization platforms that mostly
is. This procedure is not only an easy graphical way to determine allow the passive interaction with the data:
the orientation of a planar geological structure; it also has some
distinct advantages over, for example, using a compass in the  A birds-eye view computer animation of the reconstructed
field. First it allows the orientation of features with ill-defined scene is rendered by 3dsMax and added to the original Adobe
boundaries (such as banks of coarse gravel) to be accurately Premiere video composition allowing the comparison of the
determined. Second it increases the sampling area for the orien- video material to a four dimensional, animated map.
tation measurement, providing a more representative ‘‘average  Various applications focus on the geospatial aspect of our data,
strike and dip‘‘ than a point measurement. first and foremost an ArcGIS project including the bathymetric
On a larger scale, a ‘‘master section‘‘ can be used to vertically map, the ROV track, the photo mosaics of the reconstructed
slice the orientated ‘‘section objects‘‘ along with the outcrop outcrops, event marks such as sample locations and the final
model, to provide a proportionally accurate geological profile geologic map layers. Some limited control over the temporal
through the outcrop which can be directly imported into vector coordinates of data are available, too.
drawing programs (Fig. 5a).  The bathymetric post-processing and visualization software Fle-
A grain size estimate can be derived from the models by dermaus is capable of displaying most of the GIS layers within a
creating appropriate spheres, ellipses or boxes around the clasts four dimensional space. The simulation is based around a bath-
to be measured, and reading the size of the object along the ymetrical height field and the animated ROV track. The mode of
respective axes of interest. Working with a multitude of indivi- visualization is passive as the software is designed primarily for
dual measurements, objects can be batch renamed and assembled the dissemination of bathymetry data, which has to be pre-
into groups while the arrangement of these groups into visibility processed. A major drawback to date is the restriction of import
layers allows order to be maintained. As the textural resolution is interfaces to support merely untextured models or monochrome
higher than the geometrical resolution, clasts can also be mea- point clouds providing a relatively inferior representation of the
sured based on the texture alone, allowing work down to the outcrop reconstruction effort.
centimeter scale.  Virtual Globes such as Google Earth and World Wide Telescope
Thanks to the powerful scripting interface of 3dsMax, we digest distribution formats (photo mosaics, 3ds and Collada
developed a suite of import and export scripts for camera poses, models using accompanying KML files) directly exported from
grain size data and orientation measurements converted to Photoscan. By the nature of the applications, navigation on a
geologic notation, and to visualize statistical parameters of the scale of meters is challenging.
reconstruction step. This feature also opens endless possibilities  An even easier way to examine the unedited reconstructions
for future, case-sensitive data mining routines as well as further with low-level tools is the U3D format contained in PDF
optimization of the models themselves. The scripts are available format, which allows passive interaction with the model
from the leader author on request. through Adobe Acrobat or Reader on any operating system.
Quantitative measurements of sizes, angles and even orienta-
tion can be made.
8. Visualization
KML samples of the data discussed in this article can be
Next to the quantitative evaluation of the models, another downloaded as online supplements.
main goal of our project is the enhancement of qualitative
analysis by means of appropriate data visualization. Particularly
ROV visual data is generally only made available to users in the 9. Consideration of errors
geologically irrelevant temporal dimension (time stamped rather
than georeferenced) meaning that much of the information it In the absence of hard constraints on size and orientation of
contains is difficult to access retrospectively. We wanted to use seafloor features provided by artificial gauges (e.g. parallel laser
the 3D models to transpose the ROV visual data set into a beams), evaluation of errors in the reconstructed geometry had to
geographical frame of reference and so provide access to the rely on indirect methods and assessment calculations in which
video information via its position. we vary influences of imperfect navigation and optical distortion.
Once more, 3dsMax serves as a powerful editor to prepare the We assume the positional uncertainty within the navigational
models for use in real time visualization software. But in addition data to exceed the drift of the geometrical reconstruction. We
it is also a visualization tool in its own right. Several outcrop infer this from a comparison of the reconstructed orientation data
models can be loaded at once, and upon re-establishing their with the orientation data derived from the on-board sensors
relative positions, the correlation of geologic features between (certified to be accurate within one degree). The in situ optical
adjacent outcrops is as natural as in the field on land. The distortion parameters could not be constrained rigorously. Never-
software allows appropriate visualization geometry to be con- theless, examining the contribution of different parameters to the
structed to illustrate the findings. Camera objects represent the overall result can elucidate the robustness of the reconstruction
poses in the scene. Since these originate from a film sequence and method. In all the cases outlined below, nonlinear optimization
the ROV followed a track, the path of the ROV across the outcrop was omitted in order to reveal the differences between acoustic
can be animated and navigated using a time line. navigation and optical measurements.
The pre-processed ROV track record can be visually compared For dive 059, with a total distance traveled of 3354 m at an
against the reconstructed ROV path, along with digital elevation average of 0.076 m/s during 12.2 h (Fig. 3), the expected drift of the
224 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226

Table 2 DVL sensor based on manufacturers specification lies at 795 m.

Top to bottom: Pose deviation of the best reconstruction attempt compared to Nevertheless, we found that, relative to USBL fixes (which have lower
effects of improper filtering, timing, capture interval, missing calibration aligned
precision (quoted by the manufacturer as 1% of the operational water
to the first pose (1) and the attempt to fit the uncalibrated model to the track
record (2). Values are averages or root mean square errors of the entire series with depth) but should not be subject to time-dependent drift) our
respect to the telemetry data. instrument deviated by 458 m (24% of the true distance between
start and end points) even after cleaning of the DVL record to remove
Orientation differences Positional differences obviously erroneous episodes (e.g. apparent movement of vehicle
(average in 1) (RMSE in m)
above its maximum velocity or movement when the video showed
Pitch Roll Heading Err x Err y Err z Net the vehicle was stationary). The USBL on RV Meteor had been
calibrated at depths comparable to our deployment relative to a
Proper 3.2 0.0 2.0 0.10 0.06 0.08 0.14 fixed seafloor beacon on a preceding cruise and been shown to have
USBL smooth  22.3 13.9 50.9 0.24 0.63 0.23 0.71
positional RMS error values of 13.7 m (x), 13.3 m (y) and 6.5 m (z)
USBL – 3 s  24.5 14.0 57.1 0.17 0.63 0.23 0.69
6 s early 1.2  0.9 1.8 0.23 0.21 0.38 0.49 relative to ship’s axis.
6 s late 4.7  0.4 3.8 0.24 0.17 0.27 0.40 In our example model, the overall, timestamp corrected posi-
0.5 Hz interval 3.1  0.2 2.3 0.10 0.06 0.09 0.14 tional offset of the reconstruction against the track is 0.143 m as
Missing calib. (1) 20.0 2.9 1.0 0.54 1.41 2.10 2.58 laid out in Table 2. Fig. 6 shows a graphical representation of such
Missing calib. (2) 1.9  4.4 10.7 0.55 0.24 0.51 0.79
offsets based on motion tracks and illustrates that, despite

[m] [°]
-0.2 0 0.2 0.4
-0.3 -0.1 0.1 0.3 0.5 -2 -1 0 1 2 3 4 5 6 7

2m RMS Error
Error X Error pitch
Error Y Error roll
Error Z Error heading

Fig. 6. Visualization of the deviation of reconstruction values against the navigational data. (a) Spatial deviation along the track path is plotted as a color coded spherical
marker with the radius of the resulting error around the assumed position. (b) Spatial deviation of the individual axes along the track including the resulting error vector.
(c) Orientation deviation along the three axes.
 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226 225

considerable uncertainties in the overall positioning, the local fit

among acoustical navigation and optical derivation of camera
poses can be very good. The concentration of positional misfit c f
(Fig. 6a) around turning points throughout the sequence suggests g
that high frequency movement has been suppressed during the
track synthesis or that the DVL sensor occasionally exaggerated
the amount of vehicle motion in cases of fast movement. Mean-
while, the resulting differences in camera orientation stay below
0.11. The overall accuracy of the reconstruction can be further
constrained by a comparison of the camera orientation data to the
original track, which yields a cumulative average deviation of 5.21
(Fig. 6c, Table 2).
A vital prerequisite for successful georeferencing is the
availability of the high frequency DVL signal. Referencing the
data only on the long wavelength component of the USBL
a
d
signal increases the positional misalignment and suggests a b
temporal misalignment of 3 s between ROV track and recon-
structed track towards a new positional alignment optimum
resulting in an ever increasing orientation imprecision (Table 2).
5m
In order to illustrate the importance of correct timing, we
compare the alignment at 6 s to the alignment at þ6 s, result-
ing in comparable deviations in orientation but leading to a
considerable degradation in positional accuracy. It should be e 5m
noted though that the degree of misalignment also depends on 5m
the length and complexity of the tracks to be fitted. It is advisable
to work with as large a model as possible in order to suppress Fig. 7. Warping effects due to missing lens distortion parameters
local disturbances. (a) superimposed on the correct reconstruction (b). Both models have been
aligned at the first camera pose (c), where deviations in the model geometry
The temporal resolution required is governed by the amount of
and position are already apparent. The largest dislocation (gray arrow) in position
overlap between the images and thus by the speed of the ROV and camera angle is found between the last images, (d) showing the warped path
over ground. We achieve good results for a 1-s interval at vehicle and (e) the correct path, deviating 291 in pitch, 81 in roll and 1.81 in heading.
speeds around 0.25–0.5 m/s, although a 2 or 3-s interval often still Crosses mark the location of a corresponding feature referenced in the text.
provides sufficient information. To investigate effects of varying Measurements of a corresponding bedding plane (white planes) indicate a strong
deviation in strike (671, lines) and dip (121, arrows). The light transparent model
data coverage on reconstruction quality and to define a minimum (f) and camera planes (g) illustrate the model, which has been aligned to the track
shooting interval, we attempted reconstruction from a subset of coordinates, resulting in positioning and also scaling errors. The white grid
photographs chosen at ever-increasing intervals. For our setup, represents the true horizontal plane.
we find a sampling frequency of 0.5 Hz (one image every 2 s) or
higher to be sufficient. The models created at 1 and 2-s sampling
intervals are essentially identical in their extent, resolution and
orientation (Table 2). Longer intervals create coverage gaps that 10. Lessons learned
cause the reconstruction to halt. Since the algorithm does not look
for unlinked clusters beyond the first one it finds, the remaining Having developed a workflow based on already existing data,
images must be identified and a reconstruction may be attempted we present here a number of observations and best practice rules
in a new project. Such unlinked snippets of the entire outcrop to be applied during data acquisition which, with minimal
must then be individually georeferenced, introducing additional additional effort when preparing and carrying out a ROV dive,
errors. can greatly improve the quality and quantity of subsequent
To illustrate the effect of an erroneous camera calibration we reconstructions.
removed the calibration data for the nonlinear lens distortion
parameters K1–K3 (Brown, 1966). This resulted in a warping of the 1. First and foremost, the quality of reconstructed models
scene along the x-axis expressed by a propagating deviation of the depends critically on the quality of the camera technology
camera tilt, the location of the point of view and a corresponding used to acquire the data. Using a high-resolution sensor with a
warping of the model around the x-axis and vertical stretching. low noise level is the critical factor, as poor image quality
Instead of using the usual rigid alignment to the known vehicle track, significantly contributes to erroneous matching of images.
we superimposed the initial (i.e. the topmost) camera poses of the The optical system should be as simple as possible, utilizing
‘‘correct‘‘ (i.e. with camera calibration) and warped reconstructions a lens with a very wide depth of focus or fixed focus balanced
to observe the propagation of drift throughout the model. Deviations with a small relative aperture.
developed as given in Table 2 resulting in an offset of 7.6 m in 2. Although an optically corrected ‘‘dome port‘‘ window for the
position ( 4.6 m in x, 5.1 m in y and  3.2 m in z) for a feature at the camera pressure housing is ideal, a minimal requirement is
bottom of the model. The largest offset is experienced for any that corrections can be made for optical aberrations due to
orientation measurements of faults and bedding planes towards the air/glass/water interface, requiring in situ calibration
the lower end of the model, which are effectively rendered useless as of the camera image using appropriate test objects (e.g. a
illustrated in Fig. 7. Once the usual rigid body alignment to the checkerboard).
vehicle track was carried out, the misfit was distributed throughout 3. While the wish to invest in multifunctional deep-diving camera
the model but should still be regarded as unacceptable (Table 2). equipment to satisfy a range of user requirements is under-
Moreover, a strong deviation in scale becomes apparent. This standable, it should be realized that the use of zoom optics
experiment underlines the vital importance of proper calibration of during photogrammetry is highly detrimental and can only be
intrinsic camera parameters. accommodated if focal length can be logged accurately, and the
226 T. Kwasnitschka et al. / Computers & Geosciences 52 (2013) 218–226

concomitant changes in optical aberration calculated, for every Acknowledgments

picture taken. Additionally the reconstruction software must be
able to incorporate these data into its model-generation pro- We would like to thank the Captain and crew of RV Meteor for
cess; otherwise zoom will be interpreted as motion closer to the their excellent assistance at sea, and the crew of ROV Kiel 6000 for
object, rendering track alignment impossible. their support during the development of such a specific applica-
4. Photogrammetry is a computationally very intensive process. tion, and for clarifying many engineering details. T. Kwasnitschka
Our project was processed on a workstation equipped with a would like to thank Anne Jordt for an introduction to underwater
twin Intel Xeon Processor, 12 GB RAM and one ATI Crossfire photogrammetry and many valuable comments. This work was
4800 Series graphics card (which is used with the GPU features carried out within the Jeddah Transect project (www.jeddah-tran
of Photoscan), which matched our HD video source in required sect.org).
performance. A crucial bottleneck was the amount of RAM
available. With this hardware, a 200,000 polygon reconstruc-
tion based on 1-s samples of 2 min of HD video required just Appendix A. Supporting information
over 4 h to process.
5. The camera should be held orthogonal to the objects, angles Supplementary data associated with this article can be found in
larger than 451 may lead to a failure in reconstruction or to the online version at http://dx.doi.org/10.1016/j.cageo.2012.10.008.
inaccuracies. Rapid movements (which lead either to blur- These data include Google map of the most important areas described
ring of the images or inaccuracies in the vehicles position in this article.
and attitude determinations, possibly as a result of timing
errors) should be avoided. An ideal platform for photogram-
References
metric survey in the deep sea would move on a continuous
gridded track (‘‘mowing the lawn‘‘). Tracks should cross each
Beall, C., Lawrence, B.J., Ila, V., Dellaert, F., 2010. 3D reconstruction of underwater
other frequently or ideally run parallel with considerable structures. In: Proceedings of Intelligent Robots and Systems (IROS), 2010
overlap. IEEE/RSJ International Conference, pp.4418–4423, http://dx.doi.org/10.1109/
IROS.2010.5649213.
Brown, D.C., 1966. Decentering distortion of lenses. Photogrammetric Engineering
32 (3), 444–462.
Finally, it should be noted that our selection of software to pre- Condell, J., Moore, G., Moore, J., 2006. Software and methods for motion capture
process and interpret the data is to be regarded as preliminary. and tracking in animation. In: Arabnia, H.R. (Ed.), Proceedings of 2006
Originating from the entertainment industry, many programs International Conference on Computer Graphics & Virtual Reality (CGVR’06),
Las Vegas, pp. 3–9.
are barely able to cope with the amount of data generated at Gaich, A., Fasching, A., Fuchs, R., Schubert, W., 2003. Structural rock mass
sea. The long duration of ROV dives requires workarounds in parameters recorded by a computer vision system. In: Culligan, P.J., Einstein,
video editing, and 3D animation packages also require a rescaling H.H., Whittle, A.J. (Eds.), Proceedings of Soil and Rock America 2003, 12th
Panamerican Conference on Soil Mechanics and Geotechnical Engineering,
of geographic coordinates and time (e.g. minutes expressed as
Cambridge, MA, pp. 87–94.
seconds to fit a limited timeline). Johnson-Roberson, M., Pizarro, O., Williams, S.B., Mahon, I., 2010. Generation and
visualization of large-scale three-dimensional reconstructions from under-
water robotic surveys. Journal of Field Robotics 27 (1), 21–51.
Jones, A.M., Cantin, N.E., Berkelmans, R., Sinclair, B., Negri, A.P., 2008. A 3D
modeling method to calculate the surface areas of coral branches. Coral Reefs,
27, pp. 521–526, http://dx.doi.org/10.1007/s00338-008-0354-y.
11. Conclusion Kreylos, O., Bernardin, T., Billen, M.I., Cowgill, E.S., Gold, R.D., Hamann, B., Jadamec,
M., Kellogg, L.H., Staadt, O.G., Sumner, D.Y., 2006. Enabling scientific work-
We present a review of available software and a practical flows in virtual reality. In: Proceedings of the 2006 ACM international
conference on Virtual reality continuum and its applications, ACM, New York,
workflow to virtually replicate morphological, geological and
pp. 155–162, http://dx.doi.org/10.1145/1128923.1128948.
biological features of the seafloor accurately enough to conduct Kwasnitschka, T., 2008, Volcanic and Tectonic Development of the Ilopango
scientific studies. Our aim was to create synthetic model visua- Caldera, El Sal-vador: stratigraphic correlation and visualization of emplace-
lizations of the seafloor to provide more geologically useful and ment. Unpublished Diploma Thesis, Christan-Albrechts-Universität zu Kiel,
Kiel, 218pp.
quantitative information than individual video frames. Although Mach, R., Petschek, P., 2007. Visualization of Digital Terrain and Landscape Data.
we recognize that technical improvements can be made in order Springer, New York 374pp.
to further substantiate and calibrate the interpretations inferred Pizarro, O., Eustice, R., 2009. Large area 3-d reconstructions from underwater
optical surveys. IEEE Journal of Oceanic Engineering 34 (2), 150–169.
from our surveying method, we have demonstrated that a Sedlazeck, A., Köser, K., Koch, R., 2009. 3D reconstruction based on underwater
georeferenced reconstruction of the features imaged during a video from ROV Kiel 6000 considering underwater imaging conditions. In:
video transect is (a) possible and (b) yields quantitative geological Proceedings of IEEE OCEANS 2009, Bremen, pp. 1–10.
Sedlazeck, A., Koch, R., 2011. Calibration of housing parameters for underwater
information not otherwise accessible. The example presented stereo-camera rigs. In: Hoey, J., McKenna, S., Trucco, E. (Eds.), Proceedings of
shows the potential of quantitative geoscientific measurements the British Machine Vision Conference, BMVA Press, pp.118.1–118.11.
and fieldwork on the deep ocean floor provided there is sufficient Snavely, N., Simon, I., Goesele, M., Szeliski, R., Seitz, S.M., 2010. Scene reconstruc-
tion and visualization from community photo collections. Proceedings of the
coverage by adequate surveying. Similar benefits can be expected IEEE 98 (8), 1370–1390.
for biological habitat mapping as well as for a wide variety of Triggs, B., McLauchlan, P.F., Hartley, R.I., Fitzgibbon, A.W., 2000. Bundle adjust-
industrial seabed monitoring applications. At the same time, our ment – a modern synthesis. In: Vision Algorithms: Theory and Practice,
Lecture Notes in Computer Science, Springer, Berlin/Heidelberg, 1883,
method is capable of processing archived data meeting minimal
pp. 153–177.
standards in order to re-evaluate remote targets or to monitor Verhoeven, G., 2011. Taking computer vision aloft – archaeological threedimen-
changes at sites of high temporal variability such as hydrothermal sional reconstructions from aerial photographs with Photoscan. Archaeological
vents. In addition, the method provides a means to re-cast ROV Prospection 18, 67–73.
Yoerger, D., Bradley, A., Jakuba, M., German, C.R., Shank, T., Tivey, M., 2007.
video data in a geographical rather than a time-line reference Autonomous and remotely operated vehicle technology for hydrothermal vent
frame. discovery, exploration, and sampling. Oceanography 20 (1), 152–161.