International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012

XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

RIGOROUS IMAGE FORMATION FROM AIRBORNE AND SPACEBORNE DIGITAL

ARRAY SCANNERS

H. J. Theiss

InnoVision Basic and Applied Research Office, Sensor Geopositioning Center, National Geospatial-Intelligence
Agency (contractor) PA Case 12-189, 7500 GEOINT Dr, Springfield, VA 22150, USA – [email protected]

Commission I, WG 3

KEY WORDS: geometric, modelling, optical, orientation, processing, rectification, restitution, scanner

ABSTRACT:

Sensor builders in the digital era have design limitations due to the constraint of maximum available digital array size. A
straightforward solution exists, for example, when four cameras that each simultaneously captures an image from essentially the
same perspective centre; they can be re-sampled to form a virtual large format image that can be exploited using a single (instead of
four separate) instantiation of a frame model. The purpose of this paper is to address the less trivial time-dependent cases where the
sensor scans the ground and the detector arrays obtain chips of imagery that need to be stitched together to form a single
conveniently exploitable image. Many operational techniques warp the imagery to form a mosaic, or ortho-rectify it using an
imperfect digital surface model (DSM), thus eliminating the possibility for accurate geolocation and uncertainty estimation. This
algorithm, however, forms a single virtual image with associated smooth metadata, which can be exploited using a simple physical
sensor model. The algorithm consists of four main steps: 1) automated tie point matching; 2) camera calibration (once per sensor);
3) block adjustment; and 4) pixel re-sampling based on an “idealized” virtual model. The same geometry model used to form the
image, or its true replacement, must be used to exploit it. This paper verifies the algorithm using real imagery acquired from the
Global Hawk (GH) UAV. Registration of the virtual image to a WorldView1 stereopair using four tie points yielded an RMS below
0.6 meters per horizontal axis.

1. INTRODUCTION obtaining 14 cross-track scans of 10 frames each. The

algorithm forms a single virtual image with associated smooth
1.1 Problem Statement metadata, which can be exploited using a simple generic
whiskbroom sensor model. Thorough metric analyses have
Image acquisition in the era of film recording fell primarily into been performed on these virtual image products to demonstrate
three main classes: frame, strip, and panoramic. For each of no loss of precision in the derived coordinates relative to the
these classes the recorded image was identical to the image reference imagery, i.e. a stereo pair of WorldView1 images with
exploited by the user since no resampling of pixels was required 0.5 meter GSD. The motivation for implementing this new
to form the image. Hence, the sensor model used to perform image formation approach is that many operational techniques
photogrammetric operations corresponded to physical warp the imagery to form a mosaic thus eliminating the
parameters associated with the original recorded image. As we possibility for accurate geolocation and uncertainty estimation.
entered into the digital age, sensor builders encountered The only remaining option for rigorous exploitation had been to
challenges due to the maximum size digital array that could be exploit the 140 frames independently, but users had found it too
manufactured. An example of how this limitation was cumbersome.
overcome is the DMC® from Z/I Imaging® which
simultaneously acquires four high resolution panchromatic The algorithm, which can be applied to most any time
images from essentially the same perspective centre, and re- dependent imaging system besides GH, consists of four main
samples the pixels to form a single virtual large format image. steps: 1) obtain tie point correspondences in the narrow
Photogrammetric exploitation of the resulting image is forward- and side-lapping regions of the frames; 2) perform
relatively simple in that a single frame sensor model is used camera calibration; 3) perform block adjustment; and 4) re-
instead of four separate instantiations of a frame model. The sample the pixels to obtain a single virtual image with
purpose of this paper is to address the less trivial time- associated smoothed metadata. The first step, tie point
dependent cases, such as pushbroom or whiskbroom, where the matching, is performed using normalized cross correlation. The
sensor scans the ground and the detector arrays obtain chips of camera calibration is performed only once for the sensor, and
imagery that need to be stitched together to form the image. the recovered values are used to form subsequent images. The
block adjustment solves for corrections to roll, pitch, and yaw
1.2 Overview of Approach angles for each frame, and a single set of three translations for
the entire block. Tie points are free to move in the two
This paper provides an algorithm and photogrammetric directions perpendicular to the line of sight, while they are
modelling results for forming a single image, which can be constrained in the line-of-sight direction by an amount that is a
exploited using a simple generic physical sensor model, given function of the convergence angle between the rays. A post-
several image chips collected over time. One data set used to adjustment range map across the scene is generated as a
verify the algorithm is real imagery acquired from the Global function of the tie point 3D locations for use in the next step.
Hawk (GH) UAV. It scans a square digital frame array Prior to performing the actual re-sampling, the position and

159
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

attitude data are smoothed by fitting low order polynomials. To

form the final virtual image, the algorithm requires that the
following steps be performed per pixel in the virtual image.
First, use the generic whiskbroom sensor model associated with
the virtual image, its smoothed metadata, and the range map to
perform an image-to-ground (i2g) transformation. Given this
ground point, perform a ground-to-image transformation into
the nearest frame using the frame sensor model and its
associated interior orientation parameters obtained by the
camera calibration. Finally, bi-linearly interpolate to obtain the
gray value to assign to the virtual image pixel.

In order to exploit the virtual image, the same geometry model

must be used that was employed to perform the i2g step of re- Figure 1. Image collection scene from the GH UAV
sampling associated with the image formation. After registering
the virtual image to a WorldView1 stereopair using 4 tie points,
comparison with over 140 check points demonstrated an RMS 3. AUTOMATED TIE POINT MATCHING
below 0.6 meters per horizontal axis which was essentially
equivalent to the accuracy of the WorldView1 derived check The purpose of the automated tie point matching step is to find
points with respect to a local datum. Notably, the rigorous at least one tie point in each of the overlap regions between
image formation approach provides over an order of magnitude adjacent image chips. Imaging systems that scan to collect
accuracy improvement compared to the operational image staggered arrays of chips, or use a whiskbroom collection to
warping approach. sweep an image chip, collect overlapping pixels that look very
similar compared to an imaging system that collects a stereo
1.3 Paper Outline pair with a wide convergence angle. In our case with scanners,
the time difference between exposures for the pair of image
Following this introduction section, the paper proceeds with chips is small enough that perspective differences, and therefore
Section 2, a description of the real airborne data set used to relative relief displacement effects, do not pose a challenge to
demonstrate the success of this image formation algorithm. the automated matcher. Likewise, scale, rotation, and
Then Section 3 provides a high level overview of the automated radiometric differences are negligible. Furthermore, the
tie point matching. Sections 4 and 5 discuss the camera metadata errors over the short time between subsequent image
calibration and bundle adjustment, respectively. Section 6 chip collections are highly correlated; while their absolute errors
describes the steps involved in the image formation, and Section may be significant, the relative error between the two times is
7 provides results using the real airborne data set. Conclusions likely small. For all of these reasons an area-based matching
are provided in Section 8. technique, such as normalized cross correlation, is quite
effective. Problems arise due to cloud cover or areas of non-
distinct features such as would occur in the water, forest, or
2. AIRBORNE DATA SET grassland. Automated tie point matching failures can be
accommodated by using an outlier detection and removal
Figure 1 illustrates the standard image collection scene from the technique, such as iterative re-weighting, in the bundle
Global Hawk UAV. A scene consists of 140 total frames adjustment step. The automated matching process was not the
collected in a serpentine pattern of 14 scans of 10 frames per main thrust of this research; instead, an industry partner
scan. It scans these 10 frames at 30 Hz, and requires 0.2 provided the capability in the form of an executable file.
seconds of turnaround time before scanning in the reverse
direction again at 30 Hz. The angular field of view (FOV) of a
frame is 0.3 degrees; hence a 10-frame scan has an FOV of 4. CAMERA CALIBRATION
approximately 3 degrees. A single frame contains roughly 1000
by 1000 pixels; therefore, a scene converted to a mosaic at full Camera calibration is a critical step to ensure that features align
resolution would contain roughly 10,000 by 14,000 pixels. The along all borders of all chips. Performing a bundle adjustment
UAV is typically flown at approximately 16,800 meters above with added parameters, each time an image is formed, is
ground level and 45 degrees off-nadir, predominantly roll about possible. However, we instead recommend performing the
the flight direction with only a few degrees of pitch. The camera calibration once, and using the recovered interior
nominal ground sample distance (GSD) was approximately 0.11 orientation parameter values as constants in the bundle
and 0.16 meters in the cross-range and range directions, adjustment step. The next two sub-sections define the approach
respectively. and parameter recovery, respectively.

4.1 Approach

The ideal configuration of airborne or spaceborne imagery to

support camera calibration is to obtain a large number of
overlapping images from geometries with large convergence
angles and various rotations of the image chips about the optical
axis. So it is a major advantage if the photogrammetrist has the
ability to task the system that needs to be calibrated. However,
typically imagery from these scanning systems is available only
in its “mapping” mode that tries to efficiently sweep up as much
ground coverage as possible by minimizing image chip overlap.

160
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

Another required input is reasonable values for the exterior

We did not have the ability to task the imagery collected from orientation (EO) parameters associated with each frame.
our airborne whiskbroom data set; therefore, we had to take full Sometimes they are not provided explicitly in the metadata of
advantage of our image chip overlap areas. Specifically, we the imagery and need to be estimated using geometric
manually measured tie points not only in the overlap areas techniques. One example was in our specific data set in which
between adjacent chips but also measured all possible four-ray the coordinates of the camera perspective centre were reliable
tie points. Thus we used many more and better tie points to while the camera attitude was not. Consequently, we used the
perform the one-time calibration than we used in the bundle latitude, longitude, and height coordinates associated with the
adjustment step that we would run for each image formation four image corners to back out the three Euler angles describing
instance. the camera attitude. In another data set, not provided in this
paper, the only metadata available with the images was cubic
4.2 Recovery of Parameters rational polynomial function coefficients which provide a
means to perform image-to-ground and ground-to-image
Satellite imagery camera calibration of the staggered array transformations. To solve for the perspective centre of the
typically consists of a laboratory calibration, checked with real camera, we generated 3D lines in object space that represent
in-situ data after launch, to obtain precise field angle mappings imaging loci and then used least squares adjustment to find the
for the ends of each image chip. However, for our airborne best estimate of the intersection point where the imaging system
whiskbroom case, the camera calibration algorithm is a bundle was located during acquisition. The endpoints of lines defining
adjustment with added parameters. Hence, the algorithm is the the imaging loci were obtained by simply running the image-to-
same as provided in Section 5 with the exception that a single ground function at a nominal height and then again at a new
set of up to 10 interior orientation parameters (IO) are recovered height, and then generating a line through these points.
for the whole block of images. The following equations are
used to calculate x and y corrections to the image points as a An optional input to the algorithm is a digital surface model
function of the IO parameters (Fraser, 1997). (DSM); not having one available just means that ranges or
terrain heights need to be estimated during the algorithm as
x discussed in the next sub-section.
x  c  x r 2 K1  x r 4 K 2  x r 6 K 3  (2 x 2  r 2 ) P1  2 P2 x y  b1 x  b2 y
c
y (1)
y  c  yr 2 K1  yr 4 K 2  yr 6 K 3  2 P1 x y  (2 y 2  r 2 ) P2 Another optional input is error covariance data for all of the
c
x  xb  xo aforementioned inputs. Availability of this uncertainty
y  yb  yo information has many advantages. First, it means that the a
r2  x2  y2 posteriori error covariance matrices, which are an automatic by-
product of the bundle adjustment, can be used to generate valid
in which: a priori error covariance matrices describing the idealized
xb,yb are the observed image coordinates in the image metadata associated with the output image product. Second, it
coordinate system, provides correct relative weighting among all input
xo,yo,c the principal point offsets with respect to the image measurements, thereby resulting in an optimal least squares
coordinate system, (xo,yo), and the camera principal distance, adjustment solution. Finally, assigning reasonable
c principal distance correction parameter, uncertainties, instead of excessively large ones, to the adjustable
parameters will prevent image warping and will usually make
K1,K2,K3 three radial lens distortion parameters,
outlier detection procedures more effective.
P1, P2 two decentering lens distortion parameters, and
b1, b2 two in-plane distortion parameters (scale differential
and skew). 5.2 Algorithm

The bundle adjustment follows closely to what can be found in

Results of a camera calibration for a real airborne whiskbroom photogrammetry text books (Mikhail, et al., 2001). The main
imagery data set are provided in Section 7.2.
difference between our adjustment and a standard one is that the
pairs of imaging rays that determine a tie point have a very
weak convergence angle. In order to eliminate instabilities in
5. BUNDLE ADJUSTMENT the solution, one out of the three coordinate components of the
tie point must be constrained. The first choice, if a DSM is
The purpose of the bundle (aka block) adjustment is to refine
available, is to constrain the Z (height) component such that the
the values of the adjustable parameters, which are corrections to
tie point lies on the DSM. The second choice is to judiciously
position and attitude of the sensor, associated with each image
apply some a priori weight to one component of the tie point.
chip. The adjustment serves to globally minimize the sum of
Instead of choosing between the X (East), Y (North), and Z
the squared weighted tie point image coordinate measurement
(Up) components, we chose to constrain the W component of tie
residuals, thereby preparing the data for image re-sampling that
points in the UVW coordinate system in which the W axis
will result in good alignment across the image chip boundaries.
aligns with the nominal line-of-sight direction, the U axis aligns
The next three sub-sections describe the algorithm, and its
nominally with the vehicle velocity vector, and the V axis
inputs and outputs, for the airborne whiskbroom case.
completes a right-handed coordinate system. In our real data,
the sensor scanned cross-track so the time difference was
5.1 Inputs
approximately 1/30th second and ½ second between adjacent
Obvious inputs to the block adjustment, following from the frames in the cross-track and along-track directions,
discussion in the last two sections, are the image coordinates of respectively. Therefore, we allowed more movement (less
tie point measurements and the IO values of the camera weight) on the along-track tie points compared to the cross-
calibration. track ones which were held essentially fixed in the W direction.

161
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

5.3 Outputs 6.1.2 Virtual Geometries from Airborne Frame: Using

the 140 frames of the GH UAV as an example, the two extreme
The two main outputs of the bundle adjustment are refined cases to output an image product from a ground station are to
values for the EO parameters of each frame, and a grid of provide all of the original frame images, or to generate a single
interpolated slant ranges from the sensor to the ground. This ortho-mosaic from all of the frames. The former case would
grid of interpolated ranges is derived as a function of the allow for the most rigorous exploitation of the imagery, but the
recovered EO parameters and tie point ground coordinate least convenient. The latter case, however, would result in the
values. The utility of the grid of ranges, aka range image, will least rigorous exploitation of the imagery but would be the most
become apparent in Section 6.3. convenient product to someone who wants it inherently co-
registered with other map-projected products and does not want
to use a sensor model. The remainder of this sub-section
6. IMAGE FORMATION discusses three example virtual image geometry options; and the
preferred method for this airborne data set, i.e. whiskbroom,
The process of image formation requires first making a decision was used to generate the results in Section 7.
as to what the ideal, or virtual, image geometry will be for the
desired output image product. Then, it requires assignment of
6.1.2.1 Panoramic Geometry: The most rigorous virtual
ideal IO and EO parameters to the virtual image geometry.
image geometry model would be to model the scene as 14
These steps, followed by the pixel resampling, are the topics of
panoramic scans; therefore, the ground station would build 14
the next three sub-sections.
output image products, instead of 140 original frame images.
Figure 3 illustrates the panoramic geometry for a single 10-
6.1 Defining the Virtual Geometry
image scan. Recall, from Section 1, that the GH UAV captures
As mentioned in the introduction, most vendors build mosaics these 10 frames in one-third of a second. So, if the vehicle is
by stitching the image chips together using 2D-to-2D travelling at 60 meters/second, then it has only travelled 20
transformations; thus, the formed image retains no mechanism meters between the first and tenth frames, i.e. plus-or-minus 10
to perform image-to-ground or ground-to-image mappings. A meters worst case deviation from the virtual instantaneous
virtual geometry model, however, consists of parameters that perspective centre.
allow the calculation of a perspective centre location and line-
of-sight (LOS) vector associated with any pixel in the output
image. The perspective centre location is calculated from the
position components of EO, while the LOS vector is calculated
from the attitude components of EO in addition to the IO
parameters. Due to the nature of forming an output image using
a virtual geometry, per Section 6.3, it is obvious that it perfectly
replicates the original image geometry when the point of
interest lies at the same elevation (or slant range from the
sensor) that was assumed during image formation. As the error
in elevation (or slant range) increases, the geolocation error
increases. The sensitivity of geolocation errors to elevation (or
slant range) errors is a function of how closely the virtual
imaging geometry matches the actual imaging geometry.

6.1.1 Spaceborne Synthetic Linear Array Geometry: The

Figure 3. Panoramic geometry applied to 10-image scan from a
choice of virtual geometry to use for the output image can be
GH UAV
obvious, e.g. for a spaceborne staggered array scanner, or can
require good engineering judgment as in the case for the GH
UAV real data set described in Section 2. In the spaceborne
staggered array case, the virtual image geometry is a linear 6.1.2.2 Whiskbroom Geometry: The whiskbroom
array that lays half way between the leading and trailing rows of geometry is the most rigorous virtual model that can be applied
detectors that comprise the staggered arrays; see Figure 2. to the entire scene, as a single output product. A single output
product with a single sensor model is much more convenient
than 14 output products with 14 separate instantiations of a
panoramic model. If the 14 output products were concatenated
into a single product, then the far end of the FOV would have
overlaps while the near end would have gaps; see the left side of
Figure 4. In the whiskbroom geometry model, all pixels in a
given column of the output image correspond to a particular
sensor position and sensor attitude; see Figure 5. Consequently,
Figure 2. Staggered and synthetic arrays for a spaceborne the whiskbroom model removes the effect of unequal scales in
system the near and far ends of the FOV, unavoidable with a panoramic
model, by gradually compensating for the difference on an
unnoticeable pixel by pixel basis; see the right side of Figure 4.

162
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

Figure 4. An exaggerated view of footprints of three panoramic Figure 6. Smoothed attitude data
images (left side) and several columns of a whiskbroom image
(right side) It is important to note that smoothing the EO parameter data can
have significant advantages, as well as possible drawbacks,
when employed during the image formation process. The
following comments about smoothing apply equally to airborne
framing, pushbroom/whiskbroom, or spaceborne linear array
scanners. If the sensor had experienced uncommanded (but
recorded by the IMU, or recovered during bundle adjustment)
rolling during the image collection, e.g. due to air turbulence,
then straight lines would appear as wavy curves in an image
formed using this erratically varying attitude; however,
smoothing the metadata of the virtual (idealized) image would
have the effect of straightening the lines in the formed image.
Similarly, any imperfections in sensor IO, e.g. modelled lens
distortions or chip misalignments, need not be included in the
virtual (idealized) imaging geometry since it incorporates
Figure 5. Whiskbroom geometry applied to a GH UAV scene unnecessary complication into the virtual geometry model
which will ultimately need to be implemented by a downstream
exploitation tool. Even substantial smoothing of the attitude
data, or simplification of the IO parameters, will retain the
6.1.2.3 Frame Geometry: The frame geometry is a geometric integrity of the output image, albeit with changes to
simpler but less rigorous virtual model that can be applied to the how scale varies in the output image compared to the original
entire scene. It was deemed inappropriate for the GH UAV real image chips. (The reason why geometric integrity is retained,
data set due to the relatively long distance travelled between the as explained in Section 6.3, is that the re-sampling process
first and last images collected in a scene. The virtual frame requires the unsmoothed attitude and imperfect IO parameters
model may be appropriate from a platform at longer slant when performing the ground-to-image step.) Caution must be
ranges, for smaller scenes, or for faster collections. The virtual exercised, however, when smoothing perspective centre
frame model is appropriate for systems such as wide area locations since this simplification will introduce errors into
airborne surveillance (WAAS), whereby all frame images are subsequent geolocation processes as a function of the amount of
acquired at the same time from the same platform. error in the DSM (or range image) that was used in the re-
sampling process. Finally, note that while smoothing the
6.2 Generating the Idealized Metadata attitude data of the virtual image geometry will have the
desirable effect of making object straight lines appear as straight
Section 6.1 presented three different options for the virtual
lines in the image (even though they appear as wavy curves in
image geometry; and the whiskbroom geometry was chosen for
the original image), it will result in the artefact that the straight
the GH UAV data. All pixels in a given column are modelled
edges marking the extent of the original images will appear as
as though they were imaged at the same time instant; hence our
wavy edges in the output image product.
desire is to have a polynomial function that yields each EO
parameter as a function of column number. Figure 6 shows the
values of the attitude parameters recovered from the bundle 6.3 Re-sampling the Pixels
adjustment (aka triangulation), plotted as the blue curves. The Once the geometry of the virtual (idealized) image has been
red curve illustrates the result of fitting a second order defined and the associated IO and EO parameters have been
polynomial to the average value for the scan, i.e. the midpoint determined, the remaining step of re-sampling the pixels is
between the value at the fifth and sixth frames. The same relatively straightforward. First, it is recommended to add a
polynomial fitting strategy was applied to the perspective centre buffer of black pixels around all four edges of the output image
positions. Unlike the case for the attitude parameters, these product. (It is a simple procedure to crop the image if a
position parameters remained smooth throughout the duration of customer does not want to see the wavy or jagged edges in the
each scan. final output image.) For each pixel (row, column) in the output
image, the following steps are performed:
1. Image-to-ground in the virtual image. Calculate time
as a function of row and column, obtain the values of
all EO parameters as a function of time (from the

163
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

potentially smoothed functions), and then project the

ray to the ground using either the DSM (if available as
an input to the algorithm) or the interpolated range
image (obtained as a by-product of the bundle
adjustment) to provide the third dimension.
2. Ground-to-image (g2i) in the original image chip.
Perform the g2i function into the original image chip
using the detailed IO parameters obtained from
camera calibration, and the EO parameters
(unsmoothed) obtained from the bundle adjustment.
If the point exists in multiple original image chips,
then choose the one whose image coordinates lie
farthest from the edge. If the point does not exist in
any of the original image chips, then in the next step
assign the pixel intensity as black.
3. Obtain the pixel intensity to place in the output image
product row and column location. Use bi-linear
interpolation from the four nearest pixels in the Figure 8. GH UAV output image product, Example 1
original image chip to obtain an intensity value.
Nearest-neighbour or other interpolation techniques
may be chosen, instead of bi-linear, to meet the
desires of the user.

7. RESULTS

7.1 Camera Calibration

Section 4 described how camera calibration was performed on

the GH UAV dataset. Figure 7 shows the image coordinate
residuals after running the self calibration with no IO
parameters (left side), compared to the residuals after running
self calibration with focal length and de-centering lens
distortions (right side). The blue arrows are the image
coordinate residuals, while the red arrows are simply the signed
mean of the cluster of blue arrows. The blue arrows
consistently fall in four quadrants of a frame due to the design
of the automated tie point finder mentioned in Section 3. The
red arrows were intentionally shifted away from the blue arrows Figure 9. GH UAV output image product, Example 2
to improve readability. Note that camera calibration had the
effect of reducing the systematic error from 2.4 to 0.1 pixels.

Figure 7. Image coordinate residuals at the 4 corners of each

frame, before calibration (left) and after calibration (right)

Section 7.2 provides a comparison of output images produced

Figure 10. GH UAV - zoomed in version of Example 2. Notice
using a calibrated versus uncalibrated camera.
undesirable shift in linear features in Uncalibrated scene.
7.2 Image Formation
7.3 Registration
While the algorithm documented in this paper was run on
As discussed at the end of Section 1.2, exploitation of the final
several data sets, this section provides two image product
output image product is performed by implementing the same
examples; see Figures 8 and 9. Since the figures had to be
virtual image geometry model, i.e. whiskbroom, which was
reduced in resolution to fit them in the paper, Figure 10
used to form the image. The output image can be registered to a
provides a zoomed in view of the Figure 9 image.
reference base, i.e. a controlled stereopair, by incorporating
adjustable parameters into the virtual whiskbroom model and

164
 International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, Volume XXXIX-B1, 2012
XXII ISPRS Congress, 25 August – 01 September 2012, Melbourne, Australia

performing a photogrammetric resection using points extracted thus indicating that good internal relative geometry has been
from the stereopair as ground control points (GCPs). preserved during the image formation process. The a posteriori
reference variance values, provided in the last column of Table
The adjustable parameters consisted of one interior orientation 1, are close to unity therefore indicating that the one-sigma
parameter, i.e. focal length, and coefficients of a polynomial uncertainties of 1 pixel and 0.5 meters for image coordinate
function of time used to compute small corrections to the roll, measurement and GCP uncertainty, respectively, were
pitch, and yaw angles about the x, y, and z sensor coordinate appropriate. They also show that the adjustable parameter of
system axes, respectively, i.e. the green axes in Figure 5. The second power for pitch is significant, but not roll.
polynomial functions are:

  a 0  a1t  a 2 t 2 (2)
  b0  b1t  b2 t 2
  c 0

in which:
,  ,  are the roll, pitch, and yaw angles, respectively,
a 0 , b0 , c 0 are constant terms of the polynomials,
a1 , b1 are first order terms of the polynomials,
a 2 , b2 are second order terms of the polynomials, and
t is the normalized time associated with the column
number (sample) of the coordinate of interest in the formed
image; thus t ranges smoothly from -0.5 to +0.5 from the first to
the last column of the image. Table 1. GH UAV – resection results

One hundred and fifty conjugate points were manually

measured on the output image and on a WorldView1 stereopair 8. CONCLUSIONS
that had a GSD of 0.5 meters. The output image had roughly
the same GSD as the input image frames whose geometry was This paper provided the image formation concepts and steps to
described in Section 2. Figure 11 shows the image coordinate build products from airborne and spaceborne digital array
residuals when 150 points are used to control the resection and scanners, i.e. automated tie point matching, bundle adjustment,
the same 150 points are evaluated as check points. It refers to defining the virtual (idealized) image geometry, and re-
the baseline case of six adjustable parameters, whereby the sampling the pixels. It emphasized the importance of
second order terms in equation 2 were omitted from the performing camera calibration prior to using the resulting
adjustment. While 150 GCPs are not practical, the example was parameters to form image products. This photogrammetric
run in order to show what relative deformations exist in the approach to image formation was verified using real GH UAV
output image. Note how closely that the mean theoretical frames to generate a mosaic product that could be rigorously
ellipse matches the empirical ellipse. The former ellipse was exploited using its sensor model to perform processes such as
derived by performing error covariance propagation on the resection, triangulation, and geolocation with error propagation.
single-image geolocation (to compute X and Y) for each ground
point, using the GCP’s Z value as input, and then averaging all While numerous vendors, e.g. commercial satellite optical
of the horizontal error covariance matrices. image data providers, already know how to form images, this
paper highlighted a few new concepts to some readers:
1) Although the sensor model associated with the raw
image may be complex in that it contains many
optical or focal plane distortions, the virtual geometry
model can designed to be quite simple;
2) Although the time history of adjusted attitude data
associated with the raw images may be rough, e.g. due
to air turbulence, the metadata of the virtual geometry
model can be designed with smooth variation; and
3) These virtual geometry models, and their associated
metadata elements, can be generalized such that
multiple vendors use the same models, therefore
allowing users to exploit imagery from different
vendors using the same generic sensor model.
Figure 11. Ground and image coordinate residuals after
resection with 6 adjustable parameters
9. REFERENCES
Table 1 shows the values of the semi-major (“a”) and semi-
Fraser, C. S., 1997. Digital camera self-calibration, Invited
minor (“b”) axes of the empirically calculated check point error
Review Paper, ISPRS Journal of Photogrammetry & Remote
ellipses for the 150 GCP case as well as practical cases with
Sensing, Vol. 52, pp. 149-159.
only 4 or 5 GCPs. In these latter cases, the check points
consisted of only the points that were not used as GCPs. The
Mikhail, E., J. Bethel, J. McGlone, 2001. Introduction to
reduction in number of control points, from 150 to a practical
Modern Photogrammetry. John Wiley and Sons, Inc., pp. 130-
number of 4, did not cause the accuracy to worsen significantly,
146.

165