Image Warping and Morphing: Olivier Gondry's Visit
Image Warping and Morphing: Olivier Gondry's Visit
Image Warping and Morphing: Olivier Gondry's Visit
Averaging images
Cross-fading Pretty much the compositing equation C= F +(1-) B
Averaging vectors
V= P + (1-) Q
P V Q
Morphing
Input: two images I0 and IN Expected output: image sequence Ii, with i 1..N-1
Morphing
For each intermediate frame It Interpolate feature locations Pti= (1- t) P0i + t P1i Perform two warps: one for I0, one for I1
Deduce a dense warp field from the pairs of features Warp the pixels
Linearly interpolate the two warped images User specifies sparse correspondences on the images Pairs of vectors {(P0j, PNj)}
Warping
D'Arcy Thompson
http://www-groups.dcs.st-and.ac.uk/~history/Miscellaneous/darcy.html http://en.wikipedia.org/wiki/D'Arcy_Thompson
Warping
Imagine your image is made of rubber warp the rubber
Triangular Mesh
1. Input correspondences at key feature points 2. Define a triangular mesh over the points Same mesh in both images! Now we have triangle-to-triangle correspondences 3. Warp each triangle separately from source to destination
Slide Alyosha Efros
Warp as interpolation
We are looking for a warping field A function that given a 2D point, returns a warped 2D point We have a sparse number of correspondences These specify values of the warping field This is an interpolation problem Given sparse data, find smooth function
Folding problems
Interpolation in 1D
We are looking for a function f We have N data points: xi, yi Scattered: spacing between xi is non-uniform We want f so that For each i, f(xi)=yi f is smooth Depending on notion of smoothness, different f
Kernel
Many choices In Assignment 4, we simply use inverse multiquadric
where c controls falloff Lazy way: set c to an arbitrary constant (pset 4) Smarter way: c is different for each kernel. For each xi, set c as the squared distance to the closest other xj
Enforcing interpolation
f (z) = i R(z, xi) N equations for each j, f(xj) = yj i R(xj, xi) = yj N unknowns i Just inverse the matrix
Important note
f (z) = i R(z, xi) for each j, i R(xj, xi)=yj Note that the influence of each function is non-zero everywhere at a data point, the value of the other bases is not zero In contrast to e.g. various interpolation splines
Variations of RBF
Lots of possible kernels 2 Gaussians e-r /2 Thin-plate splines r2 log r Sometimes add a global polynomial term
QUESTION?
Example
Example
Fold problems Oh well
1D equivalent of folds
There is no guarantee that our 1D RBF is monotonic Yes, it means that the notion of inverse of the warp is questionable.
input
Morphing
Input images
Feature correspondences
C= (1-t)f0t(I0)+t f1t(I1)
Recap
For each intermediate frame It Interpolate feature locations yti= (1- t) x0i + t x1i Perform two warps: one for I0, one for I1
Deduce a dense warp field from the pairs of features Warp the pixels
Movie time
Resampling
Parts are magnified Parts are minified Sometimes anisotropic Same problem for 3D texture mapping
Intuition
Plain lookup is bad
(But good news: that's all we ask for pset 4)
In magnified regions, not smooth enough In minified regions, it creates aliasing What we want In magnified regions, smooth interpolation In minified regions, take the average We need good signal processing framework to do this
point sampling
The Bible
http://www-2.cs.cmu.edu/~ph/texfund/texfund.pdf
Resampling
2D texture space
2D screen space
Notations
Input signal f(u) Forward mapping (texture-to-screen) x=m(u) Output signal g(x)
Warning: I sloppily changed my notations: f is signal, warp is m
Resampling
What do we need to do?
m f(u)
Resampling
1. Reconstruct the continuous signal from the discrete input signal 2. Warp the domain of the continuous signal 3. Prefilter the warped continuous signal 4. Sample this signal
m f(u) g(x)
Resampling
Resampling: progression
Discrete input texture f(u) for integer u m Reconstructed input texture g(x) fc(u) = f(u) r(u) = f(k) r(u-k) Warped texture gc(x)=fc(m-1(x)) Band-limited output gc(x) = gc(x) h(x) = gc(t) h(x-t) dt Discrete output g(x) = g(x) i(x)
f(u)
Resampling
source space resampling with prefiltering [Heckbert 89] destination space
reconstructed input
warped input
continuous output
Put it together
Discrete input texture f(u) for integer u Reconstructed input texture fc(u) = f(u) r(u) = f(k) r(u-k) Warped texture gc(x)=fc(m-1(x)) Band-limited output gc(x)=gc(x) h(x) = gc(t) h(x-t) dt Discrete output g(x) = g(x) i(x)
Put it together
Discrete input texture f(u) for integer u Reconstructed input texture fc(u) = f(u) r(u) = f(k) r(u-k) Warped texture gc(x)=fc(m-1(x)) Band-limited output gc(x)=gc(x) h(x) = gc(t) h(x-t) dt Discrete output g(x) = g(x) i(x)
g(x)
= gc(x) = gc(t) h(x-t) dt = fc(m-1(t)) h(x-t) dt = h(x-t) f(k) r(m-1(t)-k) dt = f(k) (x, k)
g(x)
= gc(x) = gc(t) h(x-t) dt = fc(m-1(t)) h(x-t) dt = h(x-t) f(k) r(m-1(t)-k) dt = f(k) (x, k)
10
Resampling
g ( x) = f ( xi )ri (m 1 ( x)) h( x)
i
color
resampling filter
The image space resampling filter combines a warped reconstruction filter and a low-pass filter
Resampling
Source Space Destination Space
Resampling
Source Space Destination Space
sum of resampling filters
4. Sample
Destination Space
Destination Space
Destination Space
warped reconstruction kernel
Destination Space
resampling filters
3. Filter
low-pass filter
convolution
3. Filter
Resampling
Use local affine approximation of warp Elliptical Gaussian kernels [Heckbert 89] Closed under affine mappings and convolution
g ( x) = f ( xi )ri (m 1 ( x)) h( x)
i
~ 1 ( x)) h( x) g ( x) = ci ri (m i
i
resampling filter
The image space resampling filter combines a warped reconstruction filter and a low-pass filter
= ci Gi ( x)
i
4. Sample
2. Warp
2. Warp
11
Resampling filter
Depends on local warp For perspective, approximated by local affine at center of kernel Not bad approximation because filter small at periphery
Resampling Filter
warped reconstruction kernel low-pass filter resampling filter minification
magnification
EWA resampling
EWA
trilinear mipmapping
12
Uniform morphing
Non-uniform morphing
http://www-cs.ccny.cuny.edu/~wolberg/pub/cgi96.pdf
Video
Lots of manual work
View morphing
View morphing
Seitz & Dyer
http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm
13
Main trick
Prewarp with a homography to "prealign" images So that the two views are parallel Because linear interpolation works when views are parallel
morph morph
prewarp
prewarp
input homographies
output
input
Extensions
14
Shape Vector
S model = ai S i
i =1
Tmodel = bi Ti
i =1
If number of basis faces m is large enough to span the face subspace then:
Provides alignment!
43
Subpopulation means
Examples: Happy faces Young faces Asian faces Etc. Sunny days Rainy days Etc. Etc.
Average female
Appearance T
Average male
X X = X - X
+ = + 1.7
15
EigenFaces
First popular use of PCA on images was for modeling and recognition of faces [Kirby and Sirovich, 1990, Turk and Pentland, 1991] Collect a face ensemble Normalize for contrast, scale, & orientation. Remove backgrounds Apply PCA & choose the first N eigen-images that account for most of the variance of the mean data.
face lighting variation
Figure-centric averages
Jason Salavon
Antonio Torralba & Aude Oliva (2002) Averages: Hundreds of images containing a person are averaged to reveal regularities in the intensity patterns across all the images.
16
3D morphing
Feature-Based Volume Metamorphosis Lerios, Garfinkle, and Levoy.
http://www-graphics.stanford.edu/~tolis/toli/research/morph.html
Why blurry?
3D morphing
Feature-Based Volume Metamorphosis Lerios, Garfinkle, and Levoy.
http://www-graphics.stanford.edu/~tolis/toli/research/morph.html
Automatic morphing
http://ccc.inaoep.mx/~fuentes/zanella.pdf
Recap
Idea that linear interpolation introduces blur Separation of shape and color Idea of non-rigid alignment of different images Applications to medical data Applications, related to Special effects Face recognition Video frame interpolation MPEG Scattered data interpolation
17
Refs
References
http://portal.acm.org/citation.cfm?id=134003&coll= GUIDE&dl=GUIDE&CFID=72901489&CFTOKEN =24335444 http://www.visgraf.impa.br/cgibin/morphQuery.cgi?output=html http://www.cg.tuwien.ac.at/research/ca/mrm/ http://w3.impa.br/~morph/sites.html http://w3.impa.br/~morph/sig-course/slides.html http://www.owlnet.rice.edu/~elec539/Projects97/mor phjrks/morph.html http://www.fmrib.ox.ac.uk/~yongyue/thinplate.html http://www.uoguelph.ca/~mwirth/PHD_Chapter4.pd f
Software
http://www.morpheussoftware.net/ http://www.debugmode.com/winmorph/ http://www.stoik.com/products/morphman/mm1_main.htm http://www.creativecow.net/articles/zwar_chris/morph/index.h tml http://meesoft.logicnet.dk/SmartMorph/ http://www.asahi-net.or.jp/~FX6M-FJMY/mop00e.html http://morphing-software-review.toptenreviews.com/ http://www.freedownloadscenter.com/Search/morphing.html
18