Automatic Camera Recovery for Closed or Open Image Sequences

Andrew W. Fitzgibbon and Andrew Zisserman University of Oxford

                                                       {awf,az}@robots.ox.ac.uk

000  002  004  006  008  010  000  006  012  018  024  030  000  006  012  018  024  030  001  005  010  015  020  025

Objective

Obtain camera positions and 3D structure reliably over long sequences. Fully automatic system: need point tracks through sequence and an estimation procedure which distributes error over the sequence. Build on competences: two and three view reliability is high, feature tracking is robust.

• Fundamental matrix F and trifocal tensor T estimated well.

• Bundle adjustment works well, given initial estimate.

• Sequential approaches suffer from error accumulation, batch method preferable where all images are available.

• Not yet a reliable noniterative batch

method. (Triggs’ method needs initial structure, Tomasi-Kanade not projective).

• “Pasting” projective frames [Zisserman et.

al. 95, Laveau 96], is a sequential approach readily converted to batch.

• Build hierarchical reliable batch method by “pasting” projective frames

Maximum Likelihood Estimation

At all stages, want ML estimate of structure and motion. Homogeneous 3D points X_{j} are projected by 3 × 4 camera matrices P_{i}. Detected 2D points are x_{ij} (homogeneous 3-vectors). Assuming errors on positions of image points are Gaussian distributed, the ML estimate minimizes reprojection error:

ϵ^{2} = ∑_{ij} d^{2}(x_{ij}, P_{i}X_{j})

Where the 2D Euclidean distance d^{2} is

d^{2} [( \begin{matrix}x \\ y \\ w \end{matrix} ) , ( \begin{matrix}x′ \\ y′ \\ w′ \end{matrix} )] = (\frac{x}{w} − \frac{x′}{w′})^{2} + (\frac{y}{w} − \frac{y′}{w′})^{2}

This is minimized efficiently by sparse bundle adjustment [Hartley 94].

Algorithm: T Estimation

The workhorse of the system. 1.Compute corners [Harris]. 2.Get pairwise correlation matches. 3.RANSAC fit F to each pair. 4.Make triplet matches. 5.RANSAC fit T to triplets. 6.Repeat

(a)ML estimate of T . (b)Guided matching

Until convergence Note that guided matching and estimation must use same error metric to avoid oscillation.

Typical results for T

Input Images

Corners

Inliers  Outliers  Guided

Adaptive RANSAC

RANSAC termination is generally described as requiring an estimate of the inlier proportion ε, then number of samples required is r = ⌈\frac{log(1−P)}{log(1−ε^{n})}⌉. But ε may be initialized to 0 and adaptively updated [Torr 97]. Example results for P = 99%, n = 6:

Sample Inliers ε Required r
0  9% 6532519.
0  28% 8958.
2  50% 261.
4  50% 261.
5  59% 102.
26  83% 11.

Terminate when Samples taken > Required.

Registration of trifocal tensors into consistent projective frames

Homographies of P^{3} are computed which place tensors T_{234} and T_{345} in the frame of T_{123}.

Computation of H

Point in first triplet: X_{i}; Correspondence in second: X′_{i}; Cameras: P_{j}, P′_{j}

P_{j} = P′_{j}H^{−1}  (1)
X_{i} = HX′_{i}  (2)

Methods:

I. Consistent 3D points: Minimize (2). II. Consistent cameras, one-view overlap.

Minimize (2) subject to (1). III. Consistent cameras, many-view overlap. Minimize (1).

All have linear initializations followed by nonlinear minimization of reprojection error.

Finding Correspondences

Comparison of triplet registration

Table wuz here

Hierarchical Registration

Pasting through sequence leads to

accumulation of error, such that final bundle adjustment ends in (wrong) local minima. Can be corrected by bundle adjustment after each paste, but very slow. Therefore hierarchically register subsequences: a tractable batch process.

Algorithm: Batch Estimation

1.Optimally estimate trifocal tensors T for all

consecutive image triplets. 2.Compute structure points {X_{i}} and camera positions {P_{1}, P_{2}, P_{3}} for each triplet.

3.Register triplets into consistent

sub-sequences using 3-space homographies H. 4.Bundle adjust sub-sequences. 5.Hierarchical registration of sub-sequences

into longer sub-sequences. 6.Final bundle adjustment of cameras and 3D

structure for the complete sequence.

Closure

For “closed” sequences, the additional constraint can significantly improve camera recovery. Easily imposed by the introduction of homography “hinges” into the structure.

Before and after application of the closure constraint. Suppose 3 hinges, then minimize for each 3D point X, over homographies H_{i}

∑ D^{2}(X_{1}, H_{2}X_{2}) + ∑ D^{2}(H_{2}X_{2}, H_{3}X_{3}) + ∑ D^{2}(H_{3}X_{3}, X_{1})

Results

Cameras, 3D points and lines, VRML

models.

Basement: 3D points, lines and cameras.

Basement: Planes fitted to 3D lines.

Model House: Plane fitting and

photogrammetric editing.

Dinosaur: Occluding contour volume

intersection after recovery of cameras. Right: After imposition of turntable constraint.

Castle: 3D points and cameras.

Acknowledgements: Thanks to the University of Hannover for permission to use the dinosaur sequence and to the University of Leuven for the castle. Thanks to Cordelia Schmid for the 3D line extraction.