Multi-View Geometry: An Interactive Companion

This document accompanies the Camera Pose & Epipolar Geometry interactive. It walks through the math behind what the demo shows, then sketches the natural extensions (trifocal tensor, N-view collective null space) that the interactive does not yet visualise.

Sections labelled (to do) describe concepts that are not currently implemented in the interactive. The rest mirror what you can already see and click on.

1. The Projective Camera Model

A pinhole camera maps a 3D world point P = (X, Y, Z) to a 2D image point u = (u, v) via

u ~ K [R | t] P

where:

• R (3×3 rotation) and t (3-vector translation) form the extrinsic pose: they describe the camera's position and orientation in the world.
• K (3×3 upper-triangular) is the intrinsic matrix: focal length, principal point, skew. In the interactive, fx = fy = 500, cx = 320, cy = 240.
• The ~ denotes equality up to a positive scale (homogeneous coordinates).

Geometrically: shoot a ray from the camera centre C through the world point P; the ray hits the image plane at u. The interactive draws this ray as the edge of each camera's frustum, with the inverted image plane sitting in front of the pinhole — the physically faithful "real image" convention.

In the interactive: the three cameras (A, B, C) each have their own (R, t) derived from a "look-at" pose; you can click the flyto icons in the camera panels to move the view camera into any of those poses and see the world from that camera's perspective.

2. Two-View Epipolar Geometry

Given two cameras with centres C1 and C2 viewing the same 3D-Point P, three points lie in a single plane: C1, C2, and P. This plane is the epipolar plane. Its intersection with each image plane is the epipolar line for that view.

That is the entire content of the epipolar constraint: corresponding points must lie on corresponding epipolar lines, because they are forced by geometry to live in the same plane through the two camera centres.

In the interactive: select any 3D-Point (click a sphere in the 3D scene, or a UV correspondence point in a camera panel) and you will see:

• the yellow triangle in 3D — the epipolar plane spanning C1, C2, P;
• the gold lines in each camera canvas — the epipolar lines, which are the intersection of that triangle with each image plane.

The baseline and the epipoles

The line segment between the two camera centres C1 and C2 is called the baseline. The baseline projects to a single point in each image:

• e1 = the projection of C2 into image 1 — the epipole in view 1.
• e2 = the projection of C1 into image 2 — the epipole in view 2.

Every epipolar plane contains the baseline, so every epipolar line in image 1 passes through e1, and every epipolar line in image 2 passes through e2. The set of epipolar lines forms a pencil through the epipole.

If the baseline is parallel to an image plane, that image's epipole goes to infinity, and the epipolar lines in that image become parallel — this is the rectified-stereo case.

(to do) The interactive does not yet draw the baseline as a 3D line segment between camera centres, nor mark the epipoles e1, e2 on the camera canvases. Adding these would make the "all epipolar lines pass through a common point" structure visible at a glance.

3. The Essential and Fundamental Matrices

Algebraically, the epipolar constraint between two views is

x2ᵀ F x1 = 0          (in pixel coordinates)
x̂2ᵀ E x̂1 = 0          (in normalised / metric coordinates)

where:

• E = [t]× R is the essential matrix, built from the relative rotation R and the cross-product matrix of the translation t between the two cameras. It encodes purely geometric (calibrated) two-view structure.
• F = K2⁻ᵀ E K1⁻¹ is the fundamental matrix, the un-calibrated cousin of E that operates directly on pixel coordinates.

Both are 3×3, both have rank 2 (they are singular by construction, because of the cross product), and both are defined only up to a non-zero scale.

In the interactive: the left panel shows the live numeric values of E and F for the currently selected camera pair, and the Epipolar Constraint Verification list shows that x'ᵀ F x ≈ 0 (to floating-point precision) for every 3D-Point's pair of correspondences — a numerical check that the matrix really does encode the geometry.

Point–line duality

Here is the key structural fact F (and E) carry:

• Given a point x1 in image 1, the vector l2 = F x1 is a line in image 2 — the epipolar line on which the corresponding point x2 must lie.
• Symmetrically, l1 = Fᵀ x2 is the epipolar line in image 1 corresponding to a point x2 in image 2.

So F is a map points → lines in one direction and points → lines in the other. This is the point-line duality at the heart of two-view geometry, and it is what the interactive's gold lines are: each gold line is F x (or Fᵀ x) for the selected correspondence.

Epipoles as null spaces

Because every epipolar line in image 2 passes through the epipole e2, and F x1 is always such a line, e2 lies on F x1 for every x1:

e2ᵀ F x1 = 0     for all x1     ⟹     e2ᵀ F = 0     ⟹     Fᵀ e2 = 0

So e2 is the left null vector of F. Similarly e1 is the right null vector: F e1 = 0. The two epipoles are exactly the two null directions of the rank-2 fundamental matrix. The "dashed lines of the null space" are these epipolar pencils, all converging at the null-space point e.

(to do) The interactive does not yet extract e1, e2 from F (via SVD, taking the singular vector for the zero singular value) and overlay them on the canvases. With both the baseline drawn in 3D and e1, e2 marked in 2D, the duality "baseline ↔ pair of epipoles ↔ null vectors of F" becomes visible end-to-end.

4. Three Views: The Trifocal Tensor (to do)

With three cameras, pairwise fundamental matrices F12, F13, F23 capture all pairwise constraints — but they over-count and miss a genuinely three-view constraint: that the three rays from three cameras through three corresponding points must meet at a single 3D point, and that the three epipolar planes share a single line in space.

The right object for three views is the trifocal tensor T (a 3×3×3 array, 27 entries with internal constraints leaving 18 degrees of freedom). It encodes:

• Point–line–line transfer: given a point in view 1 and a line in view 2, the corresponding line in view 3 is determined by T.
• Line–line–line consistency: three lines, one in each image, are consistent (i.e., the back-projected planes meet in a single 3D line) iff a tensor relation in T vanishes.
• Point transfer: given correspondences in views 1 and 2, the location in view 3 is determined.

Equivalently: stack the three pairwise fundamental matrices and quotient out the redundancy, and you arrive at T. The pairwise epipoles {e_ij} and the trifocal tensor live in the same algebraic family — T is the natural three-view generalisation of "F is the pairwise null-space carrier."

(to do) interactive ideas: with three cameras already in the scene, the interactive could compute T from the three known poses, visualise point-transfer (click a 3D-Point, show its prediction in view 3 from views 1 and 2 alone), and visualise the line-line-line consistency by drawing a line in two views and the predicted line in the third.

5. N Views: The Collective Null Space (to do)

For N cameras observing M 3D-Points, all the projection equations stack into one large bilinear system. Two complementary global views:

5a. The measurement matrix and projective factorisation

Stack all 2D measurements x_ij (camera i, point j) into a 2N × M measurement matrix W. Up to scaling each column by the (unknown) projective depth λ_ij, the matrix W̃ = [λ_ij x_ij] factors as

W̃   =   P · X
(2N×M)   (2N×4)(4×M)

where P stacks the N camera projection matrices and X stacks the M homogeneous 3D-Points. So the correct depth-scaled measurement matrix has rank exactly 4, and its right null space (in a 5th-or-higher direction) is empty by construction. Conversely, the failure of rank 4 measures inconsistency. This is the collective null-space view: N-view structure-from-motion is a single rank-4 constraint, and pairwise F, three-view T, etc. are projections of it onto smaller subsets of the cameras.

5b. Bundle adjustment

Once an initial (P, X) is recovered from the rank-4 factorisation (or chained from pairwise/triple-view estimates), bundle adjustment refines all camera poses and all 3D-Point positions jointly to minimise reprojection error, treating the whole problem as one large nonlinear least squares system. Its Jacobian inherits the same sparsity that the pairwise/triple-view formulations exploit.

(to do) interactive ideas: add a "global view" panel that shows W, its singular values (illustrating the rank-4 structure), and a slider that perturbs one camera's pose to watch the rank-4-ness break. With the three existing cameras and eight 3D-Points the matrix is small enough to display in full.

6. Putting It All Together

The progression the interactive walks you through (and where it goes next) is:

1. One camera: u ~ K [R|t] P. (implemented — frustum, image plane, flyto)
2. Two cameras: epipolar plane, epipolar lines, E and F, point–line duality. (implemented — yellow triangle, gold lines, live matrices, x'ᵀFx verification)
3. Two cameras' algebraic skeleton: baseline, epipoles, null-space structure of F. (partially — F is shown; baseline and epipoles are to do)
4. Three cameras: trifocal tensor, point/line transfer, three-view consistency. (to do)
5. N cameras: rank-4 measurement matrix, projective factorisation, bundle adjustment. (to do)

Each step adds one layer of algebraic structure on top of the same underlying geometry: rays from camera centres through 3D-Points, and the planes/lines those rays span.

Back to the interactive.