# Multi-View Geometry: An Interactive Companion This document accompanies the **Camera Pose & Epipolar Geometry** interactive. It walks through the math behind what the demo shows, then sketches the natural extensions (trifocal tensor, N-view collective null space) that the interactive does not yet visualise. Sections labelled **(to do)** describe concepts that are *not* currently implemented in the interactive. The rest mirror what you can already see and click on. --- ## 1. The Projective Camera Model A pinhole camera maps a 3D world point `P = (X, Y, Z)` to a 2D image point `u = (u, v)` via ``` u ~ K [R | t] P ``` where: - `R` (3×3 rotation) and `t` (3-vector translation) form the **extrinsic** pose: they describe the camera's position and orientation in the world. - `K` (3×3 upper-triangular) is the **intrinsic** matrix: focal length, principal point, skew. In the interactive, `fx = fy = 500`, `cx = 320`, `cy = 240`. - The `~` denotes equality up to a positive scale (homogeneous coordinates). Geometrically: shoot a ray from the camera centre `C` through the world point `P`; the ray hits the image plane at `u`. The interactive draws this ray as the edge of each camera's frustum, with the inverted image plane sitting in front of the pinhole — the physically faithful "real image" convention. **In the interactive:** the three cameras (A, B, C) each have their own `(R, t)` derived from a "look-at" pose; you can click the **flyto** icons in the camera panels to move the view camera into any of those poses and see the world from that camera's perspective. --- ## 2. Two-View Epipolar Geometry Given two cameras with centres `C1` and `C2` viewing the same 3D-Point `P`, three points lie in a single plane: `C1`, `C2`, and `P`. This plane is the **epipolar plane**. Its intersection with each image plane is the **epipolar line** for that view. That is the entire content of the epipolar constraint: corresponding points must lie on corresponding epipolar lines, because they are forced by geometry to live in the same plane through the two camera centres. **In the interactive:** select any 3D-Point (click a sphere in the 3D scene, or a UV correspondence point in a camera panel) and you will see: - the **yellow triangle** in 3D — the epipolar plane spanning `C1`, `C2`, `P`; - the **gold lines** in each camera canvas — the epipolar lines, which are the intersection of that triangle with each image plane. ### The baseline and the epipoles The line segment between the two camera centres `C1` and `C2` is called the **baseline**. The baseline projects to a single point in each image: - `e1` = the projection of `C2` into image 1 — the **epipole** in view 1. - `e2` = the projection of `C1` into image 2 — the **epipole** in view 2. Every epipolar plane contains the baseline, so every epipolar line in image 1 passes through `e1`, and every epipolar line in image 2 passes through `e2`. The set of epipolar lines forms a **pencil** through the epipole. If the baseline is parallel to an image plane, that image's epipole goes to infinity, and the epipolar lines in that image become parallel — this is the rectified-stereo case. **(to do)** The interactive does not yet draw the baseline as a 3D line segment between camera centres, nor mark the epipoles `e1, e2` on the camera canvases. Adding these would make the "all epipolar lines pass through a common point" structure visible at a glance. --- ## 3. The Essential and Fundamental Matrices Algebraically, the epipolar constraint between two views is ``` x2ᵀ F x1 = 0 (in pixel coordinates) x̂2ᵀ E x̂1 = 0 (in normalised / metric coordinates) ``` where: - `E = [t]× R` is the **essential matrix**, built from the relative rotation `R` and the cross-product matrix of the translation `t` between the two cameras. It encodes purely geometric (calibrated) two-view structure. - `F = K2⁻ᵀ E K1⁻¹` is the **fundamental matrix**, the un-calibrated cousin of `E` that operates directly on pixel coordinates. Both are 3×3, both have rank 2 (they are singular by construction, because of the cross product), and both are defined only up to a non-zero scale. **In the interactive:** the left panel shows the live numeric values of `E` and `F` for the currently selected camera pair, and the **Epipolar Constraint Verification** list shows that `x'ᵀ F x ≈ 0` (to floating-point precision) for every 3D-Point's pair of correspondences — a numerical check that the matrix really does encode the geometry. ### Point–line duality Here is the key structural fact `F` (and `E`) carry: - Given a point `x1` in image 1, the vector `l2 = F x1` is a **line** in image 2 — the epipolar line on which the corresponding point `x2` must lie. - Symmetrically, `l1 = Fᵀ x2` is the epipolar line in image 1 corresponding to a point `x2` in image 2. So `F` is a map *points → lines* in one direction and *points → lines* in the other. This is the point-line duality at the heart of two-view geometry, and it is what the interactive's gold lines are: each gold line is `F x` (or `Fᵀ x`) for the selected correspondence. ### Epipoles as null spaces Because every epipolar line in image 2 passes through the epipole `e2`, and `F x1` is always such a line, `e2` lies on `F x1` for every `x1`: ``` e2ᵀ F x1 = 0 for all x1 ⟹ e2ᵀ F = 0 ⟹ Fᵀ e2 = 0 ``` So `e2` is the **left null vector** of `F`. Similarly `e1` is the **right null vector**: `F e1 = 0`. The two epipoles are exactly the two null directions of the rank-2 fundamental matrix. The "dashed lines of the null space" are these epipolar pencils, all converging at the null-space point `e`. **(to do)** The interactive does not yet extract `e1, e2` from `F` (via SVD, taking the singular vector for the zero singular value) and overlay them on the canvases. With both the baseline drawn in 3D and `e1, e2` marked in 2D, the duality "baseline ↔ pair of epipoles ↔ null vectors of F" becomes visible end-to-end. --- ## 4. Three Views: The Trifocal Tensor **(to do)** With three cameras, pairwise fundamental matrices `F12`, `F13`, `F23` capture all *pairwise* constraints — but they over-count and miss a genuinely three-view constraint: that the three rays from three cameras through three corresponding points must meet at a single 3D point, and that the three epipolar planes share a single line in space. The right object for three views is the **trifocal tensor** `T` (a 3×3×3 array, 27 entries with internal constraints leaving 18 degrees of freedom). It encodes: - **Point–line–line transfer:** given a point in view 1 and a line in view 2, the corresponding line in view 3 is determined by `T`. - **Line–line–line consistency:** three lines, one in each image, are consistent (i.e., the back-projected planes meet in a single 3D line) iff a tensor relation in `T` vanishes. - **Point transfer:** given correspondences in views 1 and 2, the location in view 3 is determined. Equivalently: stack the three pairwise fundamental matrices and quotient out the redundancy, and you arrive at `T`. The pairwise epipoles `{e_ij}` and the trifocal tensor live in the same algebraic family — `T` is the natural three-view generalisation of "F is the pairwise null-space carrier." **(to do) interactive ideas:** with three cameras already in the scene, the interactive could compute `T` from the three known poses, visualise point-transfer (click a 3D-Point, show its prediction in view 3 from views 1 and 2 alone), and visualise the line-line-line consistency by drawing a line in two views and the predicted line in the third. --- ## 5. N Views: The Collective Null Space **(to do)** For `N` cameras observing `M` 3D-Points, all the projection equations stack into one large bilinear system. Two complementary global views: ### 5a. The measurement matrix and projective factorisation Stack all 2D measurements `x_ij` (camera `i`, point `j`) into a `2N × M` measurement matrix `W`. Up to scaling each column by the (unknown) projective depth `λ_ij`, the matrix `W̃ = [λ_ij x_ij]` factors as ``` W̃ = P · X (2N×M) (2N×4)(4×M) ``` where `P` stacks the `N` camera projection matrices and `X` stacks the `M` homogeneous 3D-Points. So the *correct* depth-scaled measurement matrix has rank exactly 4, and its **right null space** (in a `5`th-or-higher direction) is empty by construction. Conversely, the failure of rank 4 measures inconsistency. This is the **collective null-space** view: N-view structure-from-motion is a single rank-4 constraint, and pairwise `F`, three-view `T`, etc. are projections of it onto smaller subsets of the cameras. ### 5b. Bundle adjustment Once an initial `(P, X)` is recovered from the rank-4 factorisation (or chained from pairwise/triple-view estimates), bundle adjustment refines all camera poses and all 3D-Point positions jointly to minimise reprojection error, treating the whole problem as one large nonlinear least squares system. Its Jacobian inherits the same sparsity that the pairwise/triple-view formulations exploit. **(to do) interactive ideas:** add a "global view" panel that shows `W`, its singular values (illustrating the rank-4 structure), and a slider that perturbs one camera's pose to watch the rank-4-ness break. With the three existing cameras and eight 3D-Points the matrix is small enough to display in full. --- ## 6. Putting It All Together The progression the interactive walks you through (and where it goes next) is: 1. **One camera:** `u ~ K [R|t] P`. *(implemented — frustum, image plane, flyto)* 2. **Two cameras:** epipolar plane, epipolar lines, `E` and `F`, point–line duality. *(implemented — yellow triangle, gold lines, live matrices, x'ᵀFx verification)* 3. **Two cameras' algebraic skeleton:** baseline, epipoles, null-space structure of `F`. *(partially — F is shown; baseline and epipoles are **to do**)* 4. **Three cameras:** trifocal tensor, point/line transfer, three-view consistency. *(**to do**)* 5. **N cameras:** rank-4 measurement matrix, projective factorisation, bundle adjustment. *(**to do**)* Each step adds one layer of algebraic structure on top of the same underlying geometry: rays from camera centres through 3D-Points, and the planes/lines those rays span. --- *Back to the [interactive](./camera-pose.html).*