3D transformations

1A point in two 3D coordinate systems

A point of space with coordinates (X, Y, Z) is written in homogeneous coordinates as in the plane, by appending a 1:

Now take a second coordinate system, with axes X′, Y′ and Z′, as if it were attached to an object. It comes from the first by turning the axes and moving the origin, so call the first system the old one and the second the new one. The same point has coordinates M′ = [X′ Y′ Z′ 1]^T in the new system. Six numbers relate the two systems: three angles, ω_X, ω_Y and ω_Z, for the turn, and three for the shift.

The shift

If the axes are not turned and the new origin sits at t′ = (t′_X, t′_Y, t′_Z) in the old system, each old coordinate is the new one plus the position of the new origin:

The 1 at the end of M′ turns these three sums into one matrix product:

The last column of the matrix is multiplied by that 1, so t′ is added to every point. Without the fourth component, a shift could not be written as a matrix product, just as in the plane. The other way, the new coordinates are the old ones less t′.

2Turning about one axis

Turn the axes about Z by an angle ω_Z. The Z axis stays where it is, and X′ and Y′ are X and Y turned by ω_Z, measured from X to X′. Positive angles follow the right-hand rule: with the right thumb along the axis, the fingers curl the way a positive angle turns. Seen from the tip of the Z axis, looking back at the origin, a positive ω_Z turns X toward Y, counterclockwise.

A point that stays where it is gets new coordinates M′ = R_ZM, leaving out the final 1, with

Each new coordinate is the length of the point's shadow on a new axis, which is a dot product with that axis's unit vector. The unit vector of X′ is (cos ω_Z, sin ω_Z, 0) in the old system, so X′ = X cos ω_Z + Y sin ω_Z: the first row of R_Z. The rows of R_Z are the new axes, written in the old system.

R_Z may look like the transpose of the usual rotation matrix, with −sin ω_Z above the diagonal, and it is. That matrix turns points; R_Z keeps the point and turns the axes, and turning the axes by ω_Z changes the point's coordinates exactly as turning the point by −ω_Z would. The same pair appears in the note on 2D transformations, as R and R′.

The turns about the other two axes keep their own axis fixed in the same way:

In R_Y the minus sign sits above the diagonal. The right-hand rule turns Y toward Z about X, Z toward X about Y, and X toward Y about Z, always following the cycle X → Y → Z → X. About Y, the turn goes from Z to X, but the matrix lists X before Z, against the cycle, and that swaps the two signs.

3Any rotation from three angles

Turning about all three axes is done one axis at a time: first about Z by ω_Z, then about the new Y axis by ω_Y, then about the newest X axis by ω_X. Each turn converts the coordinates once more, into the axes as turned so far, so the matrices multiply from right to left:

For a turn alone, M′ = RM. Multiplied out, every entry mixes sines and cosines of the three angles. The first row, for example, is

The order is part of the definition. Matrix products do not commute, and neither do turns in space: give a book a quarter turn about one axis and then about another, then try the other order, and it ends up facing a different way. The same axes can be reached in another order, but with other angles. Turning first by ω_Z = 90° and then by ω_Y = −90° gives the same axes as turning first by ω_X = 90° and then by ω_Z = 90°: R_Y(−90°)R_Z(90°) = R_Z(90°)R_X(90°).

Why three numbers are enough

A 3 × 3 matrix has nine entries, but a rotation cannot have just any nine. The rows of R are the unit vectors of the new axes, so each has length 1 and every two of them are perpendicular: three conditions for the lengths and three for the right angles, all written at once as RR^T = I. Nine entries less six conditions leave three free numbers, and three angles are enough to set them. A mirror satisfies RR^T = I too, but it turns a right hand into a left hand; rotations are the matrices that also have det R = +1. The same equation says that the inverse of R is its transpose.

Turn and shift in one matrix

Now move the origin as well, to t′ in the old system. To find the new coordinates, subtract t′ and turn: M′ = R(M − t′) = RM + t, with t = −Rt′, which is the old origin written in the new system. In homogeneous coordinates that is one 4 × 4 product:

This is the Euclidean transformation of space. It moves space as a rigid piece, so it keeps lengths and angles, and it has six free numbers where its 2D counterpart has three.

Going back

To get the old coordinates back, undo the turn and then the shift: M = R^TM′ + t′. Writing R′ = R^T,

with t′ = −R′t. R′ undoes the three turns in reverse order, each by the opposite angle: R′ = R_Z(−ω_Z)R_Y(−ω_Y)R_X(−ω_X). Its columns are the new axes written in the old system, so R′M′ + t′ reads like the matrix H of the 2D note: start at t′, then walk X′ units along the X′ axis, Y′ units along Y′ and Z′ units along Z′.

When two angles do the same thing

At ω_Y = 90°, the turn about Y lays the X axis along the Z axis, pointing the opposite way, so the last turn, about X, happens about the same line as the first one, about Z. Only the difference ω_X − ω_Z matters (at −90°, the sum ω_X + ω_Z), and one of the three numbers is lost. This is called gimbal lock, and it is one reason why rotations are often stored in other forms than three angles, such as an axis and an angle.

4Perspective projection

A camera turns points of space into points of an image. The simplest model is the pinhole camera: every point M sends a straight ray through one point C, the optical center, and the image is where these rays cross a plane, the image plane. Put the origin of the coordinates at C and the Z axis perpendicular to the image plane. That axis is the optical axis, and the plane sits at Z = f, where f is the focal length. The image plane has its own axes x and y, parallel to X and Y, with their origin where the optical axis meets it, a point called the principal point.

The ray from C through M = (X, Y, Z) meets the plane at m = (x, y). Along the ray, the distance from the optical axis grows in proportion to the depth: sideways it is X at depth Z, and x at depth f. The two right triangles are similar, so

which gives Zx = fX and Zy = fY, or x = fX/Z and y = fY/Z. The division by Z is why distant things look small.

The matrix P

The division also makes the step from M to m nonlinear, so as it stands it cannot be a matrix product. Homogeneous coordinates fix that. Write the two equations, together with Z = Z, as

The left side is a multiple of m, and the right side is a matrix P times M:

The factor λ is the depth Z, since the third row of P copies Z. To read m, divide PM by its third component, as with any homogeneous vector. The division by Z has not gone away; it has moved to the end, where it no longer stands in the way of the matrix.

The camera is the picture of homogeneous coordinates

Take f = 1. Then x = X/Z and y = Y/Z: divide by the third component, which is exactly how a homogeneous vector [p q r]^T is read. The camera's space is the space with axes p, q and r from the note on homogeneous coordinates, the optical center is its origin, and the image plane is its plane r = 1. Each point of the image stands for a whole ray through C, as each point of the plane stood for a line through the origin.

That is also what one image cannot tell: how far along its ray a point was. M and 2M, twice as far in the same direction, give the same m. Recovering depth takes a second image from another place, where the same point lies on a different ray. The note on epipolar geometry shows where that ray appears in the second image, and the note on multiple views finds the point where the rays meet.

One point has no image at all: the optical center. P sends C = [0 0 0 1]^T to the zero vector, which stands for no point, since every ray starts at C.

The image behind the center

In a real pinhole camera, the image forms behind the hole, on the far side of C from the scene, at Z = −f. The rays cross at C, so a point above the axis lands below it and a point on the left lands on the right: the image is upside down. The same triangles give x = −fX/Z and y = −fY/Z, so P gets −f in place of f. The model puts the plane in front of C instead, where it catches the same rays without the flip, and that is why P usually has +f.

X-ray imaging fits the model with +f as it stands. The X-ray source plays the part of the optical center, and the object lies between the source and the detector, which is the image plane.

5A point given in another coordinate system

The projection needs M in the camera's coordinates, with C at the origin and Z along the optical axis. Points are rarely known that way: they are measured on an object, or in a room, in a coordinate system of their own. Take the camera's system as the old one, with axes X, Y and Z, and the object's as the new one, with axes X′, Y′ and Z′. Then the columns of R′ are the object's axes seen from the camera, and t′ is where the camera sees the object's origin. A point with coordinates M′ on the object is at M = S′M′ in the camera's system, as in section 3, and

Read it from right to left: first move the point into the camera's coordinates, then project it. The product PS′ is a single 3 × 4 matrix, and its third row gives λ, the depth of the point in front of the camera.

6From millimeters to pixels

The coordinates (x, y) of m are lengths on the image plane, say in millimeters, measured from the principal point. An image is read in pixels instead: u counts columns to the right and v counts rows downward, from the top left corner. Write w = [u v 1]^T for the same point in pixels.

To keep the two systems aligned, the camera's axes are chosen along the sensor: X and x point right, along the rows, and Y and y point down, along the columns. With Z pointing forward, into the scene, the three axes make a right-handed system. Then no turn is needed between (x, y) and (u, v): any turn of the camera is already in S′. What is left takes three steps.

Shift. The principal point falls on some pixel (u₀, v₀), usually near the center of the image, not at the corner where u and v start. Adding u₀ and v₀ is not enough on its own, though, since x and y are millimeters and u and v are pixels.

Scale. The sensor has α_x pixels per millimeter along x and α_y along y, so u = α_xx + u₀ and v = α_yy + v₀. The two scales differ when the pixels are not square.

Skew. If the rows and columns of the sensor are not exactly perpendicular, a step along y also moves across columns, which adds a term sy to u. For most cameras s is 0 or very close to it.

Together:

K is called the camera's calibration matrix. With its bottom row [0 0 1], it is an affine transformation of the image plane, of the kind in the note on 2D transformations, so it keeps parallel lines. Its five numbers depend only on the camera: the size and layout of its pixels, and where its optical axis meets the sensor.

7The camera model

Putting the steps together: move the point into the camera's coordinates, project it, and convert to pixels.

Multiplying K by P gives

The focal length never appears alone, only multiplied by α_x, α_y and s. A lens twice as long with pixels twice as large gives exactly the same image, so images can only tell the products, and they get names of their own: α = fα_x, β = fα_y and γ = fs.

So a camera is described by eleven numbers. Five are intrinsic parameters, α, β, γ, u₀ and v₀, which belong to the camera itself. Six are extrinsic parameters, the angles ω_X, ω_Y, ω_Z and the three components of t′, which say where the camera stands and which way it points, relative to the object. The count agrees with the matrix: the whole product KPS′ is a 3 × 4 matrix with twelve entries, and, as in any homogeneous equation, multiplying it by a number changes nothing, which leaves eleven.

This product is called the general projection matrix. It is often written P as well, and A, B, C, … when several cameras look at the same scene. The simple P of section 4 is the case K = I and S′ = I. Finding the eleven numbers from images of known points is called camera calibration.

8Summary

The key ideas next to their NumPy equivalents.