AprilTag and ArUco markers: fiducial markers and how robots use them

When a robot perceives its surroundings, it often needs to know where objects are and where it stands relative to them. For example, to find its charging station, locate a part it has to pick up or identify a work area.

One of the cheapest options for this is a 2D camera. However, images give us pixels, not distances or positions in space.

A lab room instrumented with markers: each one is a reference with a known identity and position. Photo: STAR-loc dataset, CC BY 4.0.

One way to solve this is with fiducial markers: images similar to QR codes, with patterns designed to be detected by vision algorithms.

Knowing the size of the marker and using a calibrated camera, we can estimate its position (x, y, z) and orientation (roll, pitch, yaw) relative to the camera. In addition, each pattern contains an ID that identifies the marker and lets us use it as a reference point.

There are different kinds of fiducial markers; the most common are AprilTag and ArUco, each with several families of patterns.

What a marker is

A fiducial marker looks like a QR code, but it is much simpler. Instead of storing text, it stores a small number that identifies it, and it is designed to be detected from far away and measured precisely, not to be read up close.

A marker has three parts, and each one has a job:

• The black border. It is the first thing the detector looks for, because it stands out from far away and at an angle, before the bits can be read.
• The bit grid. The black and white cells inside. For example, in AprilTag's 36h11 family the grid is 6 × 6 cells, and its pattern encodes the marker's ID.
• The white zone around it. It must be at least one cell wide. Without it, the black border blends into the background and the detector cannot find it.

What we actually use for measuring are the four corners of the border. From their position in pixels, the real size of the marker and the camera calibration, we can compute where the marker is in space. That is why a bent, glossy or partly covered marker gives a wrong position, or none at all.

The bits are not simply the ID written in binary, but an error-correcting code. In the 36h11 family there are 36 bits and any two valid IDs always differ in at least 11 of them. If lighting or blur flips a few bits, the detector can correct them or discard the reading, but it does not return a wrong ID. This is what makes it safe to trust a marker to move a robot.

The two most widely used types are AprilTag, developed at the University of Michigan and common in ROS 2, and ArUco, developed at the University of Córdoba and included in OpenCV. Both are detected with the same kind of algorithm, and OpenCV can read both. There is also the ChArUco board, which is not a marker but a chessboard with an ArUco in each white square; it is the usual choice for calibrating cameras.

A printed ChArUco board: the chessboard provides the corners and the ArUcos identify each one. Photo: OpenCV, Apache-2.0.

From pixels to 3D coordinates

Detection uses classical computer vision, with no neural networks, so it runs in real time on an ordinary processor. The detector converts the image to grayscale, applies an adaptive threshold to separate light from dark even when the lighting is uneven, looks for four-sided shapes, rectifies the inside of each one and reads its bits. If the result matches a valid ID, it refines the corner positions to better than a pixel. Up to this point everything happens in the image: the detector returns an ID and four corners in pixels.

What a pose is

Imagine a line coming straight out of the center of the camera lens. The marker's pose is six numbers measured relative to that line: how far ahead it is, how far to the side and how far up, plus how it is rotated about each of the three axes. With this information the robot knows where to go and how to arrive oriented. A closer or larger marker covers more pixels, and so it gives a more precise pose.

From the corners to the pose

A calibrated camera projects each point in space to an image pixel according to

$$s\,\begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K\,[\,R \mid t\,]\begin{bmatrix} X \\ Y \\ 0 \\ 1 \end{bmatrix}$$

where $K$ describes the lens (the intrinsic parameters) and $R$ and $t$ are the rotation and translation between camera and marker, that is, the pose. The corners have known coordinates on the marker, because it is a square of side $L$, and measured coordinates in the image. Finding $R$ and $t$ from those correspondences is the problem known as PnP. For a flat square marker the IPPE method is used, available in OpenCV as SOLVEPNP_IPPE_SQUARE.

The planar ambiguity

When the four points lie on one plane, two different poses can produce almost the same image. A marker tilted to the left and one tilted to the right look practically the same when seen head-on at medium distance. This is not a flaw in the algorithm but a consequence of the geometry. IPPE returns both solutions with their reprojection error; when they are similar, the orientation can flip from one to the other between frames. The way to avoid it is to place markers that are not on the same plane, or to rely on another source of information, such as odometry, a second marker or a second camera.

Accuracy and calibration

Position is estimated well at short range and degrades as the marker gets farther away, because it covers fewer pixels. Orientation is the delicate part: head-on, because of the planar ambiguity, and at steep angles, because corner noise turns into degrees of error. None of this works without calibrating the camera, either. Without the intrinsic parameters and the lens distortion, the detector finds the corners correctly but converts them to meters with a lens model that is not the real one. Calibration is done once per camera and resolution, with a ChArUco board and several photos from different angles.

There are also situations where detection fails or misleads: motion that blurs the corners, a reflection on a cell, a bent marker or one on a curved surface, a covered corner, or a declared size that does not match the printed one. For example, a marker printed at 97 % of its size gives distances 3 % too short, always and without any warning. So before letting a pose move a motor, it is worth checking that the reprojection error is low and stable, that distance and angle are within the expected range, that there are no jumps between frames, and that the pose is expressed in the frame the controller expects.

What the detector delivers: each marker with its ID and its three axes, already in meters. Screenshot: TagSLAM, Apache-2.0.

From the marker to the robot

The detector returns the marker's pose relative to the camera, $T_{cam \leftarrow tag}$. What the robot needs, however, is the marker's position relative to its own base. To get it, the two transformations are composed:

$$T_{base \leftarrow tag} = T_{base \leftarrow cam}\; T_{cam \leftarrow tag}$$

In ROS 2 this chain of transformations is handled by tf2. The camera publishes its image together with its calibration, the detector publishes one frame per marker, and whatever consumes it, for example Nav2's docking, localization or the arm planner, asks for the transformation it needs with its timestamp. A frequent mistake is in the axes: the camera's optical frame looks along $z$, while the robot's body frame looks along $x$. Also, the marker's $z$ axis points into the marker in AprilTag and out of it in ArUco. Before combining two poses it is worth confirming which frame each one is in.

Hand-eye calibration

When the camera is mounted on the wrist of an arm, the fixed transformation between the camera and the robot flange has to be known, and it is not something you can measure with a ruler. It is solved by posing the equation $AX = XB$: the arm goes through several poses; at each one the flange motion, which the robot knows, and the marker motion seen by the camera are recorded, and $X$ is the only transformation consistent with all the pairs. OpenCV solves it with calibrateHandEye and MoveIt integrates it in RViz. For this calibration the marker should be close and facing the camera.

The real setup: camera on the wrist and board fixed on the table. Photo: MoveIt, BSD-3.

Several markers

With a map where the pose of each marker is known, each frame is solved as a single PnP problem using all the visible corners. If the map does not exist, TagSLAM builds it by optimizing the robot poses and the marker poses at the same time. More markers is not always better: a distant or steeply tilted one adds more noise than information, and one mismeasured in the map drags the others. What helps are markers that are clearly visible and placed on different planes.

From the marker to the action

The reprojection error, that is, the difference between where the corners should fall according to the computed pose and where they were detected, is the confidence indicator. If it grows, something is wrong: the calibration, the declared size or a bent marker. With a trustworthy pose, the typical workstation case always follows the same steps:

1. The robot navigates to an approximate area using its map and its lidar.
2. It detects the station marker.
3. It estimates the error in position and orientation relative to where it should be.
4. It corrects its position, checks that it is within tolerance, and only then acts.

In manipulation the marker is usually used twice. A fixed camera on the robot base gives an approximate pose and the arm approaches; then a wrist camera sees the marker again and corrects in closed loop down to the final position.

A real station: one marker on the shelf for the part and another on the floor for docking. Photo: From Pixels to Shelf, CC BY 4.0.