Resolution is the smallest physical movement measurable. It is defined as the distance of a single count. For linear encoders resolution is represented in µm/count or nm/count; For rotary encoders resolution values are measured in counts/revolution, arc-seconds/count, or micro-radians/count.

Resolution, Accuracy, and Repeatability

While evaluating encoders, there are a few common terms that are critical to specifying the best encoder for the application. These terms are often misunderstood or misused which can cause confusion when designing motion control systems. This application note defines these key terms and how they relate to MicroE encoders. Understanding these terms helps define the encoder’s role in the systems overall servo performance. It is also important to consider that even the most accurate and repeatable encoder is only a small part of a system’s servo performance.


Resolution is the smallest physical movement measurable. It is defined as the distance of a single count. For linear encoders resolution is represented in µm/count or nm/count; For rotary encoders resolution values are measured in counts/revolution, arc-seconds/count, or micro-radians/count.

Figure 1: Resolution - Index Reference Marks

Figure 1: The majority of MicroE’s rotary and linear encoders use a 20µm pitch grating. Reflected light creates a 20 µm period analog signal that is then interpolated by factor of 4 to a factor of 16,384. The Index reference mark is also shown above the main track.

The final resolution required for the application must be considered carefully when specifying encoder resolution. The servo control system is likely to dither between two counts. If the application requires resolution of 1 micron and a 1 micron encoder is selected, the final system could dither between two counts (2 microns). Selecting a higher resolution encoder will reduce the magnitude of the dither. Selecting an encoder with a higher resolution also brings with it a few factors to consider in the design of the motion control system. A primary consideration is that motion controller and servo drive encoder inputs have a maximum frequency that can be supported on their encoder inputs. Most controllers and drives clearly specify this limit. 20 million counts/sec is a common maximum specification. For a rotary encoder system using a 5.5” diameter rotary scale like the one in Figure 1 above (20,250 lines), an encoder read head with 4000x interpolation achieves resolution of 81 Million counts/rev, but using a controller or drive with a 20 million counts/sec maximum encoder input frequency limits the maximum speed to only 15 RPM. Using an analog sin/cos output encoder instead of a digital A quad B output encoder is a common solution to this challenge. MicroE serial interface encoders like the MII6800Si are also a powerful way to achieve high resolution without sacrificing speed. See tech note TN-1301 “Understanding Binary Words and the MicroE Serial Interface Encoder” for more details.


Accuracy is the difference between the actual “true” position of the motion axis and the position as reported by the encoder read head. Measuring accuracy requires a very accurate measurement standard such as a laser interferometer. Accuracy is described in values of Sub Divisional Error (SDE), Linearity, Slope, and/or Total Linearity.

Figure 2: Typical error plot of a Mercury encoder tape scale

Sub-Divisional Error (SDE) is a cyclic error created by imperfections of the Sine/Cosine analog output. This error appears in the interpolated counts created and does not accumulate. It is measured in the <nm range RMS. Another name for this may be intra-fringe accuracy since it repeats with each diffracted fringe period. Plotting sin and cos outputs in an XY format allows us to use the Lissajous pattern for alignment. A perfect 90 degree phase shift and 1:1 amplitude, centered on (0, 0) will produce a perfect circle on the screen and would represent zero SDE. All encoder systems possess small deviations from perfection as depicted in exaggerated form in Figure 3 below. The majority of MicroE encoders offer electronic gain, phase, and offset corrections to optimize the signal and balance the offsets, in turn minimizing SDE.

Figure 3 : Lissajous figures of SDE sources. Red circle is DC offset error, Green is gain mismatch, Blue is phase≠90°.

Figure 3: Lissajous figures of SDE sources

Slope Error is represented as the maximum expected accumulative error. This is a value that can be added to a controller to cancel out the slope in the plotted data.

Linearity Error is the total range of non-accumulating positional error plotted over a meter of travel after the slope error has been removed. It is typically measured in ≤± μm (max/meter).

Total Linearity: The stability and process controls behind the manufacturing of linear gratings and scales keeps the slope error in the same range as the linearity error, allowing the total linearity to be specified in the ±μm/m range (Figure 2, range between lowest and highest error of blue line). This is also a component of rotational error as well. For more detailed information on rotational accuracy, refer to tech note TN-1101 “Alignment of Rotary Scales”.


Repeatability is the total range of positions attained when the system is commanded to one location, multiple times, under all conditions. Specified as ± (resolution), this is the repeatability of any digital servo system as measured at the encoder and does not take into consideration the effects of Abbe error, friction, etc.

Figure 4: Stage repeatability plot

The plot above (Figure 4) shows bidirectional repeatability of a linear stage with data points recorded every millimeter. This stage is driven by a ball screw and the plot is provided by a linear encoder. Again, a laser interferometer was used as the measurement standard. In a perfect system all of the readings would sit directly on top of each other and look like a single plot. The range between the highest reading and the lowest reading at one position is the actual repeatability. Possible sources of repeatability error in this plot include the encoder, bearings, backlash, and stiction (the friction that tends to prevent stationary surfaces from being set in motion).


A thorough understanding of resolution, accuracy, and repeatability is a key foundation in determining the optimal encoder specifications. Finding the right balance between all motion control system components is an integral part of specifying a motion control system that meets all application requirements. Selecting the best encoder accuracy and resolution is important but other factors must be considered. A highly accurate and repeatable optical encoder alone cannot compensate for a poorly designed positioning device nor will a great mechanical design achieve high performance if the encoder is underspecified for the application.

Please contact Celera Motion with any questions regarding these recommendations.