What determines the talbot plane (the z-height between the scale and the readhead)?

The talbot plane or talbot distance is based on the pitch of the scale and the wavelength of the incident light. The exact equation is as follows:
Talbot Distance = 2 / (Wavelength of incident light x Pitch^2)
Therefore, if you want to increase the number of lines on the scale, the pitch decreases – which means the talbot distance decreases and makes alignment of the readhead more difficult.

More information on talbot imaging in high resolution optical encoders.