With regard to the three-dimensional (3D) shape measurement, based on speckle interferometry, of an object with a fine structure beyond the diffraction limit of the objective lens, phase distribution of each speckle region in a speckle pattern and a specklegram is discussed. The experimental results confirmed that the phase distribution with respect to the shape of an object exists spatially even in the speckle region of the speckle pattern. However, the measurement results of a superfine structure by speckle interferometry indicated that the spatial continuity of the phase distribution does not exist in each speckle but in the specklegram. This feature was used to confirm that a wide range of 3D shape measurements of an object with a fine structure can be realized by analyzing specklegrams calculated from speckle patterns before and after a lateral shift in speckle interferometry. Furthermore, it is confirmed that the shape measurement of a nonperiodical structure can be performed.