In computer-generated holograms (CGH), a random phase (RP) is usually introduced to spread the object information in the hologram and avoid the frequency-filtering effect in the reconstructed image. However, this action introduces speckle noise in the reconstructed image but also permits one to reconstruct objects with bigger size than the hologram. In recent years, alternative RPs have been proposed to reduce the speckle noise in comparison with the uniform distributed RP that is usually applied. However, its effects in different types of holograms, its capacity to reconstruct big objects, and its capability to eliminate the frequency-filtering effect have not been reviewed in detail. We present an analysis of the normal distributed, pink, brown, and Perlin phases in Fourier and Fresnel holograms coded as amplitude and phase (Kinoform and complex amplitude encoded). Only in the Kinoform holograms, the RP is required to eliminate the frequency-filtering effect. In other cases, the phase is useful to record big objects in the hologram. The Perlin phase had a better performance than the other phases, achieving the reconstruction of objects with bigger size than the hologram and reducing the speckle noise. We present a quantitative and qualitative analysis of each distribution, including simulated and experimental results that confirm the numerical analysis. We believe these results could be helpful in the selection of an adequate RP in CGH, according to the hologram configuration.