Tissue elasticity is one of the significant diagnostic parameters to study disease progression in biological tissues. Elastography measurements hold strong potential for determining the mechanical properties of malignant tissues in clinical practice. Optical methods for quantifying the elastic properties have received the attention of both research and medical community. In the current work, we calculate the elastic modulus of tissue-mimicking phantoms and ex-vivo biological tissue using holographic imaging by tracing Rayleigh wave propagation on the tissue surface. Rayleigh waves produced by the electromechanical actuator over a set of excitation frequencies are imaged by an ultrafast camera. The phase map reconstruction of the Rayleigh wave is carried out using a phase-shifting algorithm and the distance-time slope method is analyzed to compute Rayleigh wave velocities. The elastic modulus calculated by holographic imaging is validated against the gold standard mechanical compressional testing data. The quantitative results obtained from this current optical system combining holographic imaging and Rayleigh wave tracing are highly effective in ex-vivo and in-vivo tissue characterization and carry high potential as a promising tool in clinical translations and commercial adaptations.