The band-averaged damping is determined by modal damping of the modes inside/outside the frequency band. In this paper, the effect of modes inside/outside a frequency band on the accuracy of the decay rate method (DRM) in vibration damping test was studied. First, to study the effect of modes inside a frequency band on the accuracy of DRM, the relationship between the damping loss factors (DLFs) from DRM and that from statistical energy analysis (SEA) was deduced theoretically. As shown from the analytical results, the DLF from DRM is close to that from SEA when the differences of the modal loss factors of the modes in the band are small; while the DLF from DRM is much less than that from the SEA if the differences of the modal loss factors in the band are large. Second, the influence of energy leakage from modes outside a frequency band on damping test results was studied numerically. The study reveals that when there is no mode in a frequency band, the effect of the mode outside the band on the decay rate (DR) of DRM is only related with the DR of the mode instead of the location of the mode; while when the band contains modes, the DR of DRM is influenced by the DR, amplitude, and location of the modes outside the band and the influence has a positive variation with DR, amplitude, and distance from the mode to the band. Finally, plate's transient impact response data from finite element simulation were used to verify the relevant conclusions.