Impulse response functions (IRFs) and frequency response functions (FRFs) are bases for modal parameter identification of single-input, single-output (SISO) and multiple-input, multiple-out (MIMO) systems, and the two functions can be transformed from each other using the fast Fourier transform and the inverse fast Fourier transform. An efficient iterative algorithm is developed in this work to directly and accurately calculate the IRFs of SISO and MIMO systems in the time domain using relatively short input and output data series. The iterative algorithm can avoid the time-consuming inversion of a large matrix in the conventional least-square method for calculating an IRF, greatly reducing the computation time. In addition, a fitting index and an error energy decreasing coefficient are introduced to evaluate the accuracy in calculating an IRF and to provide the termination criterion for the iterative algorithm. A new coherence function is also introduced to evaluate the accuracy of calculated IRFs and FRFs at different spectral lines. Two examples are given to illustrate the effectiveness and efficiency of the methodology.