An inverse direct time domain boundary element method (IDTBEM) is proposed for the reconstruction of transient acoustic field radiated by arbitrarily shaped sources. The method is based on the theory of direct time domain boundary element method (DTBEM), which is free from the calculation of hypersingular integrals, and thus, its reconstruction process is relatively simple and easy to implement. However, the formulations of DTBEM cannot be used directly for the reconstruction of transient acoustic field, and therefore, new formulations with a modified time axis are derived. With these new formulations, a linear system of equations is formed and the reconstruction is performed in a marching-on-time (MOT) way. Meanwhile, to deal with the ill-posedness involved in the inverse process, the truncated singular value decomposition (TSVD) is employed. Numerical simulations with three examples of a sphere, a cylinder, and a simplified car model are carried out to verify the validity of IDTBEM, and the results demonstrate that the IDTBEM is effective in reconstructing the transient acoustic fields radiated by arbitrarily shaped sources in both time and space domains.