The evaluation of the structural performance and durability using only measurement data without any baseline data is practical. One measurement data type is the frequency response function (FRF) data set, which does not provide enough information about the health state because of the noise included in the data. Proper orthogonal modes (POMs) extracted from the FRF data in a prescribed frequency range are utilized as an index to recognize the existence of damage. The POMs represent the principal axes of inertia formed by the distribution of data on the modal coordinate curve. This work proposes a damage detection method to trace damage based on the POM energy curvature at the element of the finite element model. The validity of the proposed method is illustrated by a numerical experiment and an experimental test.