Self-excited vibration of a flexibly supported shafting system in a gravity water tunnel was observed in the testing of friction of water-lubricated rubber bearings. The measured vibrations indicated that the self-excited vibration, characterized by a single-mode vibration modulated by the shaft speed, emerged at a specific speed and grew stronger as the shaft speed decreased, but it would cease at a very low-speed. To explain the mechanism of instability, a dynamic model of the system was built. The substructure synthesis method was employed to model the dynamics of the shafting system, which was divided into two subsystems: the flexible support and the shaft. Before the synthesis, natural frequencies and modes of the subsystems were computed by the finite element method. According to the modal parameters and connecting conditions, a synthesized model was built to take into account the friction between the shaft and the bearing. The fourth-order Runge–Kutta method was used to solve the dynamic equations, and the influences of physical parameters on vibration stability were analyzed. The results have shown that certain vibration modes of the flexible support tend to be unstable under the excitation of the friction. Both the simulation and experimental results have exhibited that the unstable modes are associated with the torsional vibration of the flexible support, which are in fact lightly damped and accordingly can be easily excited by the circumferential friction. Therefore, the coupling between the torsional vibration of the flexible support and the friction of the water-lubricated rubber bearing is the main factor leading to the vibration instability.