In recent years, there has been much interest in the use of automatic balancing devices (ABD) in rotating machinery. Autobalancers consist of several freely moving eccentric balancing masses mounted on the rotor, which, at certain operating speeds, act to cancel rotor imbalance. This “automatic balancing” phenomenon occurs as a result of nonlinear dynamic interactions between the balancer and rotor wherein the balancer masses naturally synchronize with the rotor with appropriate phase to cancel the imbalance. However, due to inherent nonlinearity of the autobalancer, the potential for other undesirable nonsynchronous limit-cycle behavior exists. In such situations, the balancer masses do not reach their desired synchronous balanced positions resulting in increased rotor vibration. To explore this nonsynchronous behavior of ABD, the unstable limit-cycle analysis of three-dimensional (3D) flexible shaft/rigid rotor/ABD/rigid supports described by the modal coordinates has been investigated here. Essentially, this paper presents an approximate harmonic analytical solution to describe the limit-cycle behavior of ABD–rotor system interacting with flexible shaft, which has not been fully considered by ABD researchers. The modal shape of flexible shaft is determined by using well-known fixed–fixed boundary condition due to symmetric rigid supports. Here, the whirl speed of the ABD balancer masses is determined via the solution of a nonlinear characteristic equation. Also, based upon the analytical limit-cycle solutions, the limit-cycle stability of three primary design parameters for ABD is assessed via a perturbation and Floquet analysis: the size of ABD balancer mass, the ABD viscous damping, and the relative axial location of ABD to the imbalance rotor along the shaft. The coexistence of the stable balanced synchronous condition and undesirable nonsynchronous limit-cycle is also studied. It is found that for certain combinations of ABD parameters and rotor speeds, the nonsynchronous limit-cycle can be made unstable, thus guaranteeing asymptotic stability of the synchronous balanced condition at the supercritical shaft speeds between each flexible mode. Finally, the analysis is validated through numerical simulation. The findings in this paper yield important insights for researchers wishing to utilize ABD in flexible shaft/rigid rotor systems and limit-cycle mitigation.