To improve the dynamic environment of a spacecraft, an octostrut vibration isolation platform is designed to replace the payload attach fitting, which can significantly attenuate the vibration transmitted to payload except that an extra isolation frequency is introduced. However, it is found from the experimental results that when the excitation amplitude is lower than a certain level, the first resonance occurs at a higher frequency (i.e., the isolation frequency). This is caused by the nonlinearity that is a result of the friction existing in every actuator. Therefore, in this paper, the friction force of the single actuator is taken into account and is described by a bilinear hysteresis model. With this friction force model, a new nonlinear model of the octostrut vibration isolation platform is established. Meanwhile, the harmonic balance method is used to solve the nonlinear equations. The theoretical and experimental results indicate that the friction plays an important role in the performance of the platform. It is identified from the study that to ensure the performance of an isolator, in its design, either the maximum friction force or the minimum excitation should be restricted.