Annular labyrinth seals often have a destabilizing effect on pump rotordynamics due to the large cross-coupled forces generated when the fluid is squeezed by an oscillating rotor. In this study, several novel groove geometries are investigated for their effect on the rotordynamic coefficients of the labyrinth seal. The groove cavity geometry of a baseline 267 mm balance drum labyrinth seal with a clearance of 0.305 mm and 20 equally spaced groove cavities was optimized for minimum leakage. From the pool of possible groove designs analyzed, nine test cases were selected for maximum or minimum leakage and for a variety of groove cavity shapes. The rotordynamic coefficients were calculated for these cases using a hybrid computational fluid dynamics (CFD) bulk-flow method. The rotordynamic coefficients obtained by this method were then used with a rotordynamic model of the entire pump to determine the overall stability. Results show that labyrinth seal’s groove shape can be optimized to generate lower leakage rates, while the effects on dynamic properties are only minimally changed. If the seal dynamic response needs to be modified in addition to targeting a lower leakage rate, for instance, to exhibit increased damping values, then the leakage rate and the damping coefficient need to be set as objective functions in the optimization loop.