Recently, research on new techniques for dissipative mufflers in dealing with the higher frequencies has been addressed. However, the shape optimization of hybrid mufflers in reducing broadband noise within a constrained space as well as a pressure-drop limit which are both concerned with the necessity of operation and system venting in practical engineering work was rarely tackled. Therefore, this study will not only analyze the sound transmission loss (STL) of a space-constrained multichamber hybrid muffler but also optimize the best design shape under a specified pressure drop. In this paper, the generalized decoupling technique and plane wave theory used to solve the coupled acoustical problem of perforated mufflers with/without sound absorbing material are presented. The four-pole system matrix used to evaluate acoustic performance is also introduced in conjunction with a genetic algorithm (GA). A numerical case for eliminating a broadband venting noise emitted from a pressure relief valve using four kinds of hybrid mufflers is also introduced. To verify the reliability of the GA optimization, optimal noise abatement for a pure tone (1000 Hz) is exemplified. Before the GA operation can be carried out, the accuracy of the mathematical models need to be checked using the experimental data. The optimal result in eliminating broadband noise reveals that the overall noise reductions with respect to various mufflers under a maximal allowable pressure drop of 100 Pa can achieve 62.6, 54.8, 32.3 and 87.8 dB. Consequently, the approach used for the optimal design of the multichamber hybrid mufflers under space and back pressure constrained conditions is indeed easy and quite effective.