Abstract: To transfer torque effectively, spline coupling is widely used in rotating machinery, and it can be adopted in helicopter transmission system. Theory analysis on stability of the supercritical rotating shaft was conducted. And the corresponding experimental verification on the helicopter tail transmission shaft was developed. By installing and uninstalling O-rings on both ends of the spline, which were used to lube grease airproofing, the work state of the spline could be altered, and the conversion of the relative stability to absolute instability of helicopter tail transmission shaft could be realized. The agreement between the theoretical analysis and experimental verification indicates that the inner damping due to friction in the spline and on the supporting surface of both ends of the spline is the springhead of self-excited vibration and the stability of the supercritical shaft can be meliorated by improving the work state of spline, and they can lay the foundation of the research about the supercritical helicopter transmission shaft design.