In aerospace industries, to solve the problem of big exit burr which leads to the difficulty in deburring operations and poor assembly quality of fastener after the traditional drilling process of the difficult-to-cut materials like titanium alloys using a twist drill, the rotary ultrasonic-assisted drilling (RUAD) technology of titanium alloys based on a novel tool (i.e., 8-facet drill) was proposed. The principles of RUAD process and motion trajectory of tool cutter tooth in RUAD were analyzed. And then, the drilling experiments were performed with a designed RUAD spindle unit, CA6140 lathe, measurement system of thrust force, measurement system of cutting temperature, high-speed video camera system and non-contact laser measurement system, and the type and size of exit burr were studied. In contrast to the common drilling (CD) of titanium alloys, the mechanisms of small or low burr height generated in RUAD were discussed and analyzed, and the theory models of burr formation mechanisms in CD and RUAD were obtained based on the high-speed camera images of burr formation and principle analysis of RUAD. The experimental results indicate that compared with CD, in RUAD process, the thrust force, maximum cutting temperature near the drilled hole exit and burr height of hole exit decrease by 16.79%-20.2%, 18.54%-21.68% and 82.27%-89.18% respectively, which greatly reduces the troubles and manufacturing costs of deburring operations and improves the production process.