No damage to your tool.
Think about it, your tool Is High Speed Steel designed for cutting tools: drills, taps, milling cutters, tool bits, gear cutters, saw blades to machine steel
So using your tool on brass or copper is like butter compared to steel.
I prefer the look of brass
I buy brass nipples for ferrules cut them to length on my bandsaw
I thing there are some exotic woods that are harder than brass, lignum vitae for example...