Collets are an overlooked wear item that should be cleaned frequently and replaced when worn.

Bits can be out of tolerance too. I once had an Amana bit show up so bad that I could see something was off right out of the package. The shank measured something like .025" under 1/2". The collet would never have held it securely.

If you really have to reef on your collet take a close look at it. Manufacturers usually provide wenches long enough to tighten their collets properly with normal torque.