This is critical. I have a Jessem that uses a spanner wrench to remove/reinstall the inserts and it works great. And I've seen the magnetic ones, too.
But the designs with 3-4 Phillips head screws? That would be a non-starter for me. No way would I mess around with multiple screws each time I wanted to change bits.