Without going too deeply into polishing esoterica, I agree, although I'd say deleterious to the edge rather than to the hardened carbon steel itself.
What you're describing, the continued scratching of sub-micron diamond on tool steel to polish an edge, is where the return on diamond diminishes and the advantages of oxide media start to kick in.
Gem faceters, especially those who facet for judged competition—a whole 'nother level from sharpening—distinguish between gem species polished with diamond and those polished with oxides. Much has been written about the two differing polishing paradigms—and I use polishing to describe the final finish on steel cutting edges and gem surfaces—and which gem species respond best to each.
Diamond conforms to the ever finer scratch paradigm; oxide polishes to the Beilby layer flow paradigm*. For those unfamiliar with Beilby flow, it explains, among other things, why some species of crystalline and non-crystalline materials can be polished by oxide polishes that may be softer than those surfaces being polishing; in other words, why iron oxide—jeweler's rouge—can polish the outermost molecular layer of hardened tool steel, rearranging and disorienting its crystalline alignments into a smoother amorphous surface.
Depending on the steel and its hardened and tempered crystalline state, one may determine a point where shifting from diamond to oxides may be desirable. As a rule of thumb, I stop at 1µ diamond and shift to .5µ chromium oxide.
Fast is where it's at for woodworking tools—600-grit polycrystalline Eze-Lap to 1µ diamond paste, then optionally, to .5µ CrO2.I like one medium stone and one modern 1 micron stone. Super nice to use, and super fast.
*Aggregation and flow of solids; being the records of an experimental study of the micro-structure and physical properties of solids in various states of aggregation, by Sir George Thomas Beilby, a real page turner—well, at least for some of us.
Download at http://archive.org/details/aggregationflowo00beilrich