Warren, I'm not sure the file analogy really works, because you can pretty quickly remove a lot of the "brashness" from a freshly abraded stone, just by working the back of an iron for a few minutes. I wouldn't go downstairs and abrade my hard black just before I was about to take a final pass on something; I would typically do it at the beginning of a project, when there's lots of rough work to do.
It is highly likely, as you and Dave have suggested, that I'm leaving the higher levels of sharpness on the table, so to speak. That's ok with me; I never wanted to win the sharpening competition, but rather to get my tools sharp enough to work with. But there are other factors besides particle size. Abrading the stones regularly makes it a lot easier (for me; ymmv) to keep the back of the tool in good shape. And a fast-cutting stone means fewer strokes, which speeds up sharpening and makes it less likely that I will introduce the sort of geometry errors that can crop up when I have to take too many strokes.
But as I said before, I'm agnostic; people can use their oil stones however they want.