Thanks everyone for the feedback.

I have a related question. For sanding after the first coat of poly I understand that it is just supposed to be a scuff sand of the surface for adhesion. How much sanding is that? I was working on it this morning and I used a 320 grit on a sanding block and with a low amount of pressure went over the surface 3 or 4 passes in the direction of the grain. Is that sufficient? Its not the smoothest but I was going to wait until after a couple more coats before smoothing out more aggressively.