Hi guys,
I am constantly having the same problem, and while I figure it's due to the way I'm feeding it, perhaps it's also compounded by my saw blade.
It does seem to go away with a featherboard, set starting slightly before the blade and extending to the middle of the blade area. But if I don't use the featherboard, it starts drifting, unless I lean over and hold it steady from the side as it first passes the blade.
Now, I just have a cheap Ryobi portable saw that I use for general work. I've noticed about the Ryobi, that when you start it, the blade seems to jerk into position a bit. Maybe it's the junky nature of the saw, and blade doesn't stay steady. I don't seem to remember having as much trouble ripping on my older stationary Craftsman contractor saw. It's hard to find a good clamping surface on this Ryobi too, so attaching a featherboard for certain cuts in hard to do sometimes.
Do other people find their portable saws pushing the board away from the fence also? I find myself often allowing a 1/16 or so extra on cuts to accomodate for the saw slightly pushing the board away and cutting it narrower.