MDF and scraps of construction grade plywood/particle board etc.. go in the dumpster.

Cabinet grade plywood scraps get saved as long as they are big enough to cut safely on the table saw.

I save pieces of baltic birch regardless of size since it is so nice for jig making etc...

Hardwood scraps that are usable get saved, and the rest get cut down to burn. From time to time I go through the good scraps and purge some of them as well.