I'm not sure there's a right answer, I've heard arguments either way.
I usually do a little of both.
For big areas I apply filler directly to metal.
Then I prime, so it's easier to see defects.
Then I go back and fill those.
My logic (which may be flawed) is that I typically use epoxy primer which isn't really meant to be sanded (it can gum up the sanding pads). So if it's a big area that needs lots of sanding, I prefer to do it without worrying about the primer.
When I do prime, all that's left is little spots that dont require any agressive sanding.
I'll primer one more time to get a uniform finish. Then use surfacer and work out the real tiny defects with that.