If you are going to drill holes and that"s it, you could get by with a drill press, assuming smaller holes. Most drill presses don"t have a slow enough RPM for bigger bits.
If you need bigger holes and need to machine slots, mill flat/parallel/perpendicular surfaces a mill is the only way to go. Once you get a mill, you will wonder how you went so long without one.
I was looking for a Bridgeport but I fell into a deal I could not turn down, so I ended up with one of the Chinese style imports. Its maybe as good as a 30 year half worn out Bridgeport, but for what I paid I was happy. To be honest I didn"t pay anything, gave a guy a two hours of consulting on a job he was bidding and he said hey do you want this and we"ll call it even.
I am still looking for a big Bridgeport, but this one has done 99% of what I have needed so far. Plus if I need something big, a good friend owns a machine shop. It also helps to have a dad, uncle and grandpa that are all machinists, when a hack like me gets in a pinch.