The battery charger can't supply enough amps to get the job done, so it's not a good test.
Use your volt meter to check where you do and don't have the voltage. Put the meter across the battery posts and hit the starter. If the volts stay up, move one volt meter lead to the next connection and try again. Once you find where the volts drop under load, that is where the problem is.
A prime suspect would be the starter switch, but test before replacing.