The "click" "click" can be the sound of a low battery as others have suggested, but it can also the be sound of an incomplete circuit. Since you pulled the starter already, you might have introduced a short at the bell housing. Make sure that surface is clean. Maybe there's even some ice in there, preventing good surface contact.
Before you think that the health of the starter is in question, try jumping directly to the starter. Your battery may have less juice than you think.