I'm assuming you have a Simms inline pump? If so, is the cap you're talking about the rubber boot, #57 in red highlight?
The rod in question is #43, and is pushed out by spring #44. Either the spring is broken, and/or the rod is sticking in the internal bore due to lack of lube. Spraying some penetrating oil inside may help to free this up. Changing the oil inside the pump would go even farther.