I would guess, without seeing it in person that the shifter fork is distorted/bent from the pivot pin coming out on one side of the torque tube some time or other. The bearing is pressed onto the casting and the casting has flats that fit snug in the fork. It should never jump out. Also the casting has to move without drag when the clutch is pushed down. Something is either missing or bent out of shape.
Best I can do without being there is post the drawing.