I believe the outer end of the operating sleeve would be the side that faces to the belt pulley.
That operating sleeve should have some kinda pin to keep it from rotating. Make sure this is not worn letting it wiggle around.
Wonder if your fork is in right ? I remember breaking something in there a brass piece ? in mine years ago from having it in wrong.
Have you tried other parts from another tractor ? Maybe swap the whole pulley from a parts tractor ?