I finally got around to pulling the shifting tower off. Here is what it looks like underneath. The "plate" I'm pointing to had a nut that tightens. When I looked at it it was very loose and the plate freely wiggles around. I assume this is a major part of my problem?
I get that I'm not using the correct lingo but I'm trying my best. Thanks for your help.