The peat moss maybe your trouble. It is not the best for nutrients.
I do not think you still have any livestock. So get a load of manure NOW. Put it in a pile. Add your yard waste the rest of the summer, grass clipping now and leaves in the fall. If you have a loader turn the pile once every thirty days. IF no loader, shovel time. You are just making your own compost.
Then this fall cover the garden with half of it. Leave the other half until spring. Then work your garden like normal. When planting make your trench and put some compost in the trench, not a lot just a few inches. Then take your hoe and drag it through/along the bottom of the trench a few times. Mix the compost with the dirt so it is not too hot for your plants.
This will give your plants the best of both worlds. The compost for nutrients and the dirt for their roots.
Then I put some compost around the plants as they grow and work that in with a hoe. A year or two of this and you will have a much better garden. Truthfully most pastures are short on nutrients. I would bet that you garden's soil is not that good.
If your garden is low and you can stand some additional dirt. Find a farmer that feeds round bales of hay in one spot in his pasture/lots. If he has done this for a few years that is some great dirt. It has some manure for nutrients and the left over hay for organic matter. Get a few loads and cover your garden with that dirt. You will have much better luck.
I filled some flower beds a long a new side walk that way a few years ago. I put ONE tomato plant in one of the beds. I got over fifty tomatoes off that plant the first year. The wife's cannans grew 7-8 feet tall too.