I hear you on that, and understand escalation just as much as the next guy, but I won't live in fear of it. A person has no legal right to do what they are doing on either parcel of land. That is clear by virtue of the existing and hard to enforce laws.
I had the same thing at the other place, but was able to fence them out at the corner of the field, but leaving a bypass outside the fence and off our property, that solved the problem, and yes they had broke the fence numerous times with livestock in the field. That part ticks me off, the careless, selfish actions by others without any consideration to the livestock owner, loose horses/injuries/ vet bills and the work needed to repair it multiple times. I would also say that if it were to escalate here, it would eventually be regrettable for them to do so. We are not the aggressors here, they are, and though a person cannot stand watch, they are in a highly vulnerable place, and one can detect them before they get here from the house, hundreds of yards away, if you are outside or windows are open, problem is when you confront them you are surrounded, which means you have to be armed for peace of mind. I don't want any form of this kind of trouble, for any reason but the last time I stopped a crew of them, last fall during archery season they were from out of town, one was real belligerent/unpredictable and a landowner is to fear these kinds of people because of retaliation if they are told what they are doing is illegal and must stop. I don't agree with that for any reason.
For years now I've made a point of confronting everyone I saw,the majority were polite about it, a few were not. Those people are sometimes intoxicated, and unpredictable, and you never know which kind, and how many you will meet. Confront enough people and sooner or later you will meet a jerk, hardened criminal or similar. One spring early summer I tossed out 16 people, last spring, I lost count at 30 atv's in one group, thats out of control, they don't ask permission or acknowledge the signs for any reason.
I think the next step is more barricades, trees, prickers and such, repair the fence, signs, and so on, see if they want to go the extra mile, might just be a simple and more unintentionally diplomatic solution ! LOL !