Yes, its like someone wanting to steal your car, if they want it, likely it will happen.
More or less a more substantial deterrent, anyone could pack tools, a saw or what have you to get in, there is no doubt about that.
I find the riders whom appear to be "smoked up", seem to just be out enjoying a ride, you can either smell it or see their bloodshot eyes. I don't think anyone would be foolish enough to sneak in and grow anything of significance either, would be too much work to get anything out of it. People do try on a small scale, as I have seen in years past I have found amateur attempts, 1 small plot, even found some plants in the middle of a large field behind us a long time ago, while walking along just happened to look down, couple of transplants of some afghan variety according what I was able to look up, brush hog got those anyway, they would not have amounted to much.
North of us, and that would be a concern like you said, they reported taking 6000 plants. In that case you don't want to run into those folks by mistake, as they would likely be armed and dangerous protecting their interests, anything is possible here, but I think there is too much aviation traffic nearby, someone would see anything of any size, but up there its rural and a lot more places to hide. That's a job for the law to deal with for sure !