If its on your property you can do whatever you want to it. I will say though, I am the guy putting up the fences around my place. After kids kicking balls into my wifes car, loose dogs pooping all over my lawn, trash blowing into my yard... and the major thing that made me mad, a big tree they never maintain coating my whole front lawn 6" deep in leaves and me and my wife out there cleaning it up for hours (I don't have any trees on my property) and the neighbor walking by just looking at us and not offering to help or even apologizing... can't really talk to them because my Spanish is not so great. I'd put a block wall with razor wire and automatic turret guns around my house if I could... For now I settled for a mesh fence. Works pretty good to keep the leaves and trash on their property. When the dogs come around and poop all over I just fling it into their cars (when the windows are open) or onto them. Some people have no consideration (not saying this about you of course, I don't know you) and sometimes you just have to put your foot down and fight back. "Do unto them..."
I feel slighted when people mess with my property, and if someone put a fence (or anything else for that matter) on my property without talking to me I'd remove it. Just be SURE its your property before you mess with anything. On the other side of the coin, if these people touch my fence which I've definitely put up on my own property I'd break their legs.
Neighbors, can't live with em', and you can't live with em' haha
My next house is going to be out in the sticks somewhere.