In my mind the biggest limitations are the 17 inch front wheel and the non-switchable ABS. I don't ride very aggressively on gravel and always have to keep in mind how the ABS might act on steep downhills. The 17 inch wheel limits your ability to deal with obstacles. But so far I've not found any open forest service road that I couldn't handle. If you want to do single-track, that is another story.
You said it, those limitations are in your mind...
It's all about tires and rider.
17in vs 19in wheel makes no difference regarding obstacle height because you won't be slamming into obstacles so high that it would require it. Versys has only 6in ground clearance. That is the real limitation. And I say this while I don't even have a skid plate, and I don't plan to buy one either. 17in preserves the sporty behavior.
The 424 lbs (with empty tank) is manageable. Actually not much worse than the tall KLR. Just avoid topping tank before going in the trail, otherwise it's about 460lbs you have to manage, which is actually not that much felt (until you must pick it up).
The ABS should stay ON for offroad. The only time you could wish it should be off is steep downhill which you would take slowly (abs is off below 5kph) at which angle the rear brake is very inefficient anyway. Besides, if you are practically stopped, all you have to do is turn off ignition and abs will be off. You can also disable ABS (for the current ignition cycle) by spinning the rear wheel in 2nd speed while stationary (find a patch of grass ideally...)
There are many versys off road videos on youtube, some pretty gnarly, some spending time in rivers apparently...lol
I'll say this, I could do the same offroad with a ninja, granted I could have knobbies. Even a mere shinko 705 would bring a sport bike a long way. I just wouldn't like the short suspension.