They don't have a reasoned argument. They just don't like to pay. That's it. Let someone else pay. As far as they are concerned it is free land and it's already paid for, and it doesn't cost anything to maintain.
The reality is it does cost money to maintain and the question is should taxpayers pay for 100% of the cost even though they may not derive any benefit from it? Formerly, FS lands were essentially all regarded as uncut lumber. The public in general benefited from its stewardship because affordable lumber benefits everyone. Now that many stands have been set aside for recreation, this is no longer the case. The cost benefit equation is now skewed in favor of those who actually use the land for recreation and against the vast majority who pay taxes, but don't use the land to recreate.
Mark mentioned the RV industry. They have demonstrated an ability to get owners to pay license fees to raise money for the high cost facilities that are needed for their activities. It gives them a lot of clout that non-motorized users don't have.
I don't like user fees any more than most of you do, but I realize that it is a lot more complicated than just saying NO!