It's not at all surprising that most snowboarders feel awkward in hard boots, since nearly everyone learns in soft boots. Only the rare hardboot racer/carver would feel comfortable in hard boots right away outside of the ski area. Most of the rest of us struggle to figure it out over only a handful of days each year, usually without ever riding hard boots at the resort to dial in the foot positions and technique, both which are different than for soft boots.
A mountaineering style boot with hardback bindings is an okay compromise, although it does sacrifice some edge control in climb mode, and can result in calf fatigue or chafing/soreness when edging for extended periods (many popular mountain routes). A velcro shin strap on the highback might be useful to control this. I'm not sure how much a new 'snowboard mountaineering specific soft boot' would differ from this option.
There are some advantages with skis (see Tony's post for one ), but like you say, we love snowboarding for that dynamic surfy feeling, and skiing really isn't the same. However, there are certain conditions, frequently found above treeline, in which you can't really open it up and surf anyway--sometimes it's hard enough just to manage that single, deeply sidecut edge on firm slopes.
Then again like I said I think a pro hardbooter might not have so much trouble. So while I agree that the non-lift snowboarding technology should improve, it's also true that most riders will need to adapt to what is almost a completely new sport. There are a lot of mountaineering situations where it is questionable whether one should really expect to obtain a surfy softboot feeling like you can get on mellow slopes, or bombing powder lines out of a helicopter.