You're assuming people would drive on the day they didn't buy gas.
I agree, however, that not buying gas for one day wouldn't do anything to the prices. This is also why, as akhalteke suggested, a week long use boycott might make more sense. But, it's more about statements and personal experimentation. After that one day (or week) boycott, maybe people will think about how they use their vehicle and find ways of becoming more efficient.
As far as the workers are concerned, well, many gas stations sell more than just fuel, and many people go to those stations for more than just fuel. My guess is most small stations make the majority of their profit off of selling these 'other' items, and not gasoline so their profit margin wouldn't be hurt much, if at all. In fact, I'll be sure to stop by my local convenience store on my bike and pick up a nice cold drink just to make sure I'm not hurting any of the little guys.