It's not really about "taking it like a man," as you put it.
I had mine done on a Friday afternoon. Two soft, two partially impacted. No big. Went back to work the following Monday at the Opera warehouse, and was out the door and back home within an hour or two. I don't think I went back to work until Wednesday or maybe even later. If you do anything too physical too soon after having them yanked, there's a chance you'll bleed so much that there's not enough gauze in all the world to keep it under control.