On adventure climbs people are more inclined to leave things as they are found, at least in part because of the constraints of time, but also because there are fewer "bystanders" to be concerned about. Obviously, you have to avoid trundling on your belayer. So it comes down to whether the follower feels like doing a good turn by trundling an occasional loose hold.