there's nothing "unfortunate" about your kids not being climbing "phenoms", but regarding the considerations:
yeah there are. it's something i personally wrestle with, having a little one involved with the game/sport/hobby.
it's pretty obvious that we, as humanoids, are descendantly/biologically more evolved to deal with things like branches and other large grabbables, vs the tiny crimps that modern rock climbing is mainly about. as a result, kids do tend to have a few climbing-specific injuries, such as growth plate fractures in fingers.
I think the key is restraining them...I try to make sure they climb no more than 3 days out of any 7...i have had many arguments with people about how they "should" train more times a week...Personally, i think that's asking for it...the stress they put on their fingers is ENORMOUS...