What’s really wrong with designer babies?

Have you ever heard someone, like grandma, suggest that you should not marry someone of some particular quality? “…because then your children will come out like that!” Don’t marry a short person, or your kids will be short. Don’t marry a stupid person, or your kids will be stupid. Don’t marry a person of this or that race, or your kids will be this or that color. Putting aside the likely racism of the latter comment, is there anything wrong with taking any of this advice?
If you could take action to ensure or greatly increase the likelihood that your child would have blue eyes and be tall (or whatever positive qualities you choose), would you do it or would you think it wrong? Is this any different from being extra selective in our choice of mate? Is this any different from what we do after birth to mold and sculpt our children into the form we hope they become? For example, many parents read to their kids as infants, or only let them play with “brain-enhancing” toys. They enroll them in early learning programs for music, art, dance, and academics. They get braces on their teeth. They petition to have their child go to this rather than that school, etc. In all cases we are trying to control (or greatly influence) an outcome either by careful selection (filtering) or by careful modification. Is this wrong? Are these more common methods any different from pushing back the selection/modification process to the stage of embryo development? Or have we just gotten better at taking our grandma’s advice?