The 'smell' of a cigar is not harmful. It's only the active smoke eminating from a lit cigar that MAY be harmful - in prolonged doses. The remnants of an old cigar may be unpleasant to smell, but like a fart, it's not going to hurt anyone.
Think of all the 'smells' your child is exposed to every day, including carbon monoxide from the car you drive, pollution, barbecue smoke... Your dad's cigar, once in a while, even if he is smoking it when the child is around (yeah I know I'll get flak for saying this), is not going to do any more harm that any of the other zillion things us humans are exposed to. We live on earth, not a sterile bubble, and life sometimes smells!
Now if dad is shutting all the doors and windows tight, smoking 6 stogies in a row, and blowing it in your kids face... yeah, I'd say you have a right to ask him to stop. But if he's not smoking in the room with the baby while he's taking care of your child for you (no small task for a grandpa) and you repay him by saying his house doesnt smell April fresh... I'm not surprised he would be upset.