Personally, though, I don't really say "How many people are there in your family?"
Instead, I'll usually ask about the "pieces" - Do you have any brothers or sisters? or Do you have any children?
I agree. It's not a question I ask either. It's probably a cultural thing and as an American, not really something I care about.
I think it might be because Japanese people tend to live with their parents and grandparents.