That was true about NASCAR up until 10 years ago. No matter what you think about NASCAR's brand of racing, they have done a tremendous job in broadening their fan base from the confederate flag waving, straw hat wearing, chewin' tabacco crowd. IIRC, it is the fastest growing TV sport the past 5 or 6 years and the only one whose TV rights are actually increasing in value.
Sport or not, it is very popular.