I don't know why Baker felt the need to bring this issue up, but I don't see why everyone automatically dismisses what he says. The sun is going to affect white people more than it is black people. Black people have more melanin on the surface of their skin, which will help keep the UV rays from damaging the skin (heating it up, causing sweat etc). However, people originally from colder and darker climates have lighter skin to let in more UV rays, so that they get enough vitamin D. It also leaves them vulnerable to skin damage (my fellow Irish know what I mean).
I don't think the difference is that great, though. Many white baseball players, golfers, football players, etc have thrived during the daytime.