Alcatraz is a must, but book your tickets early, you can't just walk up and buy. Also, book a tour in the morning, there is less of a crowd and easier to get around everything, do the audio tour, etc. Then by the time you're done with everything you'll dodge the big crowd.
Even if the Giants are out of town, do the tour, it was like 15 bucks and well worth it.
Gotta hit up some of the piers, good food (fish) and stores there.
Bridge - rent a bike and bike across, it's pretty fun. Then you'll end up in Sausalito, which is a pretty cool town. You can then take a ferry back, very common to do so. Or you can bike it back again but most people hit up the ferry.
As Balta said Chinatown is cool to walk through. We did have some food there and it was great, don't remember which place though.
Eat dinner here and order the "The Best Damn Crab Cioppino":
http://www.yelp.com/biz/sotto-mare-san-franciscoAnd yeah it's colder than you think, obviously check the weather but you'll want some long sleeve t's and some jackets out there.