Yeah, "ask"

In case law enforcement ever tells you to delete a photo from your digital camera – or demands that you hand over your film in the name of security – here's what you're allowed to do.

If you are in a publicly accessible place – a sidewalk outside a stadium, for example – you're allowed to photograph anything you can see with reasonable equipment (that said, don't set up a giant telescope in the middle of the street and try it), and use it for informational/reportorial purposes (you may or may not be allowed to sell such a photo to a non-news organization; that's very much up to interpretation).

You will need to understand what is a publicly accessible place. If you're going to the ballpark, you are no longer in a publicly accessible place once you step onto ballpark grounds (including a parking lot).

A street is a public place, unless it is a private way.

Even municipally owned spaces might be subject to private rules if they are being rented out, or if they charge admission.

So be aware of what's going on around you, but don't cave in to people being jackasses for the sake of being jackasses.