I live in the DC metro area and almost never actually drive downtown. It just is not worth the hassle, especially when you can take the Metro (subway) which is relatively clean and fast. During rush hours it too can be crowded.

If you are staying at a hotel or with friends in the suburbs leave your car there and take the Metro. Buy a ticket with enough money on it (say $20) so that you do not have to get one for each trip.

As with most American cities, you are probably OK except in "bad" parts of the city, and at night if you stay where there are other people around.