B&W or Color?

Typically when photograph I find myself wondering whether or not I should leave things in color or if I should take the color away. Unless a client specifies, very rarely do I leave photos as they are. I find myself loving the hidden story behind what color tends to hide.