If I eat really unhealthily, but don't gain weight, is it doing the same amount of damage?

I eat really crappy foods but have a freakish metabolism...so basically I don't gain any weight. I was just wondering is it still as bad for you? Like obviously it's not good for you, but is it equally as bad as if you do gain weight?