I agree; think it depends on the person. I'm similar to you in that I hold my weight in the lower areas and that's where it stays. When I lose weight it comes straight from my chest first as well. So it's not that you're doing anything in particular to make it happen, that's just how it works for you. So don't worry too much about it, I guess it comes with the territory...win some, lose some right.
