A very good friend of mine had a single mastectomy and then started working out. She's been a gym rat for about 3 years now (cancer-free for 6 years!). If you like, I can ask her if she felt anything similar.
I can tell you that she found/finds it irritating that she is lop-sided now!

Silicone doesn't shrink. You might be lucky...
And if you're in the gym, and had the double mastectomy, may I offer my congratulations on winning the battle!