This is a great question. In short, no, it does not matter. Most of the time "clean eating" is for the sake of more volume, staying on track, etc. All the benefits that you mentioned.
There are some arguments for QUALITY of protein, lack of pesticides, etc. If you pair these things with a extreme training program then there might be something their to the importance on a small scale. There are some studies to support it.
The biggest picture comes in with the relevance of clean eating is how in shape you are in at the point of starting this eating.
A recent study in Journal of Clinical Endocrinology & Metabolism shows some interesting information on the aspect of free radical production in the body in both normal and obese subjects.
Prolonged Reactive Oxygen Species Generation and Nuclear Factor-{kappa}B Activation after a High-Fat, High-Carbohydrate Meal in the Obese -- Patel et al. 92 (11): 4476 -- Journal of Clinical Endocrinology & Metabolism
When healthy subjects took in high fiber and high fruit compared to high-fat, high carb fast food, they showed no oxidative stresses, but the obese did.
There are other studies that show digestive weakness, protein uptake, etc when in terms of refined foods vs whole.
Some have more protein intolerances and need higher quality and more variety all the way around.
I will say the most are HIGHLY ANAL and that for the most part "clean eating" is really more about the fear of bringing forth the return of the "fat loss" that didn't really happen. For example, if you go from eating really really clean and having no carbs and sodium to cheating on a fast food meal it is going to blow all that figment of "fat loss" you had on a real large scale. I tell my clients to salt more, to not eat so so clean, i think it is important to not have the the internal fear embedded into about eating a bagel and blowing up like a balloon....
/end rant.
So I hate to end this on a note of it depends, but like with most things it does. If you can get away with it than great, if you can't than you can't. I can say that the more you are participating in a solid training program, unless under extreme conditions, the less this matters. Normal body composition and strength training on a enthusiastic level needs to really not worry. Those who are overweight or those who are involved in more extreme training and athletic conditioning it may change things for on a smaller level.