2. I'm not sure I understand your last point. Even if a small change pushes the barely tolerable to intolerable you still have the opposite effect on the other side of the curve, where it provides relief. I am assuming here that in most polygenic traits the curve is not so narrow that there aren't dysfunctional individuals being produced on both extremes.

3. I concede your point that if there is less than perfect competition, this effect doesn't completely negate any effect on average welfare. It would still make such an effect smaller and less relevant as compared to other considerations, like species composition. 

I regret not having numbered my points above :P

2. 

A change in either direction will put the average away from what the individuals are best adapted to on average and under a symmetric assumption (like approximate normality), I'd expect it to hurt more individuals than it helps.

Suppose a human in a t-shirt and jeans, going for a walk. Preferred temperature: 24 °C. The weather turns and the temperature drops to 14 °C. Is this person half as physiologically stressed and half  as miserable as they would have been if the temperature had dropped to 4 °C? I don't think so. The response is not linear. In my original example, most individuals would not end up too far from their preferred temperature. Wouldn't welfare gains and losses be mostly driven by the extremes?

3.

they're just part of the environment and conditions under consideration

They could be, depending on the species, moral patients.

Some objections:

1. you're conflating fitness and welfare. Counterexample: a high resources high predator environment were being super paranoid and stressed out all the time gives you high fitness and poor welfare.
2.  most traits are polygenic and, therefore, normal-ish. Changes in the environment (e.g. higher temperature) increase fitness on one end of the distribution (e.g. the more heat tolerant individuals are doing great) while decreasing it in the other (e.g. the less heat tolerant are doing badly). What are the total welfare implications of that?
3. you're assuming a single species and an abiotic environment, but in reality almost any system has many species. And these species often respond individualistically to changes in the environment (e.g. one species is limited by temperature while another is limited by food and is indifferent to temperature within the relevant range). One species decreasing in abundance is an opportunity for others who now find themselves in abundance.
4. a regularly changing environment might select for adaptability (e.g. through phenotypic plasticity), even if the changes are not cyclical. Changes might be dealt with without decreased welfare.

[edited to add numbering]