Another reason for deterioration of sound quality could be, that the loudspeaker is part of a multi-way design, where each range of frequencies has it's own chassis. Then, altering the impedance of loudspeaker can result in a change of crossover frequencies of filtering, with the consequence that certain frequencies are becoming louder or softer, deteriorating the balance of music. In such a case you can only exchange the loudspeaker by a fresh one, of same type of course!

Kai