Could be. The sound power output of this set is only in the hundreds of millwatts, so an efficient speaker is necessary to get acceptable volume without distortion.
If you can find a decent set of high-impedance headphones, give them a try (without the output transformer). If the sound is clear, chances are your speaker setup is flakey: inefficient speaker, or impedance mismatch.
If the sound using the headphones is also poor quality, I think that I would look for faults in the detector stage. A bad tube or incorrect biasing here would cause audio distortion.