Um, please tell me this statement is a simplification
From my background it seems that relying on 'perfect' data from peers is a very dangerous assumption.
You are correct this is a simplification.
In no way do we rely on perfect data.
But there are cases where data that 'seems' valid is accepted by peers running older versions.
This could 'confuse' these older version peers in accepting a block or transaction that does seem valid in the older version but is not accepted on the newer versions, this leads to a fork of the network.
Eventually if the target of the 'main' fork is higher the older peers could possibly switch back to the other fork but for this to happen all bad blocks and transactions have to be deleted and re-downloaded (automatically).
Damage is probably not the right word, better way to put it is that it causes unnecessary negative side effects (like unnecessary re-downloading or unnecessarily blocking of otherwise non-evil peers).