Wrecks definitely need to be cleaned up. Any fuel or cargo aboard a vessel could be detrimental to biodiversity. The difference is how quickly life will take hold on the wreck.
Let's consider a wreck of some kind—or any object, really. I'll give you an example: In the Saguenay fjord, a communications tower fell on a sandy bottom. There's hardly anything on the sandy bottom, but, when you look at the tower, it's hard to see the steel parts because there are things living on it. There are lots of living things, including anemones, crabs and squid.
As a recreational diver, I've never come across an accidental wreck or one that was only recently sunk.
In Ontario, near Brockville, Prescott or Kingston, there are wrecks that have been under water for over 100 years. The fuel at the time was coal, so there's a little more coal than diesel or fuel oil under water. These wrecks have been there for a long time, so life has returned.
One thing we know for sure is that the more fuel or pollution there is in the water, the longer it takes for life to take hold.