I think it might be time to move on from the property and build something new - 24 Sussex is old, decrepit, and not really suitable for the task. It's just an old house.
Preserve 24 Sussex and turn it into a museum or something, but build an official residence that actually meets modern requirements.
I think it might be asbestos-free now - they've spent the last few years gutting the place.
But yeah, I don't think I'd support tearing it down, despite the fact that the house itself supposedly isn't considered architecturally significant.