Villagers and historians are banding together fearing an important part of Harefield's history is under threat.
Author Eileen Bowlt has written many books on local history, including one called Ickenham and Harefield Past, which outlines the background and attraction of Grade I listed Breakspear House, in Breakspear Road North.
Mrs Bowlt said: "When I looked at the house recently I was horrified by what I saw and immediately contacted English Heritage and Hillingdon Council.
"There was scaffolding all round the house, but no work was being done and the windows were broken or open.
"The lodge at the entrance was in a shocking state; half the roof was off, and part of the back wall has been down for several months. The middle lodge is in a bad state and the dovecote has a great crack in it."
The Breakspear family owned the estate from the 13th century until the 15th century, when it passed to the Ashby family.
There were notables in both families: Nicholas Breakspear was the only English Pope as Adrian IV (1154-1159) and George Ashby was the Clerk of the Signet to both Henry VII and Henry VIII.
At the end of the 19th century WS Gilbert, of Gilbert and Sullivan fame, lived there, and in 1956 the house became an old people's home until it closed in 1987.
Gordon Isaacs, chairman of Harefield History Society said: "Our local society is most concerned. People here are proud of their heritage and this seems like a desecration of it."
Jean Palmer, director of planning and community services, said the council was working with English Hertiage to develop the site for housing, and added: "This scheme will provide for the restoration of Breakspear House, the other listed structures on the site and their wider landscaped setting.
"Work on the site has been delayed as the developer wished to make some alterations to the approved scheme.
"The developer has now submitted revised plans and, when they have been approved, it is hoped work will resume at the site."
Mrs Bowlt stressed time is of the essence. She said: "It is absolutely essential that work is done soon as the longer it is left the more deterioration will take place and there is more danger of vandalism."