The music mogul who reportedly fainted and fell down a flight of stairs as he went to get a hot cup of milk was seen wheeled on a stretcher by paramedics.
The 58-year-old was believed to be in a neck brace as he was rushed to hospital on Friday (October 27).
It is thought his three-year-old son Eric was in the house at the time of the incident, The Mirror reported .
Sadly due to his injuries he won't be on the ITV show due to air on Saturday (October 28) night.
However it is hoped he will return to the judging panel on Sunday (October 29)
Speaking exclusively to the Mirror on whether he could be taking part in Saturday night's first live show, Simon previously revealed: "I will be back but I don't know about tomorrow."
As a result the X Factor show has been re-planned with Louis Walsh's Boys and Sharon Osbourne's Girls singing for the public vote on Saturday night, while Simon's Groups and Nicole Scherzinger's Overs will have an extra day to prepare for Sunday's showdown.
In the show's 14-year history Simon is yet to miss a live show apart from when Gary Barlow replaced him for two series.
Girlfriend Lauren Silverman is reported to be jetting back to London, cutting short a visit to New York to see her older son Adam.
Sources said: “It’s been a scary morning. It was very early and Simon was going down his stairs to get some hot milk because he couldn’t sleep.
“An ambulance was called and he was stretchered out of the house in a neck brace surrounded by worried aides.”
Simon gave a thumbs-up to concerned fans as he walked back into his home, The Mirror reported.
He said: "I fainted and fell down the stairs. I've got a bit of concussion. I feel better now."
This year's wildcards who will get a second chance include Hounslow's Talia Dean along with Leon Mallet, Jack and Joel and Alisah Bonaobra.
The X Factor continues on ITV Saturday at 8.15pm.
Keep up to date with the latest news in west London via the free getwestlondon app.
You can even set it to receive push notifications for all the breaking news in your area.