Simon Cowell appears to have lost his pulling power, with new figures showing almost four million viewers departing the X Factor since he was previously on the show four years ago.
The show boss bowed out after the 2010 series to launch a US version, but returned this year in an effort to give viewing figures a boost.
However, latest figures show a huge drop compared to the equivalent sixth programme of the series during his previous stint as head judge.
At this stage in 2010, the show drew an average audience of 11.6 million on ITV. Sunday night's show had 7.9 million viewers - a drop of 3.7 million.
Many had put the decline of the show's fortunes on the departure of Cowell and fellow judge Cheryl Fernandez-Versini (formerly Cole), but figures since their return suggest they have not lured viewers back.
Last year, the sixth edition of the show was more than a million viewers higher at 9.6 million.
It is also down on 2012 when the same show had an audience of nine million, again including the viewers of the catch-up programme.
However, there was some comfort for ITV bosses as the programme was almost a million up on the Sunday edition a week earlier, which had an average audience of 7.6 million.
However, that show had competition from rival programme Strictly Come Dancing when they had a head-to-head clash.