Midfielder Ander Herrera has admitted Manchester United need to be more ruthless to end the suffering of their fans.
The travelling contingent at Newcastle on Wednesday night had to wait 89 minutes for their reward, when Ashley Young fired home a last-gasp winner to finally kill off the Magpies.
Home keeper Tim Krul had earlier made a string of fine saves - opposite number David de Gea had to be equally vigilant - to push the visitors all the way to the wire before they claimed a 1-0 victory.
Herrera told MUTV: "I think we deserved to win. We deserved to score before and we were a little bit disappointed at half-time because we should have been winning.
"We created chances, we kept the ball and we didn't give a lot of chances away - I think only one in the first half, in the last minute. But now we are very happy.
"When you win in the last minute, you are very happy. But I think we have to improve, we have to score earlier because our fans are suffering too much."
On an eventful night at St James' Park, United enjoyed the better of the possession but were unable to make it tell, as skipper Wayne Rooney uncharacteristically passed up two first-half opportunities and saw Krul save from Marouane Fellaini and then, in the space of seconds, Fellaini and Young.
Newcastle were aggrieved not to have been awarded a 10th-minute penalty for what looked like a clear trip by Chris Smalling on Emmanuel Riviere, and the first half ended in controversy with Jonny Evans and Papiss Cisse appearing to spit in each other's direction after an off-the-ball confrontation.
The Magpies looked to have battled their way to a hard-fought point but, with seconds of normal time remaining, Mehdi Abeid stabbed an unwise back-pass towards the unsuspecting Krul, who could only deliver the ball to Young's feet, and he made no mistake with the goal at his mercy.
Victory proved all the more sweet for the visitors with top four rivals Arsenal, Liverpool and Tottenham all collecting three points on the night, as Southampton had done 24 hours earlier.
Herrera said: "It felt much more important after the game than before. When I was looking at the results of our rivals I thought it was a very, very important victory because Arsenal won, Liverpool won, Tottenham won, Southampton won.
"You know the battle for the top four is going to be very hard but I think we are in the right way, because we are winning the games through [possession of] the ball and creating more chances than our opponent."
United's elation was in stark contrast to the despondency they left behind on Tyneside, although Newcastle head coach John Carver was able to take positives from a gritty display, and the return of midfielder Jonas Gutierrez to action following his recovery from testicular cancer.
Gutierrez was introduced as a 65th-minute replacement for emergency left-back Ryan Taylor and received the warmest of ovations from a crowd of 49,801 as he was handed the captain's armband by close friend Fabricio Coloccini on his arrival.
But the 31-year-old, whose contract expires at the end of the season, is not content just to make one cameo appearance.
He told nufcTV: "I am ready to play. It was my first game and I always look forward. I want to improve and I am going to work hard for that.
"You can have opportunities, then it depends on you that when you have the opportunity, you don't miss it. It was my first, it was incredible, it was exciting, but I want more and I am going to work for that."
It was Gutierrez's first Newcastle appearance since October 2013 and his reception was fitting.
He said: "It's been a long time, a long wait. I am really pleased to be back. I really enjoyed the welcome of the fans.
"I enjoyed it a lot and I have to say a special thank you to the fans for all the support they sent me through my illness and the welcome that they gave me today and against Aston Villa as well."