The One Direction singer was seen picking up the US songstress and carrying her over his shoulder at rehearsals for the show, where she performed on stage, reported E! News.
"Together in the lunch line (Cheeseburgers) laughing & smiling was @Harry_Styles & @taylorswift13! :) #internationalpoproyalty," said the X Factor USA's official Twitter.
All Access Pass added: "I saw @Harry_Styles throw @taylorswift13 over his shoulder to carry her out of her #xfactor rehearsal today! #CuteFactor"
A source told E! News: "Harry is in her group. He watched her on stage going through her performance. He was sitting with her mum and the rest of her entourage. After she finished, he went up to the stage. They were joking and laughing. He picked her up and threw her over his shoulder and carried her off."
The pair had been linked together earlier this year and were apparently dating briefly before Harry was photographed with model Emma Ostilly.
He has admitted they are friends, telling Australian newspaper The Herald Sun: "We met in America, she's very nice, yes. She is very talented... She's very nice... We're friends."
But he also revealed that he does not quite live up to his reputation as a ladies' man, telling Now: "I do have friends - and I'd say that more than 90% of the people I get linked with are my friends. At the moment it appears that I have, like, 7,000 girlfriends. It's just ridiculous."