Miley stripped to a flesh-toned underwear set and twerked against Robin Thicke, who joined her onstage for a duet of his single Blurred Lines, with many dubbing her antics "inappropriate".
According to Radar Online, Liam was baffled by the performance and is thinking of ending their four-year romance.
"Liam really does care about Miley, but her racy new look and sort of ghetto attitude isn't exactly what he signed up for," a source told the outlet.
"Her VMA performance absolutely mortified him and he couldn't be more embarrassed. He doesn't really see anything sexy about what she did on stage. It's pretty much over."
Liam was not present at the Video Music Awards.
Reports of his embarrassment follow months of speculation the couple's relationship is on the rocks.
Miley's manger spoke out today to address rampant criticism of the 20-year-old, revealing the star's mother cheered her on backstage.
"[Miley] couldn't have been happier about the performance," Larry Rudolph told People magazine.
"We were all cheering from the side of the stage. It could not have gone better. The fans all got it. The rest eventually will."
Miley shows no signs of toning down her demeanour. The singer recently recorded an entire song dedicated to her fascination with twerking, a style of dance often associated with strip clubs.
The duet, which features Justin Bieber and rapper Lil Twist, was leaked online today and features Miley singing: "I came up in this party time to twerk."