Obama aides said they reached out to Mr Romney's team shortly before Thanksgiving to start working on a date for the meeting.
The two men will meet in the White House's private dining room, with no press coverage expected.
In the days after his loss, Mr Romney told top donors that the president was re-elected because of the "gifts" Mr Obama provided to blacks, Hispanics and young voters, all of which are core Obama constituencies.
Many Republican officials, eager to move on quickly after the loss, disputed Mr Romney's comments and urged the party to focus on being more inclusive.
White House spokesman Jay Carney said Mr Obama was looking forward to having a "useful discussion" with his former competitor. But he said there was no formal agenda for the lunch.
While in Washington, Mr Romney will also meet with his former running mate, Wisconsin Representative Paul Ryan, according to a Romney campaign aide.