Designer feels having another child would be "unfair", but David anxious for one more.
The 39-year-old fashion designer is married to retired soccer star David and they have four children together.
While David is said to be eager to add to their family, Victoria is concerned the other kids would suffer as a result.
"David would love another baby. He always said he wanted another child and, now he's retired, he thinks it's the perfect time to start trying," a source told British magazine Closer.
"However, Victoria has her reservations. Her career takes her all over the world and she already feels guilty her four kids don't see her enough. She knows a newborn would make it even harder to give the other children enough attention and she isn't sure it would be fair to them."
The couple have decided to settle in the UK now that David has more time on his hands. He's said to have promised Victoria he'll do most of the work if they do decide to have another baby.
"David's told her he'll be around more to help out with the kids and a new baby, but it's done little to dampen Victoria's concerns," the insider added.
"David's hoping she'll come round to the idea in good time. In some ways Victoria would love more kids and she knows that at her age she'll have to make a decision soon."
It was recently reported that Victoria was enjoying being the main earner in the family after David decided to end his tenure at Paris Saint-Germain.
British magazine Heat explained she had even started to warm to the idea of living in the UK, having previously preferred her life in Los Angeles.
"Vic feels like she's back in control. When David was still playing football she never really knew what would happen next or where they might be moving to. Now she's back in the driving seat and they can make decisions for the whole family. They'll spend time in LA for the summer as they still have the house there, but they're really falling back in love with the UK," an insider told the magazine.