Mayor Bill de Blasio said Ahmad Khan Rahami, a naturalised citizen from Afghanistan, could be armed and dangerous.
"We need to get this guy in right away," Mr de Blasio said on CNN. "My experience is once the FBI zeroes in on someone, they will get them."
Police connected the New York City and New Jersey bombings and said he was wanted for both.
In Manhattan, there was an explosion that injured at least 29 people and an unexploded pressure cooker device discovered blocks away; in New Jersey, a pipe bomb blast at a Jersey shore town and five explosive devices at a New Jersey train station.
On Sunday night, FBI agents in Brooklyn stopped "a vehicle of interest" in the investigation of the Manhattan explosion, according to FBI spokeswoman Kelly Langmesser.
She would not provide further details, but a government official and a law enforcement official who were briefed on the investigation said that five people in the car were being questioned at an FBI building in Manhattan.
No one has been charged with any crime, and the investigation is continuing, Ms Langmesser said.
Democratic Governor Andrew Cuomo, touring the site of Saturday's blast that injured 29 people in Manhattan's Chelsea neighbourhood, said there did not appear to be any link to international terrorism. He said the unexploded pressure cooker device appeared "similar in design" to the bomb that exploded in Chelsea, but he did not provide details.
On Sunday, a federal law enforcement official said the Chelsea bomb contained a residue of Tannerite, an explosive often used for target practice that can be picked up in many sporting goods stores.
Mobile phones were discovered at the site of both bombings, but no Tannerite residue was identified in the New Jersey bomb remnants, in which a black powder was detected, said the official.
The pipe bomb exploded on Saturday in Seaside Park, New Jersey, before a charity 5K race to benefit Marines and sailors. The race was cancelled and no one was injured.
Late on Sunday, five suspicious devices were found near a train station in Elizabeth, New Jersey.
Elizabeth mayor Christian Bollwage said the devices were found in a bag in a rubbish bin by two men who reported seeing wires and a pipe coming out of the package. One of the devices exploded as a bomb squad used a robot to try to disarm it. No injuries were reported.
There was no immediate word on whether the devices were similar to those in nearby Seaside Park or New York City.
Officials have not revealed any details about the make-up of the pressure cooker device, except to say it had wires and a cellphone attached to it. On Sunday night, police blew up the device, rendering it safe. A forensic examination of the device will be sent to the FBI Laboratory at Quantico, Virginia, police said.
Homemade pressure cooker bombs were used in the Boston Marathon attacks in 2013 that killed three people and injured more than 260.
New York's governor has also said the Manhattan bombing could be an act of terrorism with a foreign connection.
On Sunday, a team of five FBI agents searched an Uber driver's vehicle that had been damaged in the Manhattan blast. The driver had just picked up three passengers and was driving when the explosion occurred, shattering the car's windows and leaving gaping holes in the rear passenger-side door.
The Chelsea explosion left many rattled in a city that had marked the 15th anniversary of the 9/11 attacks only a week earlier and that was schedule to hold a United Nations meeting Monday to address the refugee crisis in Syria.
Witnesses described a deafening blast that shattered store front windows and injured bystanders with shrapnel in the mostly residential neighbourhood on the city's west side.
One New Yorker, Anthony Stanhope, was in his apartment when the blast went off. At first he thought it was thunder and lightning.
"Then all of a sudden, car horns went off, and I thought, 'Oh, my God, this isn't lightning. This is too loud," Mr Stanhope said. "This is a bomb."