In the first indication that the mayor might bow to heavy pressure to seek help - if not take a leave of absence - after he admitted smoking crack and a video surfaced of him ranting and threatening to kill someone.
He said the mayor needs to say what he plans to do himself because "when you go left, he goes right".
Mr Ford has apologised for his bad behaviour but his refusal to resign or take a leave of absence has frustrated both his opponents and allies on Toronto's City Council, which has no legal way to force him out unless he is convicted of a crime.
The mayor was mobbed by reporters at City Hall today but refused to discuss what his next steps would be.
"I'm dealing with a serious personal issue right now. Please give me this time," he said before walking into an elevator.
In another sign of Mr Ford's growing isolation, the mayor's radio show was taken off the air. NewsTalk 1010 announced today that Mr Ford and his brother "have mutually determined to conclude broadcasts of The City, ending with last week's show".
Mr Ford's influential brother, meanwhile, said he has urged the mayor "to go away for a couple of weeks", although he reinforced the family's stance that Mr Ford has no reason to resign.
"I've mentioned to Rob, maybe go away for a week, a couple of weeks, and get your mind together," Doug Ford told a Toronto radio station.
Doug Ford, a city councillor, said his brother needs a holiday and needs to work on his weight. He also suggested that despite the mounting scandal, Rob Ford remains a potent political force ahead of next year's mayoral elections.
"If Rob goes away on a little vacation, a week, two weeks, comes back, loses 50 or 60 pounds, and stays on the straight and narrow ... it will be very tough to beat Rob Ford," Doug Ford said.
The mayor, a conservative who was elected three years ago on promises to limit government and "stop the gravy train at City Hall", has vowed to seek re-election. He still has a loyal following in Toronto's outer suburbs where he is considered a straight-shooter willing to take on the liberal elite.
"Yes, he is an embarrassment, but not a thief, which is better in my books," Joe Amorim, a 49-year-old Toronto resident said. "People are tired of smooth-talking politicians that waste public money and serve corporations and the wealthy."
But Mr Ford's three-year tenure has been consumed by mounting scandals ranging from the petty, making rude gestures to Torontonians from his car, to the more serious - being fired from his cherished side job as a volunteer high school football coach after making disparaging remarks about parents and their kids.
The comments from Mr Morris and the mayor's brother came a day after a video surfaced showing Mr Ford threatening to kill an unknown person. Mr Morris said yesterday was a defining day for the mayor of Canada's largest city.
The shaky, blurry footage, which appeared in the Toronto Star website, showed the mayor pacing around in a room, waving his arms rolling up his sleeves and he unleashes an expletive-laced tirade.
"No holds barred, brother. He dies or I die," the mayor tells someone else in the room, possibly the person taking the video.
The context of the video is unknown, and it is unclear who the target of Mr Ford's wrath is. The Toronto Star said that it bought the video for 5,000 US dollars (£3,100) from "a source who filmed it from someone else's computer".
The mayor said he was "extremely, extremely inebriated" in the video and "embarrassed".
The video surfaced just two day after the mayor admitted to having smoked crack while in a "drunken stupor" about a year ago. The stunning confession came a week after police said last week they had a video of that appears to show the mayor smoking crack.
News reports of the crack video's existence first surfaced in May, but it has not been released publicly.
Doug Ford insisted that the video of the mayor threatening to kill someone was an "isolated incident."
His mother and sister also downplayed the mayor's troubles, insisting in a television interview yesterday that Rob Ford was not an addict of any sort.
His mother, Diane Ford, said she has advised her son during a family meeting last week work on his "huge weight problem", curb his drinking watch the company he keeps. But she insisted that her son did not need rehab.
But Toronto's city councillors are working on efforts to drive Mr Ford from office, or at least curb his powers.
One measure, which could be voted on next week, would takes the unprecedented step of asking the province of Ontario to pass legislation to remove the mayor if he does not agree to take a leave of absence.
The provincial government, however, has expressed reluctance to interfere in Toronto's municipal affairs.