Winter plays the precocious teenager Alex Dunphy in the hit TV comedy which has won three consecutive Emmys as American television's best comedy series.
Superior Court Judge Michael Levanas issued the ruling after considering a report by child protective investigators that was critical of Ariel's mother, Chrisoula Workman.
The report found there was evidence Workman had been physically and emotionally abusive to Ariel.
Judge Levanas said the investigators' report found that claims of physical abuse were inconclusive, but an attorney for Ariel's sister said they could be proven in a subsequent report.
Celebrity website TMZ.com said Winter's mother was alleged to have slapped and emotionally abused the teen.
"It's all untrue, it's all untrue," Winter's mother, told People magazine. "I have my doctor's letter that my daughter's never been abused."
The judge set a December 12 trial date to determine whether Ariel should be placed in a permanent guardianship.