"There were people who wanted me to stand aside but if I'd stood my ground and fought my battle I'd still be president of the IFA. There is no doubt in my mind because I had no difficulty explaining anything that was done or that had to be done," he told the Farming Independent.
The 55-year-old says he was led to believe that stepping aside was the only way to protect the organisation as it faced mounting criticism over the salary of former CEO Pat Smith whose pay package amounted to almost €1m in 2013 and 2014. Mr Downey agreed to a severance package of €1m plus €1m pension payments, but claims he didn't act alone. Mr Smith has lodged papers with the High Court alleging he was defamed by the organisation.
Although Mr Downey admits he was aware of "the sort of salary" Mr Smith was on, he says he didn't set it and intended to review it once a remuneration committee was in place.
"Political people in IFA looked to see 'how can I protect my back here?' Each one of them individually. They didn't think 'how can we collectively protect the organisation?' I brought an integrity and an honesty to the job - but that integrity and honesty has cost me the job," he said.
He is still deeply frustrated with what he says were "misrepresentations" of him during a 17-hour crisis meeting with farmers at the time. He didn't attend, as he had stood aside to allow former IFA chief economist, Con Lucey, to conduct his review.
On top of the anguish of losing his job, his mother, Josie, passed away last Spring. "Obviously that was another huge trauma, it probably distracted us from the whole IFA thing and allowed the family to re-adjust to a different position. It brought me back into reality" he said.
He continues to take great pride in the work he achieved with the IFA, particularly his work on the land leasing deal and succession planning."I'm very proud of my time in IFA right to the last second. My conscience is clear in that I did absolutely everything right. I don't think I'm the villain of the piece," he said.
Factory levies row is a 'storm in a teacup'
The IFA's on-going opposition to the proposed ABP/Slaney meats deal is a "storm in a tea-cup," Eddie Downey has warned.
From a funding perspective, the former president says the IFA needs to "move away" from reliance on levies to ensure a more sustainable, independent future.
"I wouldn't have the same level of fear that the organisation seems to have over the deal. At least the 50pc share would still be owned by an Irish company,"
"My biggest concern is for the viability of Slaney Meats. We need to strengthen our relationships with meat factories going forward," he said.
"IFA need to look at other farm organisations outside Ireland, all over Europe, the UK, America, where they all have add -on businesses and independent structures that allow them to be financially secure," he said.
Mr Downey is concerned the organisation is disconnected from real farmers.
"I loved meeting farmers on the ground at the Ploughing Match, even if it was a row, I loved it. The stronger it was the better. The more they came at you the better."
"At the moment within the political system, society and in the IFA, there is a disconnect with the grassroots. There is a system in place where people are trying to be popular and unless we get someone with strength to stand up and say 'sorry this is where the real world is at' the two tier recovery will continue to expand," he said.
Broadband is crucial. "We built highways, motorways and airports to take our young people out of rural Ireland. We should bulldoze them down and build broadband so we can keep them at home," he said. He isn't worried about the IFA's reputation or membership but stresses that current president, Joe Healy, needs sound advice. "Joe is the right guy for the job but there is a huge vacuum at the top of the IFA and it's very difficult for him. He doesn't know who to trust," he said.