"I can remember drying oats right up to Christmas. It used to be all stored in big 100kg sacks and we were so tight for space that we ended up stacking them four high in the shed," he recalls. But Jimmy's dad was taking a massive gamble as he hadn't any deal with any of the local merchants to take the grain.
The merchants weren't keen on encouraging farmers to get into the drying business, so they were quite happy to see the Whytes left with piles of unsold grain.
In the end, much of the grain was loaded on to railway cars at the local train station in Balbriggan to be transported into Guinness.
But risk-taking was second nature to Peter Whyte, experiencing several scary situations over the years when interest on borrowings on the land purchases soared to over 20pc. He also took a chance on experimenting with different crops, including everything from beet to onions.
"He would have us all under serious pressure but, in hindsight, he was rarely wrong in his thinking," says Jimmy.
Today, Jimmy, along with his brothers Peter, Martin, Eddie, Ollie, Joey, Anthony and Martin's wife Mary farm 200ac of potatoes, fatten about 500 cattle and grow cereals all along the rich vein of tillage land that runs along the south Meath/north Dublin border.
While some of the brothers have "retired" to make way for the next generation - James, Peter, Dermot, Davy, Peadar, Kevin and Joseph - in effect, they are still togging out in the overalls on a daily basis.
When I bump into Jimmy, he tells me about the new one-tonne lathe that he has just bought in the west to add to their already impressive workshop.
Jimmy still finds himself down in the workshop until 11pm at night, tinkering away on one of his 240hp autocross cars, which he only recently renewed his licence to drive.
"You could hit 80 or 90mph going around the track and I thought I was going to be very sore after it, but I was pleasantly surprised," grins the septuagenarian.
However, the workshops are also where much of the repairs and, indeed, innovation on the farm happens.
The sheer scale of operations throws open alternatives that would be out of reach for most farmers.
In 2014, the family invested €125,000 into Ireland's first 2.5MW biomass grain drier designed to utilise the huge amounts of low-value straw from oilseed rape and other cereal crops produced on the farm.
The 750C of heat inside the furnace is used to dry the 9,000t annual grain harvest, replacing the €80,000 yearly spend on diesel. Even with oil prices tumbling to lows of $30 per barrel in recent years, Anthony Whyte believes the burner has almost paid for itself in just three seasons.
There was no grant aid available on the installation at the time, but the Whytes forged on, convinced of the economic and environmental benefits, with the unit reducing carbon emissions from the grain drying operation by at least 95pc, or 240t.
"Obviously, there have been a few teething issues in terms of learning what way to optimise the burn, reduce smoke and keep the thing going smoothly, but I'm convinced we're there now," says Anthony.
But to feed the furnace with massive half-tonne bales, Jimmy had to engineer and build their own bale lifter.
"We needed something that handled the bale from the narrow end, didn't need extra room to drop the bale inside the oven and held on to the straw even after all the twines melt as we're placing the bale into the flames," he said.
Branded spuds and drones the way forward for the Whytes
The next generation of sons are already managing significant parts of the Whyte business, but more youngsters may join the crew yet.
"My young lad Conor is thinking about it," said Anthony, smiling. "But there's no point in anybody coming into farming for the money because they'll just end up being miserable."
Joseph (25) is the youngest of the sons currently working full time on the farm. "I studied agriculture in Kildalton and WIT, and specialised in tillage and did a work placement in Country Crest, which was a great way of seeing how other businesses operate. I was lucky I did that because you can get comfortable in the tractor," he says.
"Myself and David look after all the fertiliser with two Vicon spreaders. One is a 6t trailed one, but you do less tracking with the 3t mounted version because we have it on a tractor with big 900mm wide wheels. They cost up to €5,000 each, but they came with the (New Holland) combines so we can just swap them around.
"We tray test the machines almost every day because there can be awful variance between each batch for fertiliser. There's a fair bit of paperwork as well, but we've everything online now so we can access it on the phones wherever we are," says Joseph.
He's also the man behind the farm's foray into the branded world via Facebook with their Whyte Bros Potatoes page that has garnered over 1,000 likes from machinery enthusiasts wowed by drone footage of the harvest.
New holland tractors and no min-till
The Whytes operate a fleet of 11 tractors and are New Holland men. The latest purchase is a 330hp T8330, with a list-price approximately €150,000. "It has a variable transmission and we'll chip it to push out 400hp," says Peadar Whyte.
"We end up buying a new tractor or combine or something every year, but at the same time, we're not obsessed about new machinery."
All crops are sown into ploughed ground except for the oilseed rape, which is direct drilled in with a Sumo sub-soiler.
"We've tried them all - min-till, strip-till, no-till, but the only one that we feel is giving us a benefit is the direct drilling for the oilseed rape. It suits us because it saves time when we're under a lot of pressure," says Anthony Whyte.
For Stories Like This and More
Download the FarmIreland App