World number five Higgins had been in fine form on Monday at the Newport Centre, beating Gareth Allen 4-1 with breaks of 120, 99 and 97, but could not replicate that display against his fellow former world champion.
Doherty took the first two frames but Higgins won a tense third, clearing from the green to settle a low-scoring and closely-fought frame.
Doherty needed several attempts to wrap up the next frame, but finally did so by potting the final brown to move within one frame of victory.
A break of 46 put him in control in the next and Higgins' response stalled at 14 before the Irishman sealed victory.
"It was a war of attrition, there was a lot of good safety. You have to match John in every department and I beat him with my safety game today," the 43 year old Dubliner said.
"I have been starved of winning matches against the top players so it's a great result for me. I'm in the twilight of my career but there's still a bit of sparkle.
"It's like a rare fruit that you find now and then and it still tastes sweet. It gives me a real boost of confidence. People think I'm just a pundit now so it's nice to show I can still play."
In the last 16 Doherty next faces the winner of Ricky Walden's match with Tom Ford.
"I have played Anthony a few times this season and he has beaten me," said the Scot.
"He's a great player and he was scoring heavily at the start. It was a good match for the first four frames and then we both struggled."
Sam Baird made short work of an upset win over Northern Ireland's Gerard Greene, a break of 73 wrapping up a 4-0 win for the world number 89.