Clunes felt unwanted in Port Isaac
The actor is back as the grumpy GP for the sixth series of the show, which began in 2004.
An army of devoted fans travel from all over the world to Port Isaac, on the north coast of Cornwall, to see where the series is shot.
"We get so many Americans and Australians and Kiwis, it's extraordinary," Martin, 51, said.
"You can get two or three hundred people watching. It's nice, they're very on-side. It's not like filming in London where people beep their horns and try to ruin your shot."
But he added: "For the last five or six years I felt semi-loathed when I was in the area. You know, people trying to extort money from us in one way or another, or very public displays of public dissatisfaction with us."
Martin, his wife Philippa Braithwaite, who's one of the show's producers, and the rest of the Doc Martin team have always tried to be sensitive about shooting in such a small village, he insists.
"It's one of the reasons we don't film it every year. Anyway, I don't want to be dragged into that," he said.
:: Doc Martin returns for a new eight-part series starting on ITV on Monday, September 2