The independent chief inspector of the agency said just 63 out of 683 threat assessments had been undertaken - with none since 2008.
John Vine said the agency "needs to improve the way it identifies and addresses threats" to the UK border in Scotland and Northern Ireland.
The inspection took place between November 2010 and January 2011 and focused on the deployment of detection staff to air and seaports, the risk assessment of small ports, the selection of people, vehicles and freight for searching and the treatment of passengers by agency officers.
It also found staff focused on passport control "potentially at the expense" of detecting drugs and other illicit items, intelligence gaps on potential risks to the border and that no seizures from freight containers had been made between September 2009 and November 2010.
Mr Vine said: "I found that the focus of staff deployment at airports was concentrated on the Primary Checkpoint potentially at the expense of illicit commodity detection. I found that only 63 out of 683 threat assessments of small air and seaports had been conducted in the whole of Scotland and Northern Ireland, with none since 2008.
"At the ports inspected, I was surprised to find that the agency had not made any seizures from freight containers for the 14 month period between the end of September 2009 and our inspection in November 2010. The agency needs to improve the way it identifies and addresses threats to the UK border in Scotland and Northern Ireland."
But he praised staff for their "commitment to identifying and seizing illicit commodities, sharing information on trends and using local knowledge to good effect".
Immigration minister Damian Green said: "This government is committed to radically reforming the immigration system and improving border security. It's clear that improvements are already under way with the agency working closer than ever with the police and Serious Organised Crime Agency, and frontline officers improving their use of intelligence at the border and within the Common Travel Area.
"In the past year alone, officers in Scotland and Northern Ireland have detected and removed more than 1,500 immigration offenders, stopped hundreds of people from illegally entering the UK and have prevented more than 100kg of drugs from reaching the streets."