The number of annual inspections of sites regulated by the Scottish Environment Protection Agency fell by a quarter over the last five years, prompting concerns about pollution.