Vermin are, strictly speaking, not a species in their own right, but a collective term applied to many species. According to Dictionary.com, the word "vermin" is defined as "noxious, objectionable, or disgusting animals collectively, esp. those of small size that appear commonly and are difficult to control, as flies, lice, bedbugs, cockroaches, mice, and rats".
Within the Redwall canon, the word is used to refer to any species considered villainous or unpleasant. This generally includes any species that is stereotypically evil in British folklore: rats, foxes, stoats, weasels, ferrets/polecats (in Britain, the term "polecat" refers to a wild ferret, not a skunk), and martens. Oddly, considering that the heroes are mostly mice, this description does not apply to cats, which appear to be the only neutral species — some are evil, but the very first one to appear in the series helps the hero Matthias. Possibly Brian Jacques liked cats in real life.
The defining characteristic of Mossflower's vermin is that they are evil, or at least not nice (though some of the Punch-Clock Villain vermin are reasonably harmless — still not particularly pleasant to hang out with, though). Only about six vermin characters in a cast of literally thousands have officially repented: Blaggut, Graylunk, Romsca, Grubbage, Crumdun, and possibly Veil, though he was somewhat up for interpretation. None of them did so because of a Mary Sue, and giving a Mary Sue the power to cause vermin to become Reformed Vermin (a term stolen shamelessly from ex-BNF Katie Sullivan's popular fic series Snowfur's Mossflower Talk) on the spur of the moment is a very big charge.
Agents Stormsong, Skyfire, and Redd fall under the Reformed Vermin heading — well, Redd's status as such may be debateable since he still has Token Lecherous Jerk qualities from his home fic. Molly Rath is an unashamedly non-reformed vermin. If Di'atwt and Twp'atwt are genuinely from Redwall, they likely count as reformed and non-reformed, respectively. Agent Laburnum has a bizarre fascination with them and has been dubbed the "Pervy Vermin Fancier" at the Official Fanfiction University of Redwall.
In its wider definition, as shown above, agents not familiar with the Mossflower continuum may use "vermin" to refer to Mary Sues and other badfic creations. This is of course a grave insult to all self-respecting rats everywhere.