(All right, I've considered a bit on how your swarms would play in general and this is basically what I've concluded. Feel free to counter-argue or come up with other ideas, I'm just laying down my though process here:
Most NPCs have a -5 in all stats and 1 HP. Basically useless in most things but can perform simple tasks that a character may not wanna waste an action on.
Your swarms, given that they are probably somewhat harder to recruit should have some benefits because of it. For one I'm thinking they should have 50hp per swarm charge, which also makes sense since that's the amount they heal your servant with if they sacrifice themselves (So naturally, a damaged swarm only heal their current HP to your servant). I can also see them having a couple of stats that are more in the -2 ballpark, depending on the type of swarm animal.
I'm also thinking that the swarms should be primarily made off of one type. You mentioned pigeons so you may wanna specify how many of your swarms are pigeons and how many are rats.
Importantly, when you try to gain more swarms, the DC lowers by 5 for each swarm you send. So starting at 19+ for a single swarm, 14+ for 2 swarms, 9+ for 3, 4+ for 4, and guaranteed for 5. If you are yourself performing this action you get to add your verminmancy, and should you fail, the DC will be reduced by 10 for the next time. If you merely send your swarms to do it on their own, they use their own swarm stat (between -5 and -2) and upon failing, no progress is made for the next time. Either you or your Servant have to actually spend an action to be guaranteed a swarm in the next phase.
Does this all work for you or would you like to amend anything for how it works?)
1
