Hi,
Your going to get a very mixed bunch of advice from us, but in this instance I would advise neutering.
I say this as someone who did not have my own male dog neutered he stayed entire his whole life and all bar one family male dog none have been neutered, so it is not something I usually advocate. Unless there is a problem I basically do not see any point in neutering a male dog, (a female dog I have a different view althogether

)
My reasonings in your case are he is fully mature now, so we can't blame his behaviour on adolescents if he were attacking adolescent males I can also understand this behaviour, but apart from attacking neutered dogs he has also attacked a fully mature female. He has snapped/bitten a person too. He's an accident waiting to happen, and yes you have been very lucky, probably due to your apologetic nature and that you have taken full responsibility, unlike many.
Your socialising, your training, and you seem to have a good grasp on male dogs behaviour and your very responsible to muzzle at the moment, so what is left................. there is only to neuter and hope this will stop his aggressive behaviour.
As you probably know to neuter a less confident dog can actually cause aggression, so unless you feel he strikes out
because he is afraid, I think neutering is the way to go with him.
I'm sure you will get many view points but this is mine.