That's why I whitelist my IPs in $ignoreip in jail.conf.
If you are already able to whitelist your (valid) login points, why would you need fail2ban? Just whilteliste them in your firewall and/or /etc/hosts.allow.
Personally I've yet had anyone bruteforce my ssh-key (although, as I run Debian, that is just luck as it turned out...). Still, fail2ban wouldn't really have helped against an attacker that knows/can figure out my access token off line...
I need to support multiple roaming users. Manually maintaining whitelists would be a burden.
Fail2ban isn't a firewall. It monitors logs for suspicious activity and responds with an action (not limited to banning an IP). When you expose services publicly, it's one of many tools you can use to limit bad behaviour without penalizing or inconveniencing legitimate users. I also use iptables (including the recent and string modules), RBLs, and a host of other access controls. Security is a layered approach and redundancy isn't a bad thing.
Avoiding banning certain ips (to avoid denial of service) is a form of white listing. So either you're open to denial of service, or you're able to whitelist all essential access paths?
If you are already able to whitelist your (valid) login points, why would you need fail2ban? Just whilteliste them in your firewall and/or /etc/hosts.allow.
Personally I've yet had anyone bruteforce my ssh-key (although, as I run Debian, that is just luck as it turned out...). Still, fail2ban wouldn't really have helped against an attacker that knows/can figure out my access token off line...