Fully agree. Limiting the networks which can access your server will help, e.g. limit access to just your local provider or your workplace and you'll see no attempts from Brazil, China, ... unless you are located there, of course ;-)
That's manageable with a bit of preparation: when I'm travelling, I allow access from other networks, e.g. those from phone providers. Or add a web form where I activate the IP address with a cryptographically signed "token" which the server can verify and then add the IP address to the set of allowed ones.
Used one or the other every now and then in the last 10+ years and still have my attackable footprint small the rest of the time.
