Bombarded with attacks
So you've generated a public/private key pair to log in to your server, set SSH not to accept passwords and disabled root login. You may have even changed the port number on which SSH listens (probably a waste of time). A problem remains, however. Your server will constantly be bombarded with SSH login attempts, taking up bandwidth, CPU clock cycles and log file space.
IP address whack-a-mole
A popular solution is to ban IP addresses where the attacks are coming from by adding them to the internal server firewall for a certain length of time. However, this is just playing whack-a-mole. Surely there is a better solution.
These days virtual private server (VPS) providers have added a cloud firewall feature to their offerings. Examples are DigitalOcean and Amazon Lightsail, and it comes included for free. The benefit of an external firewall is that it will take the pressure off of the internal firewall, leaving your server in peace. You can also restrict access to a port to just a single IP source.
So this is what I do. I restrict access to the SSH port to my current IP address. When my IP address changes, I can just log in to my server provider and update the cloud firewall settings. That can be inconvenient sometimes, but there is always a trade-off between security and convenience.
You could also specify a range IP addresses that are allowed access. An example might be the range of IP addresses that your ISP could assign to you. Even that would reduce the number of attacks reaching your server to virtually nothing. You would also not need to update the firewall rules so often.
IP addresses are, of course, spoofable, but unless someone is performing a targeted attack, has my private key and knows the encryption password, that is not a very concerning issue.
A more secure server
By using a cloud firewall and severely restricting the IP address allowed through it to the SSH port, server load and security improve.