In case you haven't noticed yet, your machine may be under attacks from botnets. Go check your /var/log/auth.log (at least under Debian) and see for yourself. Botnets are attempting to crack your passwords using brute-force.
- Change port. I've picked 2022.
- Disable password logins. Use only authentication using secure keys. Too much of a hassle for me, especially on a server I share with others, since we'd have to carry our key files around.