Related very closely to question and answers like:
How to fix “ssh_exchange_identification: read: Connection reset by peer” error?
My reputation is too low to comment or provide answer suggestion. Below is the case description and the aswer.
The error message is the the same as in above question (ssh_exchange_identification: read: Connection reset by peer), when attempting to ssh the server.
doe@server1 ~ $ ssh (MY_SERVER_IP)
ssh_exchange_identification: read: Connection reset by peer
ssh -v does not give any relevant additional information. As the error message suggests, the connection was reset, so there is no connectivity issue and the connection was established ssh service is running on the server just fine.
So, I found that the host was being added to /etc/hosts.deny. Entry was: sshd: (MY_HOST_IP). Whenever I deleted the entry row from the hosts.deny file (using different IP address on the client), I could login typically once from the originally failing host IP address. The same limitation did not apply to other IP addresses. I can login from other IP addresses as many times I like.
The server is Ubuntu 18.04.5 LTS was in a local network. Server has lot of webserver stuff and experiments. Quite often server is exposed to Internet so it has firewall to keep only selected ports open. It has also DenyHosts as well as Fail2Ban to handle the flood of unauthorized login attempts. Server was upgraded couple of days ago and this problem has started to occur after that. Anyways, supposedly the programs that prevent access are the the prime suspects. I started digging into them by the proposals at the previous question mentioned above.
DenyHosts uses hosts.deny to deny the access based on /etc/denyhosts.conf configuration file and the /var/log/auth.log log file analysis it does.
Looking into (MY_CLIENT_IP) in to the state of DenyHosts:
$ sudo grep -r "(MY_CLIENT_IP)" /var/lib/denyhosts/
/var/lib/denyhosts/users-hosts:doe - (MY_CLIENT_IP):1:Sun Oct 25 12:16:06 2020
/var/lib/denyhosts/users-hosts:jane - (MY_CLIENT_IP):2:Tue Nov 10 18:16:20 2015
/var/lib/denyhosts/hosts-root:(MY_CLIENT_IP):0:Sun Oct 25 12:16:06 2020
/var/lib/denyhosts/hosts-restricted:(MY_CLIENT_IP):0:Sun Oct 25 12:16:06 2020
/var/lib/denyhosts/hosts:(MY_CLIENT_IP):0:Sun Oct 25 12:16:05 2020
/var/lib/denyhosts/hosts-valid:(MY_CLIENT_IP):1:Sun Oct 25 12:16:06 2020
Removing (MY_CLIENT_IP) entries gives temporary resolution because DenyHosts will put it back there after it has done it’s math on the analysis. During this troubleshooting I found also changed behaviour of fail2ban and below is the stuff related to that. Basically fail2ban also started to hate me logging in.
Fail2Ban works towards the same goal bit differently (uses IPTABLES for blocking) and also the ssh login error is different due to firewall blocking access:
doe@server1 ~ $ ssh (MY_SERVER_IP)
ssh: connect to host (MY_SERVER_IP) port 22: Connection refused
The thing is that now also fail2ban is starting to add (MY_HOST_IP) into the sshd jail it uses to block an IP. I need to run also:
$ sudo fail2ban-client set sshd unbanip (MY_CLIENT_IP)
After first removing entries from the above files/locations (after stopping the services), I also made a copy of auth.log for possible further analysis then removed auth.log and restarted server. I thought clearing the auth.log is necessary to prevent blocking from re-occurring.
$ sudo cp /var/log/auth.log auth.log.copy
$ sudo rm /var/log/auth.log
$ sudo reboot
Was this ok or what would have been the correct way of dealing with the issue? This resolved the issue, but was it a bad practise and did I misuse or fail to use the applications properties? Is there a way to make DenyHosts and Fail2Ban start reading auth.log from restart time fordward, or are those supposed to do that by default? Should I have done something else or more elegant steps to clear the situation?
PS. Yes I know, I could white-list my host, start using keys or start typing more carefully.