ssh connection refused on Debian 7

I cannot ssh to my Debian 7 server anymore. It happened after I tried to upgrade some packages on this old distro (including libc6…). Uptime is almost 2200 days! I do not even want to risk restarting it.

ssh -V returns OpenSSH_6.0p1 Debian-4+deb7u7, OpenSSL 1.0.1t 3 May 2016

I somehow still have an active ssh connection on Putty as a root so I can still run commands. Hopefully I will not lose it before I fix this massive issue!

nothing outputs when I run /etc/init.d/ssh restart, it just prompts a new line for new commands.

ps -f -p $(pgrep sshd) shows a pid and uid root

/usr/sbin/sshd does not exist

cat /etc/ssh/sshd_config shows nothing any different than it has always been. I run it on port 22000. No issues since 6 years until now. I now explicitly opened the port 22000 on iptables just to make sure but nothing changed.

sftp -oPort=22000 -vvv root@redacted_for_privacy shows

OpenSSH_6.0p1 Debian-4+deb7u7, OpenSSL 1.0.1t  3 May 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to redacted_for_privacy (123.456.redacted_for_privacy.123) port 22000.
debug1: connect to address 123.456.redacted_for_privacy.123 port 22000: Connection refused
ssh: connect to host redacted_for_privacy port 22000: Connection refused
Couldn't read packet: Connection reset by peer

command netstat -tulpn | grep 22000 returns nothing, however it does return udp6 for 22

command ps aux | grep sshd returns :

root     12247  0.0  0.0  29920  1252 pts/0    T    Nov21   0:00 vim /etc/ssh/sshd_config
root     12248  0.0  0.0  29924  1308 pts/0    T    Nov21   0:00 vim /etc/ssh/sshd_config
root     25974  0.0  0.0   6312   788 pts/0    S+   01:09   0:00 grep sshd
root     31003  0.0  0.0  81232  3932 ?        Ss   Nov21   0:14 sshd: root@pts/0

vim is because i set PermitRootLogin yes (it had always been set at without-password). 31003 is sshd pid

I also tried to reinstall:
apt-get install openssh-server but it returned
openssh-server : Depends: openssh-client (= 1:6.0p1-4+deb7u4) but 1:6.0p1-4+deb7u7 is to be installed
and E: Unable to correct problems, you have held broken packages. which is probably very true considering the ugly tinkerings I tried in order to upgrade libc6!

tail -f /var/log/auth.log when I try to ssh via WinSCP does not show any new line appending. either trying on port 22 or custom 22000.

netstat -ntlp returns

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      12431/proftpd: (acc
tcp        0      0 0.0.0.0:3129            0.0.0.0:*               LISTEN      4374/(squid)
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      4917/nginx
tcp        0      0 127.0.0.1:3306          0.0.0.0:*               LISTEN      7231/mysqld
tcp        0      0 0.0.0.0:10000           0.0.0.0:*               LISTEN      10330/perl
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      4917/nginx

I tried behind a different ip, I’m sure I did not jail myself.

The 4 websites hosted on it currently keep working as if nothing ever happened.

I am out of ideas and desperate. What else should I check/try? I believe it is not running sshd but I have not idea how to restart it. I do not have systemctl, neihter service sshd restart, and nothing in /etc/init.d/

Adding Flatpak program to Debian alternatives

When I try to add Firefox installed with Flatpak on Debian 10 to alternatives with the following command, I get an error:

Command:

sudo update-alternatives --install /usr/bin/gnome-www-browser gnome-www-browser "/usr/bin/flatpak run org.mozilla.firefox" 50

Error:

update-alternatives: error: alternative path /usr/bin/flatpak run org.mozilla.firefox doesn't exist

How can I successfully input a program installed with Flatpak to alternatives?

nvidia – Issue when using GPU using Bumblebee (Debian)

I just upgraded to debian bullseye and I had to set up again my Geforce MX150.
I followed the instructions in here yet I am finding some weird behavior:

  1. If I run ‘optirun glxgears -info’ I get 70 frames/sec and glxgears appears in nvidia-smi (meaning, it is running via GPU) yet my CPU is running glxgears at 110%.

  2. When I run ‘primusrun glxgears -info’ it runs at 60 frames/sec and glxgears appears in nvidia-smi and the CPU usage is 90%

  3. If I run ‘glxgears’ runs at 60 frames/sec, it doesn’t show up in nvidia-smi and the CPU usage is 7%.

So I am confused, in the test 1 and 2 the CPU usage should be a way lower since it says (at least nvidia-smi) that it is running on the GPU, and in test 3 the CPU usage should be higher since it is running on CPU. Plus, the frame rate seems to be the same in the 3 different tests.

Is my GPU actually running the processes? Any suggestion?

Thank you very much

linux – Debian : MPI code – [Hardware Error]: Unified Memory Controller Error: DRAM ECC error

When running an executable compiled with intel mpiicc, I get, after 30 minutes of running, the
following errors :

 kernel:(29585.573874) (Hardware Error): Corrected error, no action required.

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573881) (Hardware Error): CPU:2 (17:31:0) MC18_STATUS(Over|CE|MiscV|-|AddrV|-|-|SyndV|-|CECC): 0xdc2041000000011b

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573887) (Hardware Error): Error Addr: 0x0000000a6c12d280

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573888) (Hardware Error): IPID: 0x0000009600550f00, Syndrome: 0xc54c00040a800611

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573891) (Hardware Error): Unified Memory Controller Extended Error Code: 0

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573893) (Hardware Error): Unified Memory Controller Error: DRAM ECC error.

Message from syslogd@pablo at Nov  8 09:53:25 ...
 kernel:(29585.573895) (Hardware Error): cache level: L3/GEN, tx: GEN, mem-tx: RD

I have aa AMD EPYC 7702P 64-Core Processor with 1TB of RAM and a Debian OS :

Linux pablo 4.19.0-9-amd64 #1 SMP Debian 4.19.118-2+deb10u1 (2020-06-07) x86_64 GNU/Linux

From I have seen, I did the command : dmidecode -t memory that gives :

# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.2.0 present.

Handle 0x0023, DMI type 16, 23 bytes
Physical Memory Array
    Location: System Board Or Motherboard
    Use: System Memory
    Error Correction Type: Multi-bit ECC
    Maximum Capacity: 2 TB
    Error Information Handle: 0x0022
    Number Of Devices: 8

Handle 0x002B, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x002A
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL A
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F701
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x002E, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x002D
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL B
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F3ED
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x0031, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x0030
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL C
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F4BA
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x0034, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x0033
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL D
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F396
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x0037, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x0036
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL E
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F67D
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x003A, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x0039
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL F
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F394
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x003D, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x003C
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL G
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F48A
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

Handle 0x0040, DMI type 17, 84 bytes
Memory Device
    Array Handle: 0x0023
    Error Information Handle: 0x003F
    Total Width: 72 bits
    Data Width: 64 bits
    Size: 128 GB
    Form Factor: DIMM
    Set: None
    Locator: DIMM 0
    Bank Locator: P0 CHANNEL H
    Type: DDR4
    Type Detail: Synchronous Registered (Buffered) LRDIMM
    Speed: 2933 MT/s
    Manufacturer: Samsung
    Serial Number: 03C6F3FB
    Asset Tag: Not Specified
    Part Number: M386AAG40MMB-CVF
    Rank: 4
    Configured Memory Speed: 2933 MT/s
    Minimum Voltage: 1.2 V
    Maximum Voltage: 1.2 V
    Configured Voltage: 1.2 V
    Memory Technology: DRAM
    Memory Operating Mode Capability: Volatile memory
    Firmware Version: Unknown
    Module Manufacturer ID: Bank 1, Hex 0xCE
    Module Product ID: Unknown
    Memory Subsystem Controller Manufacturer ID: Unknown
    Memory Subsystem Controller Product ID: Unknown
    Non-Volatile Size: None
    Volatile Size: 128 kB
    Cache Size: None
    Logical Size: None

I don’t know where these DRAM ECC error come from, Maybe there are incompatibilies between my motherboard, CPU model or bad version of Intel compiler SDK ?

These errors appears roughly every 5 minutes during the execution.

I am using the intel compilers version compilers_and_libraries_2020.1.217.

I should modify maybe an option in the BIOS but I am not sure.

If someone had an idea to solve this issue, this would be fine to tell it.

ubuntu – IS this error an Ancient “su – hostile” vulnerability in Debian 8 & 9 ? $ bash: cannot set terminal process group (-1): Inappropriate ioctl for device

Just received this error >>

bash: cannot set terminal process group (-1): Inappropriate ioctl for device

then the disk partition went into READ-Only mode ; also noticed gnome-software was communicating both upload and download over the internet.

Here is a description of this 2012 Security Issue (below) ** ; so I expect, but don’t know how to confirm True/False, that 2020 Security patches MAY have “Fixed This” by Bombing Out and Setting Disk Read-ONLY as I just Witnessed. 2012 Security Issue Description ** Ancient “su – hostile” vulnerability in Debian 8 and 9 ? Here >> https://www.halfdog.net/Security/2012/TtyPushbackPrivilegeEscalation/ and Here >> https://news.ycombinator.com/item?id=17311808

Steps Taken so far: (Simply) Reinstalled existing software as per : gnome-software (version 3.20.5-0ubuntu0.16.04.13) will be re-installed gnome-software-common (version 3.20.5-0ubuntu0.16.04.13) will be re-installed

Any suggestions to confirm the hack failed, or to prevent the hack from setting partition to Read-Only from happening again appreciated.

linux – Nautilus does ‘t open in Debian 10

The command nautilus does’t show any window and does’t return.
There is no error message.

I tried apt purge nautilus && apt install nautilus and reboot without any success.

I am using Debian GNU/Linux 10 (buster) and Gnome 3.30.2.

My more general question is how to track this kind of issue? Is there any log file where gnome prints warnings and errors?

debian – Frozen/dropped TCP connections in AWS

We have a number of AWS EC2 instances within the same AZ that transmit large amounts of network traffic to each other. In a small fraction of the connections, when a client on host A connects to a server on host B and sends a large amount of data (e.g. 20 GB) from A to B at a high rate, the TCP connection freezes or times out. I’ve investigated this and the symptoms aren’t always the same, but typically it appears that when a connection is impacted by this problem the sender (on host A) stops receiving the ACKs that the receiving side (host B) is sending to A after some time. So at first all ACKs pass through, and then they get blocked in the middle of the connection. Also, VPC Flow Logs show that some packets returning from host B (the receiver) to host A (the sender) are rejected.

This happens on a number of EC2 instances (typically r5a.xlarge) that run Debian Linux 10 with Linux kernel 5.3.9 and the ENA AWS network driver that’s shipped as part of Debian’s kernel. They run Docker 18.09.1, installed via the docker.io Debian Buster package. Interestingly, I’m not able to reproduce the issue on Amazon Linux 2 (with Docker installed).

I’ve been able to reproduce it by letting the following simple experiment run in a loop for some time:

# Host B (server receiving data)
docker run -it --rm -p 20098:20098 debian:buster bash
apt-get update && apt-get -y install netcat-openbsd
while true; do date; nc -l -p 20098 | dd of=/dev/null bs=1M; done

# Host A (client sending data)
docker run -it --rm debian:buster bash
apt-get update && apt-get -y install netcat-openbsd
while sleep 1; do date; dd if=/dev/zero bs=1M count=20480 | nc -q 1 <server> 20098; done

The vast majority of times the experiment will succeed in sending 20 GB over the wire, but every once in a while (sometimes within minutes, sometimes within a few hours or even days) the transfer will get stuck or get cut short due to an unexpected disconnect/timeout. On some hosts I can reproduce the problem a lot more easily than on other hosts. The hosts where I can reproduce this more quickly tend to have more Docker containers and network activity, but I’m not sure yet if there’s a causal relationship there. I was also able to reproduce the issue directly when running the above netcat experiment directly the host rather than within a Docker container, although it does seem a lot harder to reproduce this way. This happens on hosts within the same VPC, AZ, and even subnet so we can rule out cross-region/cross-AZ/cross-subnet connectivity issues as a cause.

Here’s example tcpdump output that shows network activity when this happens. I’m skipping many successfully transmitted and ACK’ed TCP packets within this same connection. This information was captured with tcpdump -i eth0 -p -G 600 -s 80 -w ... host ... and port 20098. This is captured on the host’s network interface, not inside the Docker network, so network address translations have already been applied.

Tcpdump output on host A (172.20.3.188, the sending client):

08:00:03.615061 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322435576:322444525, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615064 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322444525:322453474, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615066 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615069 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322462423:322471372, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615071 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322471372:322480321, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615073 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322480321:322489270, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615076 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322489270:322498219, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 8949
08:00:03.615140 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322435576, win 256, options (nop,nop,TS val 683441101 ecr 4223113896), length 0
08:00:03.615178 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322453474, win 117, options (nop,nop,TS val 683441101 ecr 4223113896), length 0
08:00:03.824740 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223114105 ecr 683441101), length 8949
08:00:04.256748 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223114537 ecr 683441101), length 8949
08:00:05.084733 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223115365 ecr 683441101), length 8949
08:00:06.748724 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223117029 ecr 683441101), length 8949
08:00:10.108720 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223120389 ecr 683441101), length 8949
08:00:16.764722 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223127045 ecr 683441101), length 8949
08:00:30.076723 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223140357 ecr 683441101), length 8949
08:00:57.724718 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223168005 ecr 683441101), length 8949
08:01:50.972736 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223221253 ecr 683441101), length 8949
08:03:37.468722 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223327749 ecr 683441101), length 8949
08:05:38.304715 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223448585 ecr 683441101), length 8949
08:07:39.132913 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223569414 ecr 683441101), length 8949

Tcpdump output on host B (172.20.3.89, the receiving server):

08:00:03.615206 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322435576:322453474, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 17898
08:00:03.615225 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322498219, ack 1, win 491, options (nop,nop,TS val 4223113896 ecr 683441101), length 44745
08:00:03.615228 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322453474, win 117, options (nop,nop,TS val 683441101 ecr 4223113896), length 0
08:00:03.615256 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 0, options (nop,nop,TS val 683441101 ecr 4223113896), length 0
08:00:03.615908 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 1642, options (nop,nop,TS val 683441102 ecr 4223113896), length 0
08:00:03.616389 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 3373, options (nop,nop,TS val 683441102 ecr 4223113896), length 0
08:00:03.618742 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 6862, options (nop,nop,TS val 683441105 ecr 4223113896), length 0
08:00:03.621737 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 13913, options (nop,nop,TS val 683441108 ecr 4223113896), length 0
08:00:03.824879 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223114105 ecr 683441101), length 8949
08:00:03.824905 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683441311 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:04.256895 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223114537 ecr 683441101), length 8949
08:00:04.256929 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683441743 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:05.084873 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223115365 ecr 683441101), length 8949
08:00:05.084908 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683442571 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:06.748872 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223117029 ecr 683441101), length 8949
08:00:06.748901 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683444235 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:10.108863 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223120389 ecr 683441101), length 8949
08:00:10.108889 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683447595 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:16.764877 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223127045 ecr 683441101), length 8949
08:00:16.764905 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683454251 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:30.076864 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223140357 ecr 683441101), length 8949
08:00:30.076881 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683467563 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:00:57.724863 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223168005 ecr 683441101), length 8949
08:00:57.724877 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683495211 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:01:50.972908 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223221253 ecr 683441101), length 8949
08:01:50.972922 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683548459 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:03:37.468882 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223327749 ecr 683441101), length 8949
08:03:37.468902 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683654955 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:05:38.304895 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223448585 ecr 683441101), length 8949
08:05:38.304942 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683775791 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0
08:07:39.133073 IP 172.20.3.188.35506 > 172.20.3.89.20098: Flags (.), seq 322453474:322462423, ack 1, win 491, options (nop,nop,TS val 4223569414 ecr 683441101), length 8949
08:07:39.133092 IP 172.20.3.89.20098 > 172.20.3.188.35506: Flags (.), ack 322498219, win 24576, options (nop,nop,TS val 683896619 ecr 4223113896,nop,nop,sack 1 {322453474:322462423}), length 0

Notice how host A stops receiving packets from host B after it receives the 08:00:03.615178 ... ack 322453474 packet.

Here is the output of VPC Flow Logs during a failed connection (captured at a different time than the tcpdump output above):

VPC Flow Log output

Given that Amazon Linux 2 doesn’t seem to exhibit this problem I’ve tried to bring the network stack on Debian a bit more closely in line with Amazon Linux. I’ve tried to do the following on the Debian instances:

  • Apply some of the network sysctl settings from Amazon Linux to Debian
  • Upgrade the Linux kernel to 5.8.10
  • Upgrade the ena driver to 2.2.11
  • Upgrade Docker to 19.03.13
  • Explicitly allow ingress and egress traffic to/from ephemeral ports (32768-65535) to/from all IPs within our VPC in the security group that these hosts use

None of these seem to resolve the issue I’m having. What could possibly cause these dropped/rejected packets?

debian – SSH changing port issue

I’m running Debian 10. I’ve read like 10 instructions on how to change default SSH port from 22 to any desired but none of it seem to work.

I changed #Port 22 to Port 1111 in /etc/ssh/sshd_config. Restarted the service with service ssh restart and even rebooted the server.

And then I tried to connect: ssh -p 1111 user@hostname to no avail. It looks like the port is not open but I have no firewall whatsoever.

Seconly I tried to connect the usual way: ssh user@hostname which gives me ssh: connect to host hostname port 22: Connection refused.

service ssh status gives me

Nov 04 05:32:04 localhost systemd(1): Starting OpenBSD Secure Shell server...
Nov 04 05:32:04 localhost sshd(904): Server listening on 0.0.0.0 port 1111.
Nov 04 05:32:04 localhost sshd(904): Server listening on :: port 1111.
Nov 04 05:32:04 localhost systemd(1): Started OpenBSD Secure Shell server.

~# ss -tulpn | grep 1111
tcp   LISTEN 0      128                              0.0.0.0:1111       0.0.0.0:*                                                                                users:(("sshd",pid=904,fd=3))                                                  
tcp   LISTEN 0      128                                 (::):1111          (::):*                                                                                users:(("sshd",pid=904,fd=4))                                                 

Am I doing something wrong?

debian – Make OpenVPN Client use eth0:1 IP address of OpenVPN Server

I have a VPS (Debian 10) from Hetzner that has eth0 -> IP 22.22.22.22

Then I added a floating IP and now there is also eth0:1 -> IP 44.44.44.44

On OpenVPN server.conf I added local 44.44.44.44 so it listens on the floating IP.

Then I restarted OpenVPN with /etc/init.d/openvpn restart

This is what I have on iptables -S:

-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -i tun0 -j ACCEPT
-A INPUT -i eth0:1 -p udp -m udp --dport 1194 -j DROP
-A FORWARD -i tun0 -o eth0:1 -j ACCEPT
-A FORWARD -i eth0:1 -o tun0 -j ACCEPT

On client .ovpn config file I added this:

remote 44.44.44.44 1194

All works fine, the only issue is that on OpenVPN client (Windows 10) when I connect to the OpenVPN server it uses the IP address of eth0 22.22.22.22 and not 44.44.44.44 of eth0:1

Do you have any suggestions to fix this?

I’d like to make the OpenVPN client use the IP 44.44.44.44 (eth0:1) from OpenVPN server.