Issue with Realtek rtl8168h Ethernet Interface since last auto update

LaurenceGough · January 3, 2024, 11:58pm

Hello all,

I have spent many hours today investigating this issue. I have a ClearLinux server on a mini PC. This PC has a Realtek rtl8168h Ethernet Interface.

Ever since what I assume was the last automatic system update I cannot run anything which puts a medium to high load on the network interface. CPU and local processing is all fine. No other changes have been made apart from automatic ones. I can do very minor network tasks, but the second you put load on it such as downloading a bundle or running a speed test it goes.

Testing with Ubuntu live USB does not have the issue.
Using a USB C Ethernet adapter (same cable and port) does not have the issue.
Using the live ClearLinux server USB bootable has the exact same issue.

There are no related logs that I could find in the journal.

version: 6.6.9-1394.native
firmware-version: rtl8168h-2_0.0.2 02/26/15
expansion-rom-version:
bus-info: 0000:01:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

lsmod | grep r8169
r8169                 135168  0
mdio_devres            12288  1 r8169
libphy                225280  3 r8169,mdio_devres,realtek

01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

Is the info on it.

Running a speedtest, or running any container that does moderate to high network traffic causes the vast majority of pings to drop (sometimes it’s 4-5 seconds per a ping) to and from the device for up to 5 minutes (depending on how long the attempt is). 100% repeatable every time. I have confirmed everything else on the network is fine. Pings to the gateway are just fine at the same time. SSH goes down of course so I am having to console.

Perhaps an issue with the latest Stable 6.6.9-1394 Linux Kernel? Doing research on this issue finds nothing at all.

For reference the Ubuntu live USB is running:

6.2.0-26-generic
Driver r8169
Version 6.2.0-26-generic
Firmware version rtl8168h-2_0.0.2 02/26/15 (same)

lsmod | grep r8169
R8169          114688   0

I have followed the instructions here and various other repairs but no luck. clear-linux-documentation/source/guides/maintenance/fix-broken-install.rst at 160e230e7acfee7cd08e26b86611ab0d7c66c0f0 · clearlinux/clear-linux-documentation · GitHub

I was looking into rolling back the kernel version, this is the furthest it seems I can go back, is this safe to do?

org.clearlinux.native.6.6.9-1394 (current version running)
org.clearlinux.ltscurrent.6.6.8-1393

Then using clr-boot-manager set-kernel

Any help would be much appreciated.

Thanks,

Laurence

Edit:

turns out this was an issue introduced by a change, fix is here:

sudo sh -c ‘echo 0 > /proc/sys/net/core/busy_poll’
sudo sh -c ‘echo 0 > /proc/sys/net/core/busy_read’

github.com/clearlinux/distribution

Issue with Realtek rtl8168h Ethernet Interface since last auto update

opened 10:20PM - 03 Jan 24 UTC

LaurenceGough

bug new

Hello, I have spent many hours today investigating this issue. I have a Clear…Linux server on a mini PC. This PC has a Realtek rtl8168h Ethernet Interface. Ever since what I assume was the last automatic system update I cannot run anything which puts a medium to high load on the network interface. CPU and local processing is all fine. No other changes have been made apart from automatic ones. I can do very minor network tasks, but the second you put load on it such as downloading a bundle or running a speed test it goes. - **Testing with Ubuntu live USB does not have the issue.** - **Using a USB C Ethernet adapter (same cable and port) does not have the issue.** - **Using the live ClearLinux server USB bootable has the exact same issue.** There are no related logs that I could find in the journal. ``` version: 6.6.9-1394.native firmware-version: rtl8168h-2_0.0.2 02/26/15 expansion-rom-version: bus-info: 0000:01:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: yes supports-priv-flags: no lsmod | grep r8169 r8169 135168 0 mdio_devres 12288 1 r8169 libphy 225280 3 r8169,mdio_devres,realtek 01:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15) ``` Is the info on it. Running a speedtest, or running any container that does moderate to high network traffic causes the vast majority of pings to drop (sometimes it's 4-5 seconds per a ping) to and from the device for up to 5 minutes (depending on how long the attempt is). 100% repeatable every time. I have confirmed everything else on the network is fine. Pings to the gateway are just fine at the same time. SSH goes down of course so I am having to console. Perhaps an issue with the latest Stable 6.6.9-1394 Linux Kernel? Doing research on this issue finds nothing at all. For reference the Ubuntu live USB is running: 6.2.0-26-generic Driver r8169 Version 6.2.0-26-generic Firmware version rtl8168h-2_0.0.2 02/26/15 (same) lsmod | grep r8169 R8169 114688 0 I have followed the instructions here and various other repairs but no luck. https://github.com/clearlinux/clear-linux-documentation/blob/master/source/guides/maintenance/fix-broken-install.rst Any help would be much appreciated. Thanks, Laurence

it doesn’t stick on a reboot so I’m just trying to get that sorted now.

Kingfisher · January 4, 2024, 4:22am

I had this exact same issue with my interface and your fix works perfectly! You can make changes persist between reboots by editing the /etc/clr-power-tweaks.conf file

i@clr~ $ cat /etc/clr-power-tweaks.conf 
/proc/sys/net/core/busy_poll 0
/proc/sys/net/core/busy_read 0

see man clr-power-tweaks for more info

mirkokirchner · January 4, 2024, 7:11am

Hello,

thanks for that workarounds, because i had the same issue.

On my computer i first tried the fix @LaurenceGough wrote, but CL told me,
that the file or directory could not be found.

mkirchner@clr-desktop~ $ sudo sh -c ‘echo 0 > /proc/sys/net/core/busy_poll’
bash: /proc/sys/net/core/busy_poll’: Datei oder Verzeichnis nicht gefunden

The second thing I tried was the tip from KingFischer and voila network interface is working fine and the issue solved.

Thanks a lot!

Edit:

on my clr-server that first fix worked.

LaurenceGough · January 4, 2024, 8:22am

Many thanks that sounds like the proper fix, I was just too tired from looking at it too long trying to get it to work . I should have posted the issue on GitHub earlier but I thought surely I am not seeing things right… It’s very, very unusual for the network to work sometimes but only fail under load for a end device / server (considering a healthy network otherwise).

I’ll give that a try later today.

P.s what interface do you have exactly? The same?

Thanks again

LaurenceGough · January 4, 2024, 12:20pm

What ethernet adapter do you have, the exact same one or just in the same family?

I’m just wondering how widespread this issue is?

So it’s impacting server and client editions and it seems related to the power tweaks module which I believe every clear Linux installation receives.

Kingfisher · January 4, 2024, 1:00pm

i@clr~ $ ethtool -i enp3s0
driver: r8169
version: 6.6.9-1394.native
firmware-version: rtl8168g-3_0.0.1 04/23/13
expansion-rom-version: 
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

i@clr~ $ lspci -knn | grep -iEA3 'net|ether'
03:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 10)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: r8169
	Kernel modules: r8169, r8168

it used to look like this before fix

i@clr~ $ ip -s link
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP mode DEFAULT group default qlen 1000
    link/ether 00:e0:97:1d:c8:a2 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
     133805852  114889      0     360    3585     110 
    TX:  bytes packets errors dropped carrier collsns           
       5896068   27921      0       0       0       0

after fix

i@clr~ $ ip -s link
2: enp3s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq state UP mode DEFAULT group default qlen 1000
    link/ether 00:e0:97:1d:c8:a2 brd ff:ff:ff:ff:ff:ff
    RX:  bytes packets errors dropped  missed   mcast           
      12820530   14470      0      60       0    4733 
    TX:  bytes packets errors dropped carrier collsns           
        447734    3239      0       0       0       0

don’t mind the r8168 module lol, it fixed it enough for me to get onto the internet to get some work done, but it was not ideal in the slightest.

mirkokirchner · January 4, 2024, 3:14pm

Maybe the problem is the driver? It’s always r8169.
Although they are different network interfaces, it is the same driver.

The manual of my server-mainboard says it’s: RTL8125BG network interface
and the desktop computer’s manual just says: Realtek® 2.5GbE LAN chip (2.5 Gbit/1 Gbit/100 Mbit)

the server has this adapter(s):

mkirchner@clr-server~ $ ethtool -i enp5s0
driver: r8169
version: 6.6.8-1393.native
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version: 
bus-info: 0000:05:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

mkirchner@clr-server~ $ ethtool -i enp6s0
driver: r8169
version: 6.6.8-1393.native
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version: 
bus-info: 0000:06:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

and my desktop-computer:

mkirchner@clr-desktop~ $ ethtool -i enp9s0
driver: r8169
version: 6.6.9-1394.native
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version: 
bus-info: 0000:09:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

mkirchner@clr-desktop~ $ lspci -knn | grep -iEA3 'net|ether'
09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:e000]
	Kernel driver in use: r8169
	Kernel modules: r8169

server:

mkirchner@clr-server~ $ lspci -knn | grep -iEA3 'net|ether'
05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: r8169
	Kernel modules: r8169
06:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: r8169
	Kernel modules: r8169

pixelgeek · January 4, 2024, 6:25pm

For those finding this thread - see also Issue with Realtek rtl8168h Ethernet Interface since last auto update · Issue #3018 · clearlinux/distribution · GitHub

Chris