Solved Proxmox running VM of Ubuntu and a SMB server how to make SMB multichannel work?

[DJones]

Trying to get Smb multichannel to work on my Proxmox server. The Smb server exists on my Ubuntu VM. Anything you notice that might be wrong with this setup? additionally both client NIC's support 2 RSS channels per NIC. The server can support up to 4 RSS channels per NIC (Windows only supports up to a max of 4 channels). SMB server uses mandatory signing and mandatory encryption SMB 3

Proxmox network bridge:

server multi channel support = yes

VM network adapters: vmbr0 has a gateway; vmbr1 is two network adapters directly connect from server to client binded to smb. Ubuntu network adapters for smb are set to Link-Local so that Windows 11 handles the DHCP.

 

[DJones]

 

[DJones]

My current assumption is that because the clients are on 169.254.144.194 & 168.254.184.58 and the server is on 169.254.83.90 & 169.254.232.175 they are using different subnets so maybe that’s the problem. Well these dhcp addresses were auto assigned maybe I need to put everything in the same subnet? I read Linux should use different subnets for smb multichannel.

The server can also make SDN zones, but I’d be further ahead to use a pfsense vm than work with anything, but localhost for Proxmox. And I doubt asus merlin routers can adequately create vnet subnets

 

[Adooni]

in your case are all vNIC in a bond on the linux, or are they bonded by proxmox?

 

[DJones]

So I've further isolated each physical link into two bridges since I last posted.

Layout looks like this. Ignore vmbr0 as it's just so I can access management of proxmox. Well it's detecting RSS channels it's not actually using them or the other NIC. 169.254.83.90 is the mapped drive ip on windows. Vlan is aware on each bridge, but both are using vlan0. I had each interface in one bridge before, but I didn’t want crosstalk within the same bridge, this way it’s more representative of what baremetal would be. Each Nic client and server have both 2 set for RSS channels in each Nic.

in your case are all vNIC in a bond on the linux, or are they bonded by proxmox?

Setting this up on baremetal with Ubuntu is pretty straightforward and I thought this is pretty straightforward. But it’s like I’m missing something; I’ve said this before smb multichannel in Proxmox requires some kind of witchcraft and sacrifice to a pagan god.

 

[Adooni]

I am a bit surprised that you did not install proxmox BareMetal and then as VM set Xpenology or OMV etc.

 

[DJones]

I am a bit surprised that you did not install proxmox BareMetal and then as VM set Xpenology or OMV etc.

I need the full kernel support that Ubuntu offers. OMV and other platforms are fine if you’re using zfs, I’m not. In fact OMV, TrueNAS scale, ect, do not even easily support hardware raid or software raid of external storage bays. So it’s a no go right off the hop. Besides DIY is all about getting your hands dirty, I own turn key NAS’s, and they tend to be a pain if you do anything beyond the norm. I’m also not a fan of the ransomware that hits those OS’s due to manufacturer/developer negligence.

 

[sfx2000]

what does testparm return?

testparm

 

[sfx2000]

what does testparm return?

Something like this?

$ sudo testparm /etc/samba/smb.conf
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[share]"
Processing section "[media]"
Loaded services file OK.
Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

# Global parameters
[global]
    netbios name = TESTBOX
    security = USER
    idmap config * : backend = tdb
    hosts allow = 192.168.1.0/24


[share]
    comment = Home File Server
    path = /var/share
    force user = fileserver901
    force group = fileserver901
    group = fileserver901
    read only = No
    create mask = 0755

[media]
    comment = Home Media Server
    path = /var/media
    force user = fileserver901
    force group = fileserver901
    group = fileserver901
    read only = No
    create mask = 0755

BTW - screenshots are ok, but code is better..

 

[sfx2000]

Note that SMB3 multichannel requires two (2) NIC's - not virtual interfaces, but two real network cards...

 

[DJones]

Note that SMB3 multichannel requires two (2) NIC's - not virtual interfaces, but two real network cards...

Sorry maybe I wasn’t clear there is 4 total physical NICs involved 2 for server, 2 for client. The virtual NICs are simply to used to connect to the VM.

I'm going to just complicate the post more since I've changed the layout of my configuration again bringing them into the same subnet as my router. I've been attempting to troubleshoot through trial and error. Everything above is the same except I've attempted to route through the router, but I intend to change it back to how it was later. The smb.conf is the same except for the IP address's. I agree code is better, but their was post limitations in word count so I went with photos.

Load smb config files from /etc/samba/smb.conf
Unknown parameter encountered: "user"
Ignoring unknown parameter "user"
lpcfg_do_global_parameter: WARNING: The "encrypt passwords" option is deprecated
Unknown parameter encountered: "user"
Ignoring unknown parameter "user"
Unknown parameter encountered: "user"
Ignoring unknown parameter "user"
Unknown parameter encountered: "user"
Ignoring unknown parameter "user"
Unknown parameter encountered: "user"
Ignoring unknown parameter "user"
Loaded services file OK.
Weak crypto is allowed by GnuTLS (e.g. NTLM as a compatibility fallback)

Invalid combination of parameters for service NAS1. Level II oplocks can only be set if oplocks are also set.

Invalid combination of parameters for service NAS2. Level II oplocks can only be set if oplocks are also set.

Invalid combination of parameters for service NAS3. Level II oplocks can only be set if oplocks are also set.

Invalid combination of parameters for service NAS4. Level II oplocks can only be set if oplocks are also set.

Server role: ROLE_STANDALONE

Press enter to see a dump of your service definitions

# Global parameters
[global]
        bind interfaces only = Yes
        client signing = required
        deadtime = 30
        default service = global
        interfaces = 192.168.1.6/255.255.255.0;speed=1000000,capability=RSS 192.168.1.8/255.255.255.0;speed=1000000,capability=RSS
        load printers = No
        log file = /var/log/samba/log.%m
        logging = file
        map to guest = Bad User
        max log size = 1000
        netbios name = RACKSERVER
        obey pam restrictions = Yes
        panic action = /usr/share/samba/panic-action %d
        passwd chat = *Enter\snew\s*\spassword:* %n\n *Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
        passwd program = /usr/bin/passwd %u
        read raw = No
        server min protocol = SMB2
        server role = standalone server
        server signing = required
        server string = %h server (Samba, Ubuntu)
        socket options = SO_BROADCAST TCP_NODELAY IPTOS_LOWDELAY
        unix password sync = Yes
        usershare allow guests = Yes
        write raw = No
        idmap config * : backend = tdb
        delete readonly = Yes
        inherit acls = Yes
        inherit owner = windows and unix
        inherit permissions = Yes
        kernel oplocks = Yes
        oplocks = No
        use sendfile = Yes
        valid users =  root nobody


[NAS1]
        create mask = 0777
        directory mask = 0777
        path =
        vfs objects = io_uring
        write list =  root nobody


[NAS2]
        create mask = 0777
        directory mask = 0777
        path =
        vfs objects = io_uring
        write list =  root nobody


[NAS3]
        create mask = 0777
        directory mask = 0777
        path =
        vfs objects = io_uring
        write list =  root nobody


[NAS4]
        create mask = 0777
        directory mask = 0777
        path =
        vfs objects = io_uring
        write list =  root nobody

 

[DJones]

@sfx2000

Wasn't able to get smb multi channel working so I changed the whole layout and it's far more complex. I utilized a older nas installed Proxmox on it made a cluster of the two Proxmox machines, gave it LACP redundancy on the links between machines & router. Then from the older nas (which isn't old like 2019), moved my samba shares and files to it, since it can handle my 5Gb NIC, made a link local connection to my client and mapped the share. The share also can be accessed from my router at a slower 1Gb link for other networked devices. The share between my compute Proxmox machine and my share Proxmox machine unfortunately is still limited to basically 1Gb speeds. But I might add another share system NFS because it uses multipathing as well which might work better. But will handle that another day.

 

[sfx2000]

Wasn't able to get smb multi channel working so I changed the whole layout and it's far more complex. I utilized a older nas installed Proxmox on it made a cluster of the two Proxmox machines, gave it LACP redundancy on the links between machines & router. Then from the older nas (which isn't old like 2019), moved my samba shares and files to it, since it can handle my 5Gb NIC, made a link local connection to my client and mapped the share. The share also can be accessed from my router at a slower 1Gb link for other networked devices. The share between my compute Proxmox machine and my share Proxmox machine unfortunately is still limited to basically 1Gb speeds. But I might add another share system NFS because it uses multipathing as well which might work better. But will handle that another day.

Found this thread over in the proxmox forums, might be of some help.

[TUTORIAL] - Enabling SMB3 Multichannel in Proxmox

Hi! It was a bit challenging to find up-to-date information on how to get SMB3 multichannel and, especially, NFSv4 multipath (session and client ID trunking) working. I managed to sort out both on Linux (and Proxmox), so hopefully, this guide can save you some time. Initially, my plan was to...

forum.proxmox.com

BTW - there's a lot going on over in the smb.conf based on the testparm output - the default that ships w/samba, while complete, has a lot of things enabled that don't need to be for a small network like yours. A couple of things I didn't see though was Async IO being enabled, and not seeing that multichannel is turned on for SMB3 - this might be in your config file, but I'm not seeing it. You may have to add these lines and then kick samba to load them.

server multi channel support = yes
aio read size = 1
aio write size = 1

HTH...

 

[sfx2000]

Something like this?

here's my smb.conf for the testparm output - as you may see, it's fairly brief... if you add the SMB multi-channel and AIO settings put them in the global section.

Also note, we need aio to be enabled as a single task cannot use multiple nics, aio enables the thread support needed.

[global]
# comment the line TCP_NODELAY for some platforms - should be safe
# on gigabit ethernet, with an untuned kernel we're seeing about 115MB/Sec on large files
socket options = TCP_NODELAY
workgroup = WORKGROUP
netbios name = TESTBOX
security = user
hosts allow = 192.168.1.0/24
restrict anonymous = 2
# restrict SMB1
min protocol = SMB2

[share]
comment = Home File Server
path = /var/share
force user = fileserver901
force group = fileserver901
read only = no
browsable = yes
create mask = 0755

[media]
comment = Home Media Server
path = /var/media
force user = fileserver901
force group = fileserver901
read only = no
browsable = yes
create mask = 0755

Notice that the user fileserver901 is the linux/unix user that owns the directories and files, not the smbuser - samba uses it's own usernames internally for access...

I prefer to have dedicated SMB users, and that's defined by the smbpasswd for the smbuser

sudo useradd smbuser
sudo smbpasswd -a smbuser
New SMB password:
Retype new SMB password:
Added user smbuser

 

[DJones]

Found this thread over in the proxmox forums, might be of some help.

[TUTORIAL] - Enabling SMB3 Multichannel in Proxmox

Hi! It was a bit challenging to find up-to-date information on how to get SMB3 multichannel and, especially, NFSv4 multipath (session and client ID trunking) working. I managed to sort out both on Linux (and Proxmox), so hopefully, this guide can save you some time. Initially, my plan was to...

forum.proxmox.com

BTW - there's a lot going on over in the smb.conf based on the testparm output - the default that ships w/samba, while complete, has a lot of things enabled that don't need to be for a small network like yours. A couple of things I didn't see though was Async IO being enabled, and not seeing that multichannel is turned on for SMB3 - this might be in your config file, but I'm not seeing it. You may have to add these lines and then kick samba to load them.

server multi channel support = yes
aio read size = 1
aio write size = 1

HTH...

Thanks I’ll give it a look. Aio and server multi channel support actually is in the smb.conf I just checked. The testparm dump just didn’t include it which I find odd.

Also the latest samba versions default to aio read size = 1 and aio write size = 1 those functions are only there now incase you want to set it to 0. Any other number is now ignored by samba.

I use webmin which syncs the Unix users which is why it’s like that. The passwords are different however from sudo. One downside to webmin is it has functions that are deprecated like read raw or read prediction which doesn’t need to be included in the smb.conf. I'll likely tighten up the security on samba users.

[TUTORIAL] - Enabling SMB3 Multichannel in Proxmox

Hi! It was a bit challenging to find up-to-date information on how to get SMB3 multichannel and, especially, NFSv4 multipath (session and client ID trunking) working. I managed to sort out both on Linux (and Proxmox), so hopefully, this guide can save you some time. Initially, my plan was to...

forum.proxmox.com

Seems to be referring to getting smb multi channel to work on the proxmox host itself not the vm's it uses. You might want to do this if you have two proxmox machines or a samba share, and want to have one use a cifs share on the datacenter to store backups and such without using a proxmox cluster which has it's pro's and con's. Or the guy just wanted it that way I'm not sure. I think if I did use samba off the host and not off a vm it would work correctly as I used to run smb multichannel off this machine when I had ubuntu on baremetal.

My main priority was getting fast transfers to my main client. The router also connects to lan a switch which has 3 more computers, and my security cameras, and theirs another computer over AImesh, but I don’t really care about speed from them.