X-Mailing-List: linux-nfs@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Anton Gavriliuk Date: Sat, 29 Jun 2024 16:20:01 +0300 Message-ID: Subject: Are there any known NFS4.2 over TCP bandwidth limits ? To: linux-nfs@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Xref: photonic.trudheim.com org.kernel.vger.linux-nfs:86387 Newsgroups: org.kernel.vger.linux-nfs Path: photonic.trudheim.com!nntp.lore.kernel.org!not-for-mail Hi I have a NFS server with 6x100 Gb/s ports. The NFS server connected via two switches to the ESXi hosts. 3 NFS server ports per switch. Many ESXi hosts connected to the same switches using 25 Gb/s ports (4x25 Gb/s ports per single ESXi) The NFS server exports single mount point (raid6, many NVMe SSDs PCIe Gen5) VMs running with Oracle Linux 8.10 (kernel 4.18.0-553) There is a "magic" bandwidth number limited up to 100Gb/s between the NFS server and multiple VMs/ESXis. We tried different numbers of VMs/ESXis, different fio parameters (bs, iodepth, numjobs, reads/writes, etc...), and tried nconnect=16 option with mount command. Total bandwidth between the NFS server and multiple VMs/ESXis limited up to 100 Gb/s. Always. While iperf between the NFS and multiple VMs/ESXis shows hundreds Gb/s . Anton . From: Trond Myklebust To: "torvalds@linux-foundation.org" CC: "linux-nfs@vger.kernel.org" , "linux-kernel@vger.kernel.org" Subject: [GIT PULL] Please pull NFS client bugfix for Linux-6.10 Date: Sat, 29 Jun 2024 18:34:18 +0000 Message-ID: Accept-Language: en-US, en-GB Content-Language: en-US Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: base64 X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Xref: photonic.trudheim.com org.kernel.vger.linux-kernel:1263117 org.kernel.vger.linux-nfs:86392 Newsgroups: org.kernel.vger.linux-kernel,org.kernel.vger.linux-nfs Path: photonic.trudheim.com!nntp.lore.kernel.org!not-for-mail SGkgTGludXMsDQoNClRoZSBmb2xsb3dpbmcgY2hhbmdlcyBzaW5jZSBjb21taXQgNmJhNTlmZjQy Mjc5MjdkM2E4NTMwZmMyOTczYjgwZTk0YjU0ZDU4ZjoNCg0KICBMaW51eCA2LjEwLXJjNCAoMjAy NC0wNi0xNiAxMzo0MDoxNiAtMDcwMCkNCg0KYXJlIGF2YWlsYWJsZSBpbiB0aGUgR2l0IHJlcG9z aXRvcnkgYXQ6DQoNCiAgZ2l0Oi8vZ2l0LmxpbnV4LW5mcy5vcmcvcHJvamVjdHMvdHJvbmRteS9s aW51eC1uZnMuZ2l0IHRhZ3MvbmZzLWZvci02LjEwLTMNCg0KZm9yIHlvdSB0byBmZXRjaCBjaGFu Z2VzIHVwIHRvIDZkZGM5ZGVhY2MxMzEyNzYyYzJlZGQ5ZGUwMGNlNzZiMDBmNjlmN2M6DQoNCiAg U1VOUlBDOiBGaXggYmFja2NoYW5uZWwgcmVwbHksIGFnYWluICgyMDI0LTA2LTIxIDE3OjI2OjAy IC0wNDAwKQ0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tDQpORlMgY2xpZW50IGJ1Z2ZpeGVzIGZvciBMaW51eCA2LjEwDQoN CkJ1Z2ZpeGVzOg0KLSBTVU5SUEMgb25lIG1vcmUgZml4IGZvciB0aGUgTkZTdjQueCBiYWNrY2hh bm5lbCB0aW1lb3V0cw0KDQotLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0t LS0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tDQpDaHVjayBMZXZlciAoMSk6DQogICAgICBTVU5SUEM6 IEZpeCBiYWNrY2hhbm5lbCByZXBseSwgYWdhaW4NCg0KIG5ldC9zdW5ycGMvc3ZjLmMgfCA1ICsr KystDQogMSBmaWxlIGNoYW5nZWQsIDQgaW5zZXJ0aW9ucygrKSwgMSBkZWxldGlvbigtKQ0KDQot LSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNsaWVudCBtYWludGFpbmVyLCBIYW1tZXJz cGFjZQ0KdHJvbmQubXlrbGVidXN0QGhhbW1lcnNwYWNlLmNvbQ0KDQoNCg== .