Harnessing the Power of Dedicated GPU Clusters: How MarQi Cloud Handles Parallel Training Jobs
May 14, 2026
Software-Defined Storage and Cloud SAN: Why SDS Is Rewriting the Rules of Enterprise Storage
May 14, 2026
NVMe over Fabrics vs. iSCSI for Cloud SAN: Which Storage Protocol Actually Wins in 2026?
How do NVMe arrays compare with traditional SANs for sub-millisecond latency?
NVMe over Fabrics maps commands directly onto the NVMe command set with parallel queues, removing the SCSI translation layer that iSCSI still carries. For latency-sensitive, queue-depth heavy workloads such as Oracle-backed trading systems that protocol overhead is the difference — the underlying media is often identical.
If you’re running block storage over a network — whether that’s a private cloud, a hosted SAN, or a hybrid setup — the protocol you choose matters more than most people realize. For years, iSCSI was the default answer: it ran on standard Ethernet, required no special hardware, and worked well enough. But “well enough” is losing ground fast. The right starting point for evaluating your options is understanding your storage architecture and where protocol choice fits within it.
NVMe over Fabrics (NVMe-oF) has moved from a niche high-performance option to a genuine mainstream competitor. In 2026, the performance gap between these two protocols is wide enough that staying with iSCSI without evaluating NVMe-oF means leaving measurable performance on the table. This post breaks down what each protocol actually does, where the numbers diverge, and how to decide which one belongs in your storage architecture.
What Is iSCSI, and Why Did It Dominate for So Long?
iSCSI — Internet Small Computer System Interface — maps SCSI commands over TCP/IP. Standardized in 2003, its longevity reflects how well it solved the problem of its era: give servers access to remote block storage without proprietary hardware or fiber runs.
The appeal was real. iSCSI runs on commodity Ethernet switches, works with standard NICs, and is supported by every major OS, hypervisor, and enterprise SAN vendor. Configuration is manageable, monitoring tooling is mature, and most ops teams know how to troubleshoot it.
Where iSCSI shows its age is the protocol stack itself. Every I/O request travels: application → OS → SCSI command → iSCSI encapsulation → TCP → IP → Ethernet → back the other way on the target side. Each layer adds latency. The SCSI command set was designed for spinning disks — it assumes storage is slow, with queue depths and command ordering that made sense in 1994 but actively hurt performance when your backend is NVMe SSDs doing sub-100 microsecond operations.
Measured 4K random read latency over iSCSI typically sits around 0.8–1.0ms in production environments. For large sequential transfers that may not matter. For databases, AI inference pipelines, or anything requiring high IOPS at low latency, that overhead is a real constraint.
NVMe over Fabrics: Built for the Storage Hardware of Today
NVMe was designed from scratch for solid-state storage. It exposes up to 65,535 I/O queues per device, each with up to 65,535 outstanding commands — compare that to a single iSCSI queue with 32 commands and you begin to understand the architectural difference.
NVMe over Fabrics extends that architecture across a network. Rather than wrapping NVMe in SCSI and then TCP, NVMe-oF transmits native NVMe commands directly. The protocol stack is flatter, the queue model scales to modern workloads, and the result is latency that reaches as low as 10–20 microseconds for remote storage — compared to iSCSI’s 800–1,000 microseconds.
NVMe-oF runs over multiple transports. NVMe/FC uses existing Fibre Channel fabrics and is favoured by organizations with established FC infrastructure. NVMe/RDMA (RoCE) delivers the lowest possible latency but requires lossless Ethernet with DCB/PFC configuration. NVMe/TCP is the fastest-growing variant: it runs on standard TCP/IP over any Ethernet network, has no specialized hardware requirements, and delivers latency in the 50–150 microsecond range — far below iSCSI, and deployable on existing infrastructure.
Real-World Performance: Where the Numbers Diverge
In controlled benchmarks with identical NVMe backend storage:
| Metric | iSCSI | NVMe/TCP | NVMe/RoCE |
|---|---|---|---|
| 4K Random Read Latency | ~900µs | ~120µs | ~15µs |
| 4K Random Read IOPS | ~180K | ~850K | ~1.2M+ |
| Sequential Read (per core) | ~4 GB/s | ~12 GB/s | ~18 GB/s |
| CPU overhead per GB/s | High | Medium | Low |
These numbers reflect the architectural differences directly. The queue depth advantage of NVMe-oF means more concurrent operations per CPU core, which translates to better throughput without adding hardware. For AI training workloads in particular, where GPUs sit idle waiting on data reads, that throughput gap maps directly to training time and cost.
Network Fabric Considerations
Protocol choice doesn’t exist in isolation — it’s inseparable from your underlying network infrastructure. iSCSI runs on any IP network and tolerates moderate packet loss, making it forgiving in environments where the network fabric isn’t perfectly tuned. NVMe/RoCE requires lossless Ethernet with Priority Flow Control (PFC) configured — errors in fabric configuration here translate directly to performance degradation. NVMe/TCP is considerably more tolerant, running on standard Ethernet without specialized configuration, which is why it’s the entry point most organizations use for NVMe-oF adoption.
When to Stay on iSCSI
iSCSI remains a legitimate choice in specific situations. If your workloads don’t require sub-millisecond storage latency, your ops team has deep iSCSI expertise, and your existing SAN infrastructure is stable and amortized — the migration cost likely doesn’t justify the performance gain. iSCSI also has broader hypervisor and OS support for legacy systems that may not have NVMe-oF initiator support.
The calculation changes when your workloads are storage-latency-sensitive, your IOPS requirements are growing, or you’re provisioning new infrastructure from scratch. In those cases, starting with iSCSI is building technical debt from day one.
Migration Path: From iSCSI to NVMe-oF
For most organizations, the migration is incremental rather than a rip-and-replace. NVMe/TCP is the recommended starting point — no new hardware required, and most modern Linux kernels (5.14+) include NVMe/TCP initiator support. Start by running NVMe/TCP alongside existing iSCSI volumes in a test environment, benchmark your specific workloads, and migrate services incrementally starting with the most latency-sensitive ones.
Data protection doesn’t pause during this migration. Your snapshots and backups strategy needs to account for volumes on both protocols during the transition period — most enterprise SDS platforms handle this cleanly, but verify before you migrate production hosts.
For teams running workloads on KVM-based infrastructure, the performance characteristics of KVM virtualization stack directly on top of storage protocol choices. A well-configured NVMe/TCP stack with KVM can close most of the gap between virtualized and bare metal storage performance for I/O-intensive workloads.
How This Fits Your Compute Layer
Storage protocol performance only matters in context of what your compute layer can actually consume. A well-tuned NVMe/TCP stack paired with underpowered servers still leaves performance on the table. Conversely, a high-core-count, high-memory compute platform with an iSCSI storage backend creates a bottleneck that limits the value of the compute investment. The protocol and compute decisions belong together in the architecture conversation.
What the Protocol Choice Signals About Your Architecture
The iSCSI vs. NVMe-oF decision isn’t purely technical — it reflects your infrastructure philosophy. Organizations defaulting to iSCSI in 2026 are often optimizing for operational familiarity and risk reduction. That’s legitimate in many environments. But if your workloads are growing, IOPS requirements are rising, and application performance is storage-bound, staying on iSCSI is a technical debt accumulation decision.
NVMe-oF adoption is accelerating in AI training and inference clusters, real-time analytics pipelines, financial transaction processing, and any application where storage latency maps directly to business outcomes. For environments that require both high performance and private, dedicated connectivity, pairing NVMe-oF storage with private circuits and peering eliminates the last remaining latency variables from the path between compute and storage.
Conclusion
iSCSI is not going away. For environments where it’s already deployed, workloads don’t require sub-millisecond latency, and operational stability is the priority, it remains solid. But the performance data in 2026 is clear: NVMe-oF — specifically NVMe/TCP — delivers materially better IOPS, lower latency, and higher throughput per CPU core across the workloads that matter most for modern cloud infrastructure.
The protocol decision belongs inside a broader storage architecture conversation: what tiers you need, how you handle multipath, how your hypervisor layer interacts with block storage, and what your application performance requirements actually are. Done well, the move to NVMe/TCP is an incremental migration, not a rip-and-replace.
If you’re evaluating cloud SAN options and want infrastructure built to handle these protocol choices correctly from day one, MarQi Cloud gives you dedicated storage performance without the overhead of legacy protocol stacks never designed for today’s NVMe-backed hardware.
Frequently asked questions
How much lower is NVMe-oF latency than iSCSI?
In controlled benchmarks with identical NVMe backend storage, 4K random read latency measures about 900 µs on iSCSI, about 120 µs on NVMe/TCP and about 15 µs on NVMe/RoCE. The gap comes from the protocol stack itself: NVMe-oF transmits native NVMe commands instead of wrapping them in SCSI and TCP.
Which NVMe over Fabrics transport should I start with?
NVMe/TCP. It runs on standard Ethernet without specialised configuration, requires no new hardware, and most modern Linux kernels (5.14 and later) already include NVMe/TCP initiator support. NVMe/RoCE is faster but requires lossless Ethernet with Priority Flow Control configured correctly.
Is iSCSI still a reasonable choice in 2026?
Yes, in specific situations: when your workloads do not require sub-millisecond storage latency, your operations team has deep iSCSI expertise, and your existing SAN infrastructure is stable and amortised. The calculation changes when workloads are latency-sensitive, IOPS requirements are growing, or you are provisioning new infrastructure from scratch.
How many IOPS does each protocol deliver in testing?
On the same NVMe backend, 4K random read throughput measures roughly 180K IOPS on iSCSI, 850K on NVMe/TCP and over 1.2M on NVMe/RoCE. Sequential read per core scales the same way: about 4 GB/s, 12 GB/s and 18 GB/s respectively.
Do I have to migrate everything at once?
No. For most organisations the migration is incremental rather than rip-and-replace. Run NVMe/TCP alongside existing iSCSI volumes in a test environment, benchmark your specific workloads, then migrate services one at a time starting with the most latency-sensitive. Verify that your snapshot and backup strategy covers volumes on both protocols during the transition.
