L15 - Networking

Question 1

What is the communication hierarchy in a data center?

Answer 1

Server NIC –> Top of the Rack Switch –> Cluster Switch

Question 2

What are 4 interface mechanism for networking?

Answer 2

1. Register and queue based device interface
2. Memory mapped I/O, virtual memory, user/kernel modes
3. Direct memory access (DMA)
4. Interrupts and polling

Question 3

What is DMA?

Answer 3

separate engine to transfer blocks of data between an I/O device & memory (without involving the CPU)

Question 4

How does DMA work?

Answer 4

1. CPU describes the transfer to the DMA engine (source address, destination address, transfer size, type)
2. DMA engine performs memory/IO read/writes
3. DMA engine completes transfer by sending interrupt to CPU

note: CPU can execute other code during the DMA transfer

Question 5

What are the advantages of DMA?

Answer 5

It decouples data transfer from processing and thus:

1. CPU does not need to copy data to/from device
2. Doesn’t pollute CPU cache
3. Can be processed when the CPU (or OS) decides
4. Higher-performance: CPU and device work in parallel

Question 6

When is DMA disadvantageous?

Answer 6

for very small transfers: higher setup overhead

Question 7

Steps of DMA for ‘writing’?

Answer 7

1. Allocate DMA-able buffer with packet data
2. Write DMA descriptor to NIC
3. NIC DMA engine copies data for transmission
4. TX completion notification (interrupt)
5. Deallocate buffer

Question 8

Steps of DMA for ‘reading’?

Answer 8

1. Allocate buffer for received packet
2. Write DMA descriptor to NIC
3. NIC DMA engine copies data to host
4. RX notification to host (interrupt)
5. Forward received packet to user application

Question 9

What are the possible ways of solving DMA and cache inconsistencies?

Answer 9

1. CPU can map DMA buffers non-cacheable
2. Cache can “snoop” DMA bus transactions (but doesn’t scale beyond small SMP systems)
3. OS can explicitly flush/invalidate cache regions
4. Direct cache access

Question 10

What are the consequences of using physical addresses for DMA?

Answer 10

If physical address, DMA engine does not need to worry about translation. But there are a few things to watch out for:

• Need to use pinned addresses (cannot remap for non-IO use)
• The physical addresses in a multi-page transfer may not be contiguous –> need to break up into page-size DMA transfers

Question 11

What are the consequences of using virtual addresses for DMA?

Answer 11

DMA engine needs to handle translation.

Particularly useful when we have virtual machines sharing I/O device.