diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/architecture.txt | 98 | ||||
| -rw-r--r-- | docs/mellanox.txt | 88 | ||||
| -rw-r--r-- | docs/todo.txt | 7 |
3 files changed, 193 insertions, 0 deletions
diff --git a/docs/architecture.txt b/docs/architecture.txt index 2b00ab3..e38da42 100644 --- a/docs/architecture.txt +++ b/docs/architecture.txt @@ -16,3 +16,101 @@ Configuration - Metadata seems preserved while passing trough standard UFO filters. So, this is easiest way out. - We can build a stand-alone subgraph for each plane, but this likely involves a full data copy. - This probably OK for simple use case: "raw storage + visualization" as only two instances, but for could be too challenging for multi-plane analysis. + + +Overloaded Mode +=============== + x How/when do we stop capturing in fixed frame mode. [ After building required number of frames ] + - After capturing theoretical number of packets required for reconstruction (or plus one frame to address half) + * Initail packets are likely broken as our experiments with LibVMA shows... Likely will be less... + * As work-around, we can _always_ skip first few frames to allow system become responsive. + * Overall, this seems a less reliable system. + => After initial building of the required number of packets. + * Unlike previous case, this may never finish if processing is bottleneck as buffers will be overwritten... + * Alternative is to stop capturing frames if buffers are exhausted... + x Do we pause receiving when buffer is exhausted or do we start over-writting (original ufo variant is overwritting). [ stop ] + - For streaming, the overwritting is better as it is better skipping older frames rather than newer. + => But for capturing fixed number of frames, we need to stop streaming (in overloaded case) or the frames will be continuously overwritten. + * This is more repeatable way to handle the frames and something has to be optimized anyway if we are to slow in streaming mode, + but in this mode we can reliably get first frames even if event-building is overloaded. + x Do we always skip first frames to keep system responsive or we process normally? [ Not now, TBD later if necessary ] + - Technically, skipping first frames could allow faster stabilization. + - Could only cause problems if streaming is started by some trigger and first frames are really important. + - This would be mandatory in fixed-frame-mode if stopping independent of reconstruction. + +Data Flow Model +=============== + x MP-RQ vs recvfrom_zcopy vs SocketXtreme + => MP-RQ is least overhead method (seems to remove one memcpy) with requirements fitting our use-case + + x How to abstract read/LibVMA read access? + => Return pointer to buffer, number of packets, and padding between packets. + + x Possible Models: Buffer independent streams or buffer sinograms + - Sinogram-buffer (push-model) + * Sinograms are buffered. The readers are directly writting to appropriate place in the sinogram buffers and increment fragment + counter for each buffer. + * Readers are paused on the first item out of current buffer and wait until buffer start is advanced. + * After receving fragment for a new sinogram, the reader informs controller about number of missing framgements in the buffer. + E.g. counting 'missing' fragments using atomic array (on top of completed). + * The controller advances buffer start after 'missing' is increased above the specified threshold. + - Stream-buffers (pull-model) + * Data is directly buffered in the independent receive-buffers. So, there is no memcpy in the receiving part of the code. + * Master builder thread determines sinogram to build (maximum amongst buffer). + * Builder threads skip to the required sinogram and start copying data until missing fragment is detected + * On missing fragment, new sinogram is determined and operation restarted (when to skip large amount?) + + x Locking with global buffer (No problems if buffering independent streams) + - We can syncrhonize "Advances of Buffer Start" and "Copy out" with locks as this is low frequency events, but we need to ensure that + copy fragments are lock-less. + - In the 'counting' scenario this is problematc: + - Different threads write to diffent parts of the buffer. If the buffer start moves, it is OK to finish the old fragment. It will be + rewritten by the same thread with new data later. No problems here. + - But how to handle framgent counting? Atomics are fine for concurrent threads. But if we move buffer, the fragment count should not + be increased. This means we need execute 'if' and 'inc' atomicly (increase only if buffer has not moved and the move could happen between + 'if' and 'inc'). This is the main problem. + - We can push increases to pipe and use the main thread (which also advances the start of the buffer) to read from the pipe and + increase counts (or ignore if buffer moved). Is it faster than locking (or linux pipes perform locking or kernel/user-space + switches anyway?) + ? Can we find alternative ways without locks/pipes? E.g. using hashes? Really large counting arrays? + + ? Performance considerations + - Sinogram-buffer advantage + - Sinogram-buffer works fine with re-ordered data streams. The stream-buffer approach can handle the re-odered streams in theory, but + with large performance penalty and incresed complexithy. But in fact, there is little reason why re-ordering could happen and experiments + with existing switch doesn't show any re-ordering. + - There is always uncached reads. However, in sinogram-buffer it is uncached copy of large image. And in case of stream-buffer, the + many small images are accessed uncached. + ? With Sinogram-buffer we can prepare data in advance and only single memcpy will be required. With stream-buffer all steps are performed + only once the buffer is available. But does it has any impact on performance? + ? With stream-buffer, building _seems_ more complex and requires more synchronization (but only if we could find a simple method to avoid + locking in the sinogram-buffer scenario). + + => Stream-buffer advantage + - With MP-RQ we can use Mellanox ring-buffer (and probably Mellanox in general) as stream-buffer. + - Stream-buffer incures singifincantly less load during the reading phase. If building overloads system, with this approach we can + buffer as much data as memory permits and process it later. This is not possible with sinogram-buffer approach. But we still have + socket buffers... + - Stream-buffer removes one 'memcpy' unless zero-copy SocketExtreme is used. We also need to store raw data with fastwriter and the + new external sinogram buffer could be used to store data also for fastwriter. Hence, removing the necessity to memcpy there. + I.e. solved with SocketExtreme otherwise either performance penalty or additional complexity here. + ? Stream-buffer simplifies lock management. We don't need to provide additional pipes to reduce amount of required locking. Or is + there a simple solution? + + ? Single-thread builder vs multi-thread builder? + + ? LRO (Large Receive Offload). Can we use it? How it compareswith LibVMA? + + + + + + +Dependencies +============ + x C11 threads vs pthreads vs glib threads + * C11 threads are only supported starting with glibc 2.28. Ubuntu 18.04 sheeps 2.27. There are work-arounds, but there is + little advantage over pthreads. + * We still will try to use atomics from C11. + + diff --git a/docs/mellanox.txt b/docs/mellanox.txt new file mode 100644 index 0000000..ed20048 --- /dev/null +++ b/docs/mellanox.txt @@ -0,0 +1,88 @@ +Terminology +=========== + - Send/Receive Queues + QP (Queue Pair): Combines RQ and SQ. Generally, irrelevant for the following + RQ (Receive Queue): + SQ (Send Queue): + CQ (Completion Queue): Completed operations reported here + EQ (Event Queue): Completions generate events (at specified rate) which in turn generate IRQs + WR/WQ (Work Request Queue): This is basically buffers (SG-lists) which should be either send or used for data reception + *QE (* Queue Event) + + Flow: WQE --submit work--> WQ --execute--> SQ/RQ --on completion-> CQ --signal--> EQ -> IRQ + * Completion Event Moderation: Redeuce amount of reported events (EQ) + + - Ofloads + RSS (Receive Side Scalling): Distribute load across CPU cores + LRO (Large Receive Offload): Group packets and deliver to user-space as a large single grouped packet [ ethtool -K shows if LRO on/off ] + + - Various + AEV (Asynchronous Event): Errors,etc. + SRQ (Shared Receive Queue): + ICM (Interconnect Context Memory): Address Translation Tables, Control Objects, User Access Region (registers) + MPT (Memory Protection Table): + RMP (Receive Memory Pool): + TIR (Transport Interface Receive): + RQT (RQ Table): + MCG (Multicast Group): + +Driver +====== + - Network packets is/are streamed to ring buffers (with all Ethernet, IP, UDP/TCP headers). + The number of ring buffers dependents on VMA_RING_ALLOCATION parameter: + 0 - per network interface + 1 - per IP + => 10 - per socket + 20 - per thread (which was used to create the socket) + 30 - per core + 31 - per core (with some affinity of threads to cores) + + - The memory for ring buffer is allocated based on VMA_MEM_ALLOC_TYPE: + 0 - malloc (this will be very slow if large buffers are requested) + 1 - contigous + => 2 - HugePages + + - The number of buffers per ring is controlled with VMA_RX_BUFS (this is total in all rings) + * Each buffer VMA_MTU bytes + * Recommended: VMA_RX_BUFS ~ #rings * VMA_RX_WRE (number of WRE allocated on all interfaces) + +LibVMA +====== + There is 3 interfaces: + - MP-RQ (Multi-packet Receive Queue): vma_cyclic_buffer_read + This is useful for processing data streams when packet size stays contant and the packet flow doesn't change + drastically over time. Requires ConntextX-5 or newer. + + * Use 'vma_add_ring_profile' to configure the size of ring buffer (specifies buffer size & the packet size) + * Set per-socket SO_VMA_RING_ALLOC_LOGIC using setsockopt + * Call 'vma_cyclic_buffer_read' to access raw ring buffer, specifies minimum and maximum packets to return + + * The returned 'completion' structure referencing the position in the ring buffer. Packets in ring buffer + include all headers (ethernet - 14 bytes, ip - 20 bytes, udp - 8 bytes). + * New packets meanwhile are written in the remaining part of the ring buffer (until the linear end of the + buffer - consequently the returned data is not overwritten). + * The buffer rewinded only on call to 'vma_cyclic_buffer_read'. Less than the specified minimum amount of + packets can be returned if currently near the end of buffer and not enough space to fullfil the minimum + requirement. + + * To ensure enough space for the follow up packets, synchronization between buffer size and min/max packet + is required. It should never happen that the space for only few packets is left when end of the buffer is + close. + + - SocketXtreme: socketxtreme_poll + More complex interface allowing more control over process particularly processing packets with varing size. + Requires ConnectX-5 or newer. + + * Get ring buffers associated with socket 'get_socket_rings_num' and 'get_socket_rings_fds' + * Get ready completions on the specified ring buffer with 'socketxtreme_poll' (pass 'fd' returned with 'get_socket_rings_fds') + * Two types of completions: 'VMA_SOCKETXTREME_NEW_CONNECTION_ACCEPTED' and 'VMA_SOCKETXTREME_PACKET'. + * For the second type, process an associated list of buffers and keep reference counting with 'socketxtreme_ref_vma_buf', + 'socketxtreme_free_vma_buf'. + * Clean/unreference received packets with socketxtreme_free_vma_packets + + - Zero Copy: recvfrom_zcopy + The simplest interface working with ConnectX-3 cards. The packet is still written to ring-buffers. The data is not copied out + of ring buffers. This interface provides a way to get pointers to locations in ring buffer. There is a slight overhead compared + to MP-RQ approach to prepare list of packet pointers. + + diff --git a/docs/todo.txt b/docs/todo.txt index a9ab4c8..07258f3 100644 --- a/docs/todo.txt +++ b/docs/todo.txt @@ -11,6 +11,13 @@ Main - Try UFO visualization filter - "Reconstructed data storage" and "Visualization + raw data storage" modes. Implement stand-alone 'roof-converter' filter. +Network +======= + - Implement MP-RQ and corresponding abstractions + - LRO (Large Receive Offload). Can we use it? How it compareswith LibVMA? + - Check we can pre-allocate big enough buffers with LibVMA to receive required number of sinograms without loses + + If necesary =========== - Task 'roof-ingest-missing' to ingest zero-padded broken frames (and include get_writer()) |