This is question about architecture in an application that uses POSIX IPC to communicate between threads. The application uses multiple threads (clients) to send data to a single receiving (server) thread. Each thread is assigned to a separate core using its affinity mask. The threads are all within a single process – no process boundaries to cross. The most important factors are performance and reliability.
Currently I use a named pipe (FIFO) to communicate between the multiple writers and the single reader. The writers all use the same file descriptor and the reader reads from a single pipe.
However, the data must be processed in core (thread) order, with core 0 first, then core 1, then core 2, etc. With only a single pipe the application must organize the incoming messages in core order which adds extra processing overhead. The messages are added to a memory buffer maintained by the server side.
A better architecture from the standpoint of the reader (server) would be to use a separate pipe/socket/shared memory (or other IPC method) for each client. The server would read from each of the client file descriptors in core order, processing each record as it comes in, then read and process data from the next core, in a round-robin fashion. That way the server does not need to organize and process the records in core order, which is expensive. The server just receives them one at a time and processes them immediately upon receipt, then read from the next core in sequence, etc. No expense of a memory buffer or the overhead of organizing the records as they come in.
My question is, given the requirement described above, which of the POSIX IPC methods would be the best and most performant solution for this situation? I’m planning to go up to as many as 64 cores, so I would need up to as many as 63 file descriptors for the client side. I don’t need bidirectional commo.
The lowest system overhead would (I think) be an anonymous pipe. The server side could simply loop through an array of file descriptors to read the data. However, I’m not clear whether an anonymous pipe can be used for threads in a single process because, “It is not very useful for a single process to use a pipe to talk to itself. In typical use, a process creates a pipe just before it forks one or more child processes.” https://www.gnu.org/software/libc/manual/html_node/Creating-a-Pipe.html#Creating-a-Pipe
I currently use named pipes, which do work with threads in a single process, and which should work with multiple file descriptors.
I have also used UNIX domain datagram sockets with a single socket. My impression is that multiple sockets may be more system overhead than I need for this situation, but they may be the most performant solution.
Finally, I have considered POSIX shared memory, where each client thread has its own shared memory object. Shared memory is often described as the fastest IPC mechanism (https://www.softprayog.in/programming/interprocess-communication-using-posix-shared-memory-in-linux)
But with shared memory, there is the problem of synchronization. While the other IPC methods are basically queues where the data can be read one record at a time, shared memory requires a synchronization object like a semaphore or spinlock. As the man pages say, “Typically, processes must synchronize their access to a shared memory object, using, for example, POSIX semaphores.” (https://www.man7.org/linux/man-pages/man7/shm_overview.7.html.)
My concern is that the extra synchronization overhead may reduce the usefulness of shared memory in this situation.
Moreover, despite being billed as the fastest method, I am concerned about possible cache contention with shared memory. “(M)any CPUs need fast access to memory and will likely cache memory, which has two complications (access time and data coherence).” https://en.wikipedia.org/wiki/Shared_memory.
I could test each of these solutions, but before I choose one it would be helpful to ask the opinions of others on which IPC method would be the best for multiple pipes/sockets/shared memory for multiple clients, as described above.