Message-based interfaces in Legato are implemented in layers. This low-level messaging API is at the bottom layer. It's designed to support higher layers of the messaging system. But it's also intended to be easy to hand-code low-level messaging in C, when necessary.
This low-level messaging API supports:
This API is integrated with the Legato Event Loop API so components can interact with each other using messaging without having to create threads or file descriptor sets that block other software from handling other events. Support for integration with legacy POSIX-based programs is also provided.
The Legato low-level messaging system follows a service-oriented pattern:
Clients and servers can both send one-way messages within a session. Clients can start a request-response transaction by sending a request to the server, and the server can send a response. Request-response transactions can be blocking or non-blocking (with a completion callback). If the server dies or terminates the session before sending the response, Legato will automatically terminate the transaction.
Servers are prevented from sending blocking requests to clients as a safety measure. If a server were to block waiting for one of its clients, it would open up the server to being blocked indefinitely by one of its clients, which would allow one client to cause a server to deny service to other clients. Also, if a client started a blocking request-response transaction at the same time that the server started a blocking request-response transaction in the other direction, a deadlock would occur.
Servers and clients have interfaces that can been connected to each other via bindings. Both client-side and server-side interfaces are identified by name, but the names don't have to match for them to be bound to each other. The binding determines which server-side interface will be connected to when a client opens a session.
Server-side interfaces are also known as "services".
When a session is opened by a client, a session reference is provided to both the client and the server. Messages are then sent within the session using the session reference. This session reference becomes invalid when the session is closed.
Communication between client and server is done using a message-based protocol. This protocol is defined at a higher layer than this API, so this API doesn't know the structure of the message payloads or the correct message sequences. That means this API can't check for errors in the traffic it carries. However, it does provide a basic mechanism for detecting protocol mismatches by forcing both client and server to provide the protocol identifier of the protocol to be used. The client and server must also provide the maximum message size, as an extra sanity check.
To make this possible, the client and server must independently call le_msg_GetProtocolRef()
, to get a reference to a "Protocol" object that encapsulates these protocol details:
myproto_Msg_t
contains a union
of all of the different messages included in the protocol, thereby making myproto_Msg_t
as big as the biggest message in the protocol.When a server creates a service (by calling le_msg_CreateService()) and when a client creates a session (by calling le_msg_CreateSession()), they are required to provide a reference to a Protocol object that they obtained from le_msg_GetProtocolRef().
Sending a Message
Receiving a Non-Response Message
Closing Sessions
Multithreading
Sample Code
Clients that want to use a service do the following:
le_msg_GetProtocolRef()
.le_msg_CreateSession()
, passing in the protocol reference and the client's interface name.le_msg_SetSessionRecvHandler()
.The Legato framework takes care of setting up any IPC connections, as needed (or not, if the client and server happen to be in the same process).
When the session opens, the Event Loop will call the "session open handler" call-back function that was passed into le_msg_OpenSession().
le_msg_OpenSessionSync() is a synchronous alternative to le_msg_OpenSession(). The difference is that le_msg_OpenSessionSync() will not return until the session has opened or failed to open (most likely due to permissions settings).
le_msg_TryOpenSessionSync() is like le_msg_OpenSessionSync() except that it will not wait for a server session to become available if it is not already available at the time of the call. That is, if the client's interface is not bound to any service, or if the service that it is bound to is not currently advertised by the server, then le_msg_TryOpenSessionSync() will return an error code.
Before sending a message, the client must first allocate the message from the session's message pool using le_msg_CreateMsg(). It can then get a pointer to the payload part of the message using le_msg_GetPayloadPtr(). Once the message payload is populated, the client sends the message.
If no response is required from the server, the client sends the message using le_msg_Send(). At this point, the client has handed off the message to the messaging system, and the messaging system will delete the message automatically once it has finished sending it.
If the client expects a response from the server, the client can use le_msg_RequestResponse() to send their message and specify a callback function to be called when the response arrives. This callback will be called by the event loop of the thread that created the session (i.e., the thread that called le_msg_CreateSession()).
If the client expects an immediate response from the server, and the client wants to block until that response is received, it can use le_msg_RequestSyncResponse() instead of le_msg_RequestResponse(). However, keep in mind that blocking the client thread will block all event handlers that share that thread. That's why le_msg_RequestSyncResponse() should only be used when the server is expected to respond immediately, or when the client thread is not shared by other event handlers.
When the client is finished with it, the client must release its reference to the response message by calling le_msg_ReleaseMsg().
When a server sends a message to the client that is not a response to a request from the client, that non-response message will be passed to the receive handler that the client registered using le_msg_SetSessionRecvHandler(). In fact, this is the only kind of message that will result in that receive handler being called.
The payload of a received message can be accessed using le_msg_GetPayloadPtr(), and the client can check what session the message arrived through by calling le_msg_GetSession().
When the client is finished with the message, the client must release its reference to the message by calling le_msg_ReleaseMsg().
When the client is done using a service, it can close the session using le_msg_CloseSession(). This will leave the session object in existence, though, so that it can be opened again using le_msg_OpenSession().
To delete a session object, call le_msg_DeleteSession(). This will automatically close the session, if it is still open (but won't automatically delete any messages).
Additionally, clients can choose to call le_msg_SetSessionCloseHandler() to register to be notified when a session gets closed by the server. Servers often keep state on behalf of their clients, and if the server closes the session (or if the system closes the session because the server died), the client most likely will still be operating under the assumption (now false) that the server is maintaining state on its behalf. If a client is designed to recover from the server losing its state, the client can register a close handler and handle the close.
However, most clients are not designed to recover from their session being closed by someone else, so if a close handler is not registered by a client and the session closes for some reason other than the client calling le_msg_CloseSession(), then the client process will be terminated.
The Low-Level Messaging API is thread safe, but not async safe.
When a client creates a session, that session gets "attached" to the thread that created it (i.e., the thread that called le_msg_CreateSession()). That thread will then call any callbacks registered for that session.
Note that this implies that if the client thread that creates the session does not run the Legato event loop then no callbacks will ever be called for that session. To work around this, move the session creation to another thread that that uses the Legato event loop.
Furthermore, to prevent race conditions, only the thread that is attached to a given session is allowed to call le_msg_RequestSyncResponse() for that session.
Processing Messages from Clients
Sending Non-Response Messages to Clients
Cleaning up when Sessions Close
Removing Service
Multithreading
Sample Code
Servers that wish to offer a service do the following:
Once the service is advertised, clients can open it and start sending it messages. The server will receive messages via callbacks to the function it registered using le_msg_SetServiceRecvHandler().
Servers also have the option of being notified when sessions are opened by clients. They get this notification by registering a handler function using le_msg_AddServiceOpenHandler().
Both the "Open Handler" and the "Receive Handler" will be called by the Legato event loop in the thread that registered those handlers (which must also be the same thread that created the service).
The payload of any received message can be accessed using le_msg_GetPayloadPtr().
If a received message does not require a response (i.e., if the client sent it using le_msg_Send()), then when the server is finished with the message, the server must release the message by calling le_msg_ReleaseMsg().
If a received message requires a response (i.e., if the client sent it using le_msg_RequestResponse() or le_msg_RequestSyncResponse()), the server must eventually respond to that message by calling le_msg_Respond() on that message. le_msg_Respond() sends the message back to the client that sent the request. The response payload is stored inside the same payload buffer that contained the request payload.
To do this, the request payload pointer can be cast to a pointer to the response payload structure, and then the response payload can be written into it.
Alternatively, the request payload structure and the response payload structure could be placed into a union together.
Whenever any message is received from a client, the message is associated with the session through which the client sent it. A reference to the session can be retrieved from the message, if needed, by calling le_msg_GetSession(). This can be handy for tagging things in the server's internal data structures that need to be cleaned up when the client closes the session (see Cleaning up when Sessions Close for more on this).
The function le_msg_NeedsResponse() can be used to check if a received message requires a response or not.
If a server wants to send a non-response message to a client, it first needs a reference to the session that client opened. It could have got the session reference from a previous message received from the client (by calling le_msg_GetSession() on that message). Or, it could have got the session reference from a Session Open Handler callback (see le_msg_AddServiceOpenHandler()). Either way, once it has the session reference, it can call le_msg_CreateMsg() to create a message from that session's server-side message pool. The message can then be populated and sent in the same way that a client would send a message to the server using le_msg_GetPayloadPtr() and le_msg_Send().
If a server keeps state on behalf of its clients, it can call le_msg_AddServiceCloseHandler() to ask to be notified when clients close sessions with a given service. This allows the server to clean up any state associated with a given session when the client closes that session (or when the system closes the session because the client died). The close handler is passed a session reference, so the server can check its internal data structures and clean up anything that it has previously tagged with that same session reference.
If a server wants to stop offering a service, it can hide the service by calling le_msg_HideService(). This will not terminate any sessions that are already open, but it will prevent clients from opening new sessions until it is advertised again.
The server also has the option to delete the service. This hides the service and closes all open sessions.
If a server process dies, the Legato framework will automatically delete all of its services.
The Low-Level Messaging API is thread safe, but not async safe.
When a server creates a service, that service gets attached to the thread that created it (i.e., the thread that called le_msg_CreateService()). That thread will call any handler functions registered for that service.
Note that this implies that if the thread that creates the service does not run the Legato event loop, then no callbacks will ever be called for that service. To work around this, you could move the service to another thread that that runs the Legato event loop.
Worthy of special mention is the fact that the low-level messaging system can be used to solve the age-old problem of coordinating the start-up sequence of processes that interact with each other. Far too often, the start-up sequence of multiple interacting processes is addressed using hacks like polling or sleeping for arbitrary lengths of time. These solutions can waste a lot of CPU cycles and battery power, slow down start-up, and (in the case of arbitrary sleeps) introduce race conditions that can cause failures in the field.
In Legato, a messaging client can attempt to open a session before the server process has even started. The client will be notified asynchronously (via callback) when the server advertises its service.
In this way, clients are guaranteed to wait for the servers they use, without the inefficiency of polling, and without having to add code elsewhere to coordinate the start-up sequence. Furthermore, if there is work that needs to be done by the client at start-up before it opens a session with the server, the client is allowed to do that work in parallel with the start-up of the server, so the CPU can be more fully utilized to shorten the overall duration of the start-up sequence.
Message buffer memory is allocated and controlled behind the scenes, inside the Messaging API. This allows the Messaging API to
Each message object is allocated from a session. The sessions' message pool sizes can be tuned through component and application configuration files and device configuration settings.
Generally speaking, message payload sizes are determined by the protocol that is being used. Application protocols and the packing of messages into message buffers are the domain of higher-layers of the software stack. But, at this low layer, servers and clients just declare the name and version of the protocol, and the size of the largest message in the protocol. From this, they obtain a protocol reference that they provide to sessions when they create them.
Security is provided in the form of authentication and access control.
Clients cannot open sessions with servers until their client-side interface is "bound" to a server-side interface (service). The binding thereby provides configuration of both routing and access control.
Neither the client-side nor the server-side IPC sockets are named. Therefore, no process other than the Service Directory has access to these sockets. The Service Directory passes client connections to the appropriate server based on the binding configuration of the client's interface.
The binding configuration is kept in the "system" configuration tree, so clients that do not have write access to the "system" configuration tree have no control over their own binding configuration. By default, sandboxed apps do not have any access (read or write) to the "system" configuration tree.
In rare cases, a server may wish to check the user ID of the remote client. Generally, this is not necessary because the IPC system enforces user-based access control restrictions automatically before allowing an IPC connection to be established. However, sometimes it may be useful when the service wishes to change the way it behaves, based on what user is connected to it.
le_msg_GetClientUserId() can be used to fetch the user ID of the client at the far end of a given IPC session.
It is possible to send an open file descriptor through an IPC session by adding an fd to a message before sending it. On the sender's side, le_msg_SetFd() is used to set the file descriptor to be sent. On the receiver's side, le_msg_GetFd() is used to get the fd from the message.
The IPC API will close the original fd in the sender's address space once it has been sent, so if the sender still needs the fd open on its side, it should duplicate the fd (e.g., using dup() ) before sending it.
On the receiving side, if the fd is not extracted from the message, it will be closed when the message is released. The fd can only be extracted from the message once. Subsequent calls to le_msg_GetFd() will return -1.
As a denial-of-service prevention measure, receiving of file descriptors is disabled by default on servers. To enable receiving of file descriptors, the server must call le_msg_EnableFdReception() on their service.
As an optimization to reduce the number of copies in cases where the sender of a message already has the message payload of their message assembled somewhere (perhaps as static data or in another message buffer received earlier from somewhere), a pointer to the payload could be passed to the message, instead of having to copy the payload into the message.
Perhaps an "iovec" version could be added to do scatter-gather too?
We explored the option of having asynchronous messages automatically released when their handler function returns, unless the handler calls an "AddRef" function before returning. That would reduce the amount of code required in the common case. However, we chose to require that the client release the message explicitly in all cases, because the consequences of using an invalid reference can be catastrophic and much more difficult to debug than forgetting to release a message (which will generate pool growth warning messages in the log).
If you are running as the super-user (root), you can trace messaging traffic using TBD. You can also inspect message queues and view lists of outstanding message objects within processes using the Process Inspector tool.
If you are leaking messages by forgetting to release them when you are finished with them, you will see warning messages in the log indicating that your message pool is growing. You should be able to tell by the name of the expanding pool which messaging service it is related to.
Copyright (C) Sierra Wireless Inc. Use of this work is subject to license.