Issue with MCTP Sockets Receiving Unexpected Packets from Different I3C PIDs

[chandu-tmba]

I am working with MCTP sockets and using MCTP Extended Addressing to communicate with a specific device over I3C. In my case, I am using the I3C PID (Provisional ID) to target a particular device.

While I am successfully receiving MCTP packets from the expected device, I am also unexpectedly receiving packets from other devices with different PIDs. This is not the expected behavior, as I would expect to receive only packets from the targeted device.

Is there a way to filter packets in the MCTP socket layer to ensure that only packets from the specified PID are received?

[chandu-tmba]

code

` // Open the mctp socket file
struct sockaddr_mctp_ext addr = { 0 };

int val;

sd = socket(AF_MCTP, SOCK_DGRAM, 0);
if (sd < 0)
	err(EXIT_FAILURE, "socket");

memset(&addr, 0x0, sizeof(addr));
addr.smctp_base.smctp_family = AF_MCTP;
addr.smctp_base.smctp_network = MCTP_NET_ANY;
addr.smctp_base.smctp_type = MCTP_CTRL_TYPE;



int ret_val = bind(sd, (struct sockaddr *)&addr, sizeof(addr));
if (ret_val < 0) {
    err(EXIT_FAILURE, "socket");
    close(sd);
    return -1;
}

addr.smctp_halen = I3C_PID_SIZE;
memcpy(addr.smctp_haddr, pid, I3C_PID_SIZE);
addr.smctp_ifindex = index;

val = 1;
rc = setsockopt(sd, SOL_MCTP, MCTP_OPT_ADDR_EXT, &val, sizeof(val));
if (rc < 0)
	errx(EXIT_FAILURE,
		"Kernel does not support MCTP extended addressing");

struct pollfd fds[1];
int rc;

// Setup first pollfd
fds[0].fd = asti3c->fd;
fds[0].events = POLLIN ;

rc = poll(fds, 1, timeout);

if (rc > 0)
	return fds[0].revents;

if (rc < 0) {
	mctp_prwarn("Poll returned error status (errno=%d)", errno);
	return -1;
}

if pollin event, the following callback is called

int ret = recvfrom(asti3c->fd, NULL, 0, MSG_PEEK | MSG_TRUNC, NULL, 0);
if (ret < 0)
{
	mctp_prerr("recvfrom size failed \n");
	return -1;
}		
read_len = (size_t)ret;

if (read_len > MCTP_I3C_BUFFER_SIZE) 
{
	mctp_prerr("read len greater than max buffer \n");
	return -1;
}

/* receive response */
int addrlen = sizeof(struct sockaddr_mctp_ext);
rc = recvfrom(asti3c->fd, data, read_len, MSG_TRUNC,
		(struct sockaddr *)&addr,(unsigned int *) &addrlen);
if (rc < 0)
	mctp_prerr("recvfrom msg failed \n");

`

[chandu-tmba]

The received packet is a discovery notify message with a null destination EID

01 00 00 C8 00 81 0D

These two devices, with different PIDs, are on two different I3C buses.

[jk-ozlabs]

While I am successfully receiving MCTP packets from the expected device, I am also unexpectedly receiving packets from other devices with different PIDs.

You have done a bind() on the socket:

memset(&addr, 0x0, sizeof(addr));
addr.smctp_base.smctp_family = AF_MCTP;
addr.smctp_base.smctp_network = MCTP_NET_ANY;
addr.smctp_base.smctp_type = MCTP_CTRL_TYPE;

int ret_val = bind(sd, (struct sockaddr *)&addr, sizeof(addr));

Indicating that you want to receive all control packets.

Consequently, you are receiving all control packets.

as I would expect to receive only packets from the targeted device

There is no "targetted device" here. The addr argument for recvmsg is an output only.

[chandu-tmba]

What if I copy PID to smctp_haddr before binding? Will it make any difference?

i want to receive all control packets coming from devcie with specififc PID

addr.smctp_halen = I3C_PID_SIZE;
memcpy(addr.smctp_haddr, pid, I3C_PID_SIZE);
addr.smctp_ifindex = index;

[jk-ozlabs]

What if I copy PID to smctp_haddr before binding? Will it make any difference?

No, the address argument to bind() represents a local address, not a remote one.

i want to receive all control packets coming from devcie with specififc PID

There is no facility to do that. can you just ignore the packets that are not from the PID that you are interested in?

[chandu-tmba]

We have a scenario where the BMC acts as a bridge and it will act as both busowner and endpoint . We are writing different handlers for bus owner and endpoint implementations

[jk-ozlabs]

You'll need to handle both in the same process then.

However, you know that there is already a MCTP control protocol implementation (as mctpd), right? While it's mainly focused on bus-owner mode, there is growing support for an endpoint mode as well.

[chandu-tmba]

I am running two different services, each with its own socket initialized and configured for MCTP extended addressing. I'm using poll() to wait for events and MSG_PEEK to check the PID(recvfrom call addr) while ignoring messages from different PIDs. However, only one socket is receiving POLLIN events, and the other is not. Is this exptected behaviour?

[jk-ozlabs]

How have you set up those two sockets? can you share the code?

[jk-ozlabs]

(but yes, that is somewhat expected; any specific message would only be delivered to one socket)

[chandu-tmba]

mctp_asti3c_sock_rx will be called when there is a polling(EPOLLIN) event

static int binding_asti3c_sock_process(struct binding *binding)
{
	int rc;
	rc = mctp_asti3c_poll(binding->data, MCTP_ASTI3C_POLL_TIMEOUT);
	if (rc & POLLIN){
		rc = mctp_asti3c_sock_rx(binding->data);
		MCTP_ASSERT_RET(rc == 0, rc, "mctp_asti3c_rx returned %d", rc);
	}

	return rc;
}

static int binding_asti3c_sock_init(struct mctp *mctp, struct binding *binding,
	mctp_eid_t eid, int n_params,
	char *const *params __attribute__((unused)))
{
struct mctp_binding_asti3c *i3c = NULL;
char *dev_file_path = NULL;
uint16_t mode = 0;
int rc = 0;
int sd = -1;
char pid[6];
unsigned int index = 0;

//mctp_prerr("eid: %d ", eid);

// Options for parameter parsing
struct {
char *prefix;
void *target;
} options[] = {
{ "dev_path=", &dev_file_path },
{ "mode=", &mode },
{ NULL, NULL },
};

binding->bindings_changed = false;


// Ensure dev_file_path is not NULL before opening
if (!dev_file_path) {
mctp_prerr("Device file path not provided");
return -1;
}

// Initialize I3C binding
i3c = mctp_asti3c_sock_init(dev_file_path, pid, &index);
MCTP_ASSERT_RET(i3c != NULL, -1, "Could not initialize I3C binding");

// Open the mctp socket file
struct sockaddr_mctp_ext addr = { 0 };

int val;

sd = socket(AF_MCTP, SOCK_DGRAM, 0);
if (sd < 0)
err(EXIT_FAILURE, "socket");

memset(&addr, 0x0, sizeof(addr));
addr.smctp_base.smctp_family = AF_MCTP;
addr.smctp_base.smctp_network = MCTP_NET_ANY;
addr.smctp_base.smctp_type = MCTP_CTRL_TYPE;


int ret_val = bind(sd, (struct sockaddr *)&addr, sizeof(addr));
if (ret_val < 0) {
err(EXIT_FAILURE, "socket");
close(sd);
return -1;
}

addr.smctp_halen = I3C_PID_SIZE;
memcpy(addr.smctp_haddr, pid, I3C_PID_SIZE);
addr.smctp_ifindex = index;
printf("      ext ifindex %d ha[0]=0x%02x len %hhu\n",
addr.smctp_ifindex,
addr.smctp_haddr[0], addr.smctp_halen);
val = 1;
rc = setsockopt(sd, SOL_MCTP, MCTP_OPT_ADDR_EXT, &val, sizeof(val));
if (rc < 0)
errx(EXIT_FAILURE,
"Kernel does not support MCTP extended addressing");

// Store file descriptor in the I3C structure
i3c->fd = sd;

mctp_register_bus(mctp, mctp_binding_asti3c_core(i3c), eid);

// Store the I3C binding in the binding data
binding->data = i3c;

enable_rx_control_message = true;
// Free dynamically allocated resources
free(dev_file_path);

return 0;
}

int mctp_asti3c_sock_rx(struct mctp_binding_asti3c *asti3c)
{
	mctp_prerr("mctp_asti3c_sock_rx called");
	uint8_t data[MCTP_I3C_BUFFER_SIZE];
	char cmd[128];
	struct sockaddr_mctp_ext addr = { 0 };
	struct mctp_asti3c_pkt_private pkt_prv;
	struct mctp_pktbuf *pkt;
	ssize_t read_len;
	int rc;

	if (asti3c->fd < 0) {
		mctp_prerr("Invalid file descriptor");
		return -1;
	}

	int ret = recvfrom(asti3c->fd, NULL, 0, MSG_PEEK | MSG_TRUNC, NULL, 0);
	if (ret < 0)
	{
		mctp_prerr("recvfrom size failed \n");
		return -1;
	}		
	read_len = (size_t)ret;

	if (read_len > MCTP_I3C_BUFFER_SIZE) 
	{
		mctp_prerr("read len greater than max buffer \n");
		return -1;
	}

	/* receive response */
	int addrlen = sizeof(struct sockaddr_mctp_ext);
	rc = recvfrom(asti3c->fd, data, read_len, MSG_TRUNC,
			(struct sockaddr *)&addr,(unsigned int *) &addrlen);
	if (rc < 0)
		mctp_prerr("recvfrom msg failed \n");

	if (!(addrlen == sizeof(struct sockaddr_mctp_ext) ||
		addrlen == sizeof(struct sockaddr_mctp)))
	{	
		return -1;		
	}


	if ((read_len > (MCTP_BTU + MCTP_HEADER_SIZE)) ||
	    (read_len < (MCTP_HEADER_SIZE))) {
		mctp_prerr("Incorrect packet size: %zd", read_len);
		return -1;
	}

	mctp_trace_rx(&data, read_len);

	/* PEC is verified at hardware level and does not
	propogate to userspace, thus do not deal with PEC byte */
	pkt = mctp_pktbuf_alloc(&asti3c->binding, 0);
	if (!pkt) {
		mctp_prerr("pktbuf allocation failed");
		return -1;
	}

	rc = mctp_pktbuf_push(pkt, data, read_len);

	if (rc) {
		mctp_prerr("Cannot push to pktbuf");
		mctp_pktbuf_free(pkt);
		return -1;
	}

	memcpy(pkt->msg_binding_private, &pkt_prv, sizeof(pkt_prv));


	mctp_bus_rx(&asti3c->binding, pkt);

	return 0;
}

[jk-ozlabs]

So I assume you're opening two sockets, with the same saddr_type value (of 0), right?

In which case: That won't work. Like other socket types (UDP, TCP, etc), you cannot have two sockets bound to the same port/address value.

So, only one of those sockets will actually receive the control packets. The second bind() should fail, but it looks like we don't check for that in the kernel at present, I will add that error case.

[chandu-tmba]

ok ,now i am following different approach based on your inputs, creating only socket to recive all type 0 messages . attached code below

in kernel i am able to recieve messages mctp-i3c-target

[ 151.942791] mctp-i3c: Transmit data: 01 1d 09 c5 00 1b 02 00 09 10 00 43
[ 153.574727] mctp-i3c: Read data: 01 09 1d c8 00 9c 0a 00 3e
[ 153.666690] mctp-i3c: Transmit data: 01 1d 09 c0 00 1b 02 00 77
[ 154.263110] mctp-i3c-target: Read data: 01 00 00 c8 00 81 0d 72
[ 157.263140] mctp-i3c-target: Read data: 01 00 00 c8 00 81 0d 72
[ 160.263111] mctp-i3c-target: Read data: 01 00 00 c8 00 81 0d 72
[ 161.856015] mctp-i3c: Read data: 01 09 1d cd 00 99 02 e7
[ 161.947083] mctp-i3c: Transmit data: 01 1d 09 c5 00 19 02 00 09 10 00 11
[ 163.263080] mctp-i3c-target: Read data: 01 00 00 c8 00 81 0d 72

but in application layer i am receviing only packets from mctp-i3c interface

mctp addr and route output

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/socket.h>
#include <linux/mctp.h>
#include <poll.h>
#include <systemd/sd-event.h>

#define TARGET_EID 0x1D            // Destination EID
#define MCTP_TYPE_CONTROL 0x00  // MCTP Control Message Type
#define MCTP_ADDR_ANY 0xff
#define MCTP_CMD_GET_EID 0x02
#define MCTP_CMD_SET_EID 0x01
#define MCTP_CMD_ALLOC_POOL_ID 0x08
#define MCTP_GET_EP_UUID 0x03
#define MCTP_GET_MSG_TYPE 0x05
#define CMD_INDEX 1


int add_local_addr()
{
    int ret;
    ret = system("mctp addr add 9 dev mctpi3c0");
    if (ret < 0) {
        perror("system command 'mctp addr' failed");
        goto err;
    }

    ret = system("mctp link set mctpi3c0 up");
    if (ret < 0) {
        perror("system command 'mctp link set' failed");
        goto err;
    }

    ret = system("mctp addr add 9 dev mctpi3c-target0");
    if (ret < 0) {
        perror("system command 'mctp addr' failed");
    }

    ret = system("mctp link set mctpi3c-target0 up");
    if (ret < 0) {
        perror("system command 'mctp link set' failed");
    }

err:
    return ret;
}


int send_discoverynotify()
{
    struct sockaddr_mctp addr = {0};
    struct sockaddr_mctp_ext recv_addr = {0};
    uint8_t buf_tx[2];
    int sd, rc, ret, len; 
    socklen_t recv_addrlen;

    /* create the MCTP socket */
    sd = socket(AF_MCTP, SOCK_DGRAM, 0);
    if (sd < 0)
        printf("socket() failed\n");

    /* populate the remote address information */
    addr.smctp_family = AF_MCTP;         /* we're using the MCTP family */
    addr.smctp_addr.s_addr = TARGET_EID; /* send to remote endpoint ID 8 */
    addr.smctp_type = MCTP_TYPE_CONTROL; /* encapsulated protocol type (eg. PLDM = 1) */
    addr.smctp_tag = MCTP_TAG_OWNER;     /* we own the tag, and so the kernel
                                            will allocate one for us */

    buf_tx[0] = 0x81;
    buf_tx[1] = 0x0d;

    /* send the MCTP message */
    rc = sendto(sd, buf_tx, sizeof(buf_tx), 0,
                (struct sockaddr *)&addr, sizeof(addr));

    if (rc != sizeof(buf_tx))
    {
        printf("sendto failed \n");
        return -1;
    }
    ret = recvfrom(sd, NULL, 0, MSG_PEEK | MSG_TRUNC, NULL, 0);
    if (ret < 0)
    {
        printf("recvfrom msg peek failed \n");
        return -1;
    }
    len = (size_t)ret;

    uint8_t* rxbuf;
    rxbuf = malloc(len);
    if (!rxbuf)
    {
        printf("malloc failed \n");
        return -1;
    }

    /* receive response */
    rc = recvfrom(sd, rxbuf, len, MSG_TRUNC,
                  (struct sockaddr *)&recv_addr, &recv_addrlen);
    if (rc < 0)
    {
        printf("recvfrom  failed \n");
        return -1;
    }

    if (!(recv_addrlen == sizeof(struct sockaddr_mctp_ext) ||
          recv_addrlen == sizeof(struct sockaddr_mctp)))
    {
        printf("unknown recv address length \n");     
        return -1;
    }

    printf("req:  message from (net %d, eid %d) type %d len %zd\n",
        recv_addr.smctp_base.smctp_network, recv_addr.smctp_base.smctp_addr.s_addr,
        recv_addr.smctp_base.smctp_type, len);

    if (recv_addrlen == sizeof(struct sockaddr_mctp_ext))
    {
        printf(" ext ifindex %d ha[0]=0x%02x len %hhu\n", recv_addr.smctp_ifindex, recv_addr.smctp_haddr[0], recv_addr.smctp_halen);
        for(int j=0; j<  recv_addr.smctp_halen; j++)
            printf("0x%02x:",recv_addr.smctp_haddr[j]);
        printf("\n");         
    }

    printf("response data:\n");
    for (int i = 0; i < len; i++)
        printf("0x%02x ", rxbuf[i]);
    printf("\n");
    free(rxbuf);
    close(sd);
    return 0;
}

int init_handshake()
{
    int rc;

    rc = add_local_addr();
    if (rc < 0) {
        printf("add_local_addr failed");
        return rc;
    }    
  
    rc = send_discoverynotify();
    if (rc < 0) {
        printf("send_discoverynotify failed");
        return rc;
    }  
}

// Callback for incoming MCTP messages
static int mctp_callback(sd_event_source *s, int fd, uint32_t revents, void *userdata)
{

    printf("mctp event receviced \n");
    //struct sockaddr_mctp src_addr;
    struct sockaddr_mctp_ext src_addr = {0};
    //socklen_t addr_len = sizeof(src_addr);
    socklen_t src_addrlen = sizeof(src_addr);
    char buffer[32];
    ssize_t len;
    char uuid[32] = {0x80,0x00,0x86,0x00,0x04,0x00,0x00,0x40,0x80,0x00,0x01,0x00,0x00,0x00,0x00,0x00};


    int ret = recvfrom(fd, NULL, 0, MSG_PEEK | MSG_TRUNC, NULL, 0);
    if (ret < 0)
    {
        printf("recvfrom msg peek failed \n");
        return -1;
    }
    len = (size_t)ret;

    uint8_t* rxbuf;
    rxbuf = malloc(len);
    if (!rxbuf)
    {
        printf("malloc failed \n");
        return -1;
    }


    // Receive the message along with the source address
    int recv_len = recvfrom(fd, rxbuf, len, 0, (struct sockaddr *)&src_addr, &src_addrlen);
    if (recv_len < 0 )
    {
        perror("recvfrom failed");
        return -1;
    }

    // Log the Source EID and message length
    printf("Received MCTP message from Src EID: %d -> Dst EID: %d, Length: %zd\n",
           src_addr.smctp_base.smctp_addr.s_addr, TARGET_EID, len);

    if (src_addrlen == sizeof(struct sockaddr_mctp_ext))
    {
        printf(" ext ifindex %d ha[0]=0x%2x len %hhu\n", src_addr.smctp_ifindex, src_addr.smctp_haddr[0], src_addr.smctp_halen);
        for (int j = 0; j < src_addr.smctp_halen; j++)
            printf("0x%02x:", src_addr.smctp_haddr[j]);
        printf("\n");
    }

    for (int i = 0; i < recv_len; i++)
    {
        printf("0x%02x ", rxbuf[i]);
    }
    printf("\n");


    if(rxbuf[CMD_INDEX] == MCTP_CMD_GET_EID )
    {
        //handle get eid message
        buffer[0] = rxbuf[0] & ~(0x80); //instance id
        buffer[1] = rxbuf[1];  //command code
        buffer[2] = 0x00; //completion code
        buffer[3] = 0x09; // eid
        buffer[4] = 0x10; // eid type
        buffer[5] = 0x00; // medium specific info
        len = 6;
    }

    if(rxbuf[CMD_INDEX] == MCTP_CMD_SET_EID )
    {
        //handle set eid message
        buffer[0] = rxbuf[0] & ~(0x80); //instance id
        buffer[1] = rxbuf[1];  //command code
        buffer[2] = 0x00; //completion code
        buffer[3] = 0x01; // requries pool allocation
        buffer[4] = rxbuf[3]; // eid assigned
        buffer[5] = 0x02; //eid pool size
        len = 6;
    }

    if(rxbuf[CMD_INDEX] == MCTP_CMD_ALLOC_POOL_ID )
    {
        //handle allocate eid pool message
        buffer[0] = rxbuf[0] & ~(0x80); //instance id
        buffer[1] = rxbuf[1];  //command code
        buffer[2] = 0x00; //completion code
        buffer[3] = 0x00; // allocation accepted
        buffer[4] = rxbuf[3]; // eid pool size
        buffer[5] = rxbuf[4]; // starting eid
        len = 6;
    }

    if(rxbuf[CMD_INDEX] == MCTP_GET_EP_UUID )
    {
        //handle get uuid
        buffer[0] = rxbuf[0] & ~(0x80); //instance id
        buffer[1] = rxbuf[1];  //command code
        buffer[2] = 0x00; //completion code
        strncpy(&buffer[3],&uuid[0], 16);
        len = 6;
    }   

    if(rxbuf[CMD_INDEX] == MCTP_GET_MSG_TYPE )
    {
        //handle get nesage type
        buffer[0] = rxbuf[0] & ~(0x80); //instance id
        buffer[1] = rxbuf[1];  //command code
        buffer[2] = 0x00; //completion code
        buffer[3] = 0x02; // no of types supproted
        buffer[4] = 0x05; // pldm type
        buffer[5] = 0x7E; // vdm type 
        len = 6;
    }


    /* for the tag used in the reply, we clear the tag-owner bit, but
    * keep the tag value */
    src_addr.smctp_base.smctp_tag &= ~MCTP_TAG_OWNER;

    /* return message to sender */
    ret = sendto(fd, buffer, len, 0, (struct sockaddr *)&src_addr, sizeof(src_addr));
    if (ret != len)
    {
        printf("sendto failed \n");
        return -1;
    }

    return 0;
}

int main() {
	
    int sockfd, rc;
    struct sockaddr_mctp addr = {0};
    sd_event *event = NULL;
    sd_event_source *event_source = NULL;

    rc = init_handshake();
    if (rc < 0) {
        printf("init_handshake failed");
        return 1;
    }  
    // Create MCTP socket
    sockfd = socket(AF_MCTP, SOCK_DGRAM, 0);
    if (sockfd < 0) {
        printf("Socket creation failed");
        return 1;
    }

    // Bind to MCTP_ADDR_ANY
    addr.smctp_family = AF_MCTP;
    addr.smctp_addr.s_addr = MCTP_ADDR_ANY;
    addr.smctp_type = MCTP_TYPE_CONTROL;

    if (bind(sockfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
        printf("Bind failed");
        close(sockfd);
        return 1;
    }

    // Create event loop
    if (sd_event_new(&event) < 0) {
        printf("sd_event_new failed");
        close(sockfd);
        return 1;
    }

    // Add socket to the event loop
    if (sd_event_add_io(event, &event_source, sockfd, EPOLLIN, mctp_callback, NULL) < 0) {
        printf("sd_event_add_io failed");
        sd_event_unref(event);
        close(sockfd);
        return 1;
    }

    // Run the event loop
    sd_event_loop(event);

    // Cleanup
    sd_event_unref(event);
    close(sockfd);

    return 0;
}

please let me know if i am missing anything here

[chandu-tmba]

Added some logs in the network layer:

[ 124.268756] mctp: mctp_pkttype_receive
[ 124.273042] mctp: mctp_pkttype_receive - Device type is ARPHRD_MCTP
[ 124.279663] mctp: mctp_pkttype_receive - dev->ifindex: 3, net: 1
[ 124.285991] mctp: mctp_pkttype_receive - mh->dest: 0x0, mh->src: 0x0, cb->net: 1
[ 124.293968] mctp: mctp_pkttype_receive - mctp_route_lookup returned NULL
[ 124.300685] mctp: mctp_pkttype_receive - !!RT (Route Error)

Somehow, the network layer is not forwarding the packets to the socket queue.

[jk-ozlabs]

Added some logs in the network layer:

[ 124.285991] mctp: mctp_pkttype_receive - mh->dest: 0x0, mh->src: 0x0, cb->net: 1
[ 124.293968] mctp: mctp_pkttype_receive - mctp_route_lookup returned NULL
[ 124.300685] mctp: mctp_pkttype_receive - !!RT (Route Error)

it's not obvious where you have added these logs in the routing path, but mctp_route_lookup() is expected to return NULL on physically-addressed packets, and we fall-back to mctp_route_lookup_null(), which attempts to find a route based on the incoming device.