About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / infiniband / user_mad.txt


Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	USERSPACE MAD ACCESS
2	
3	Device files
4	
5	  Each port of each InfiniBand device has a "umad" device and an
6	  "issm" device attached.  For example, a two-port HCA will have two
7	  umad devices and two issm devices, while a switch will have one
8	  device of each type (for switch port 0).
9	
10	Creating MAD agents
11	
12	  A MAD agent can be created by filling in a struct ib_user_mad_reg_req
13	  and then calling the IB_USER_MAD_REGISTER_AGENT ioctl on a file
14	  descriptor for the appropriate device file.  If the registration
15	  request succeeds, a 32-bit id will be returned in the structure.
16	  For example:
17	
18		struct ib_user_mad_reg_req req = { /* ... */ };
19		ret = ioctl(fd, IB_USER_MAD_REGISTER_AGENT, (char *) &req);
20	        if (!ret)
21			my_agent = req.id;
22		else
23			perror("agent register");
24	
25	  Agents can be unregistered with the IB_USER_MAD_UNREGISTER_AGENT
26	  ioctl.  Also, all agents registered through a file descriptor will
27	  be unregistered when the descriptor is closed.
28	
29	  2014 -- a new registration ioctl is now provided which allows additional
30	       fields to be provided during registration.
31	       Users of this registration call are implicitly setting the use of
32	       pkey_index (see below).
33	
34	Receiving MADs
35	
36	  MADs are received using read().  The receive side now supports
37	  RMPP. The buffer passed to read() must be at least one
38	  struct ib_user_mad + 256 bytes. For example:
39	
40	  If the buffer passed is not large enough to hold the received
41	  MAD (RMPP), the errno is set to ENOSPC and the length of the
42	  buffer needed is set in mad.length.
43	
44	  Example for normal MAD (non RMPP) reads:
45		struct ib_user_mad *mad;
46		mad = malloc(sizeof *mad + 256);
47		ret = read(fd, mad, sizeof *mad + 256);
48		if (ret != sizeof mad + 256) {
49			perror("read");
50			free(mad);
51		}
52	
53	  Example for RMPP reads:
54		struct ib_user_mad *mad;
55		mad = malloc(sizeof *mad + 256);
56		ret = read(fd, mad, sizeof *mad + 256);
57		if (ret == -ENOSPC)) {
58			length = mad.length;
59			free(mad);
60			mad = malloc(sizeof *mad + length);
61			ret = read(fd, mad, sizeof *mad + length);
62		}
63		if (ret < 0) {
64			perror("read");
65			free(mad);
66		}
67	
68	  In addition to the actual MAD contents, the other struct ib_user_mad
69	  fields will be filled in with information on the received MAD.  For
70	  example, the remote LID will be in mad.lid.
71	
72	  If a send times out, a receive will be generated with mad.status set
73	  to ETIMEDOUT.  Otherwise when a MAD has been successfully received,
74	  mad.status will be 0.
75	
76	  poll()/select() may be used to wait until a MAD can be read.
77	
78	Sending MADs
79	
80	  MADs are sent using write().  The agent ID for sending should be
81	  filled into the id field of the MAD, the destination LID should be
82	  filled into the lid field, and so on.  The send side does support
83	  RMPP so arbitrary length MAD can be sent. For example:
84	
85		struct ib_user_mad *mad;
86	
87		mad = malloc(sizeof *mad + mad_length);
88	
89		/* fill in mad->data */
90	
91		mad->hdr.id  = my_agent;	/* req.id from agent registration */
92		mad->hdr.lid = my_dest;		/* in network byte order... */
93		/* etc. */
94	
95		ret = write(fd, &mad, sizeof *mad + mad_length);
96		if (ret != sizeof *mad + mad_length)
97			perror("write");
98	
99	Transaction IDs
100	
101	  Users of the umad devices can use the lower 32 bits of the
102	  transaction ID field (that is, the least significant half of the
103	  field in network byte order) in MADs being sent to match
104	  request/response pairs.  The upper 32 bits are reserved for use by
105	  the kernel and will be overwritten before a MAD is sent.
106	
107	P_Key Index Handling
108	
109	  The old ib_umad interface did not allow setting the P_Key index for
110	  MADs that are sent and did not provide a way for obtaining the P_Key
111	  index of received MADs.  A new layout for struct ib_user_mad_hdr
112	  with a pkey_index member has been defined; however, to preserve binary
113	  compatibility with older applications, this new layout will not be used
114	  unless one of IB_USER_MAD_ENABLE_PKEY or IB_USER_MAD_REGISTER_AGENT2 ioctl's
115	  are called before a file descriptor is used for anything else.
116	
117	  In September 2008, the IB_USER_MAD_ABI_VERSION will be incremented
118	  to 6, the new layout of struct ib_user_mad_hdr will be used by
119	  default, and the IB_USER_MAD_ENABLE_PKEY ioctl will be removed.
120	
121	Setting IsSM Capability Bit
122	
123	  To set the IsSM capability bit for a port, simply open the
124	  corresponding issm device file.  If the IsSM bit is already set,
125	  then the open call will block until the bit is cleared (or return
126	  immediately with errno set to EAGAIN if the O_NONBLOCK flag is
127	  passed to open()).  The IsSM bit will be cleared when the issm file
128	  is closed.  No read, write or other operations can be performed on
129	  the issm file.
130	
131	/dev files
132	
133	  To create the appropriate character device files automatically with
134	  udev, a rule like
135	
136	    KERNEL=="umad*", NAME="infiniband/%k"
137	    KERNEL=="issm*", NAME="infiniband/%k"
138	
139	  can be used.  This will create device nodes named
140	
141	    /dev/infiniband/umad0
142	    /dev/infiniband/issm0
143	
144	  for the first port, and so on.  The InfiniBand device and port
145	  associated with these devices can be determined from the files
146	
147	    /sys/class/infiniband_mad/umad0/ibdev
148	    /sys/class/infiniband_mad/umad0/port
149	
150	  and
151	
152	    /sys/class/infiniband_mad/issm0/ibdev
153	    /sys/class/infiniband_mad/issm0/port
Hide Line Numbers


About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog