Apparent discrepancy between bhyve MSIX_TABLE_ENTRY_SIZE and PCI Local Bus Specification Revision 3.0 August 12, 2002

F

Ferocious1881

The code for the bhyve command at /usr/src/usr.sbin/bhyve/pci_emul.h line 116 checkout release/15.1.0, defines MSIX_TABLE_ENTRY_SIZE as 16. Which by pci_emul_msix_tread() and pci_emul_msix_twrite() boath at /usr/src/usr.sbin/bhyve/pci_emul.c appears to be clearly the MSI-X table entry size, at the same time the PCI Local Bus Specification Revision 3.0 August 12, 2002 at section 6.8.2. MSI-X Capability & Table Structures Figure 6-13: MSI-X Table Structure gives a diagram which basically says that from Base there are an entry in every Base+n*8 adress for N entries given by the capability structure message control, where the entry has two parts Message Address and Message Data, which read from the Figure 6-13 are half part of the entry and confirmed later in 6.8.2.6. Message Address for MSI-X Table Entries and 6.8.2.7. Message Data for MSI-X Table Entries sections of the PCI Local Bus Specification Revision 3.0 August 12, 2002. Unfortunely there is no much prose about this.

Futhermore, as by 6.8.3.2. MSI-X Configuration there is the prose:
Software calculates the base address of the MSI-X Table by reading the 32-bit value from
the Table Offset / BIR register, masking off the lower 3 BIR bits, and adding the remaining
8-byte aligned 32-bit offset to the address taken from the Base Address register indicated by
the BIR.
Click to expand...
which suggest an code of:
Code: 
uint_64 *table_entry_msg_addrs = malloc(msix_cap->table_size * 8);
memset(table_enrty_msg_addrs, 0, msix_cap->table_size * 8);
uint_64 *table_entry_msg_datas = malloc(msix_cap->table_size * 8);
memset(table_enrty_msg_datas, 0, msix_cap->table_size * 8);

uint_32 bar_addr = (current_pci_bars[msix_cap->bir] & 0xFFFF_FFF0);
for (int j = 0; j <= msix_cap->table_size; j++) {
    uint_32 table_offset = msix_cap->table_offset & 0xFFFF_FFF8;
    volatile uint_32 *y = bar_addr + table_offset + 8*j;  
    volatile uint_32 *z = bar_addr + table_offset + 8*j + 4;
    table_entry_msg_addrs[j] = *y;
    table_entry_msg_datas[j] = *z;
}
Should fill table_entry_msg_addrs and table_entry_msg_datas with the respective part of the MSI-X Table Entries, I found that after making a little UEFI loader for test it, I found that, for an virtio-9p virtual device, the index 0 of table_entry_msg_addrs and table_entry_msg_datas is full of zeros and the index 1 of table_entry_msg_addrs and table_entry_msg_datas is full of ones, futhermore when I add some debuging printf for pci_emul_msix_tread at usr/src/usr.sbin/bhyve/pci_emul.c, I got that the reading of table entries are operating at offset: 4096 msi_entry_offset: 0 tab_index: 256 size: 4 for table_entry_msg_addrs[0], offset: 4096 msi_entry_offset: 4 tab_index: 256 size: 4 for table_entry_msg_datas[0], offset: 4104 msi_entry_offset: 8 tab_index: 256 size: 4 for table_entry_msg_addrs[1], and offset: 4108 msi_entry_offset: 12 tab_index: 256 size: 4 for table_entry_msg_datas[1]. (Note BIR for MSI-X is 32-bit memory for virtio-9p).

If I instead do the following:
Code: 
uint_64 *table_entry_msg_addrs = malloc(msix_cap->table_size * 8);
memset(table_enrty_msg_addrs, 0, msix_cap->table_size * 8);
uint_64 *table_entry_msg_datas = malloc(msix_cap->table_size * 8);
memset(table_enrty_msg_datas, 0, msix_cap->table_size * 8);

uint_32 bar_addr = (current_pci_bars[msix_cap->bir] & 0xFFFF_FFF0);
for (int j = 0; j <= msix_cap->table_size; j++) {
    volatile uint_32 *y = bar_addr  + 8*j;  
    volatile uint_32 *z = bar_addr  + 8*j + 4;
    table_entry_msg_addrs[j] = *y;
    table_entry_msg_datas[j] = *z;
}
I got a more sane all zeros for the index 0 of table_entry_msg_addrs and table_entry_msg_datas followed of almos the same for index 1, except that for table_entry_msg_addrs[1] I got an 1, which is supposedly a pending bit so not much better, even so the printf debug says I all of those reads where against tab_index: 0, trying the same for the next 64 bits gives the same pattern, but now with table_index: 1 in printf debug.

All of those readings are even so more nonsensical afer reading
Software may leave unused MSI-X Table entries in their
default state of being masked.
Click to expand...
from 6.8.3.2. MSI-X Configuration, and having none of the corresponding mask bits set to zero.
I also confirmed taht retval is congruent with the UEFI loader readings, and all of this is after enabling the MSI-X enable bit.
Noted all the address fields for the vectors are read and write fields, so one may have to set them, but it does not explain the missign mask bits, the apparently incorrect offset and the incogruent table_indexes.

I also note that does not appear to be some alternate MSI-X Table Entry Size in the PCI Local Bus Specification Revision 3.0 August 12, 2002, not a suggested manner to possibly differentiate if there were to be added later.

What I am missing?
 
You must log in or register to reply here.
Back
Top