Where does the device write to the PCI MSI-X message data?

After taking too much time reading FreeBSD source code, and hacking some bhyve, to try to understand how FreeBSD handles MSI-X interrupts I am a lot confused, and not sure what happens:

Firstly, I understand more or less than the relevant source file directories are /usr/src/sys/sys, /usr/src/sys/kern, /usr/src/dev/pci, additionaly /usr/src/sys/amd64 and /usr/src/sys/x86 because I am working in amd64, and finally /usr/src/sys/dev/virtio because is where the travel starts.

The path to bus_setup_intr​

  1. Go to vt9p_attach at /usr/src/sys/dev/virtio/p9fs/virtio_p9fs.c, here the device attachment starts*, after several lines with several degrees of importance you get a call to virtio_setup_intr,
  2. whilh located at /usr/src/sys/dev/virtio/virtio.c calls, given the context, vtpci_legacy_setup_interrupts at /usr/src/sys/dev/virtio/pci/virtio_pci_legacy.c, which calls vtpci_setup_interrupts at /usr/src/sys/dev/virtio/pci/virtio_pci.c.
  3. Now vtpci_setup_interrupts, as the context is configured to use MSI-X which was checked with bhyve printf debuging, will first call to vtpci_alloc_intr_msix_pervq at ibidem, which after some variable setting vtpci_alloc_msix at ibidem, it calls pci_msix_count at /usr/src/sys/dev/pci/pci.c, which is no setting registers, then calls
  4. pci_alloc_msix also at /usr/src/sys/dev/pci/pci.c, which at the end writes the MSI-X enable bit in the relevant PCI capability, which by bhyve printf debuing goes before setting the PCI MSI-X Table Entry contents.
  5. Thus going back to vtpci_setup_interrupts at /usr/src/sys/dev/virtio/pci/virtio_pci.c, it will now call to vtpci_setup_intrs at ibidem,
  6. now going backwards, because fordwards is even more confusing, it finalizes with a call to vtpci_setup_msix_interrupts at ibidem,
  7. which, continuing backwards, finalizes with a call to vtpci_set_host_msix_vectors, which configurates the virtio contiguration vector and virtio address vector, which by bhyve prinf go after the MSI-X configuration, this the call is before this.
  8. Continuing backwards at vtpci_setup_msix_interrupts it calls vtpci_setup_pervq_msix_interrupts at ibidem,
  9. now this function as well as the rest of vtpci_set_host_msix_vectors are basically filling arguments to call to bus_setup_intr.

Trying to fnid bus_setup_intr​

Now I run grep -n bus_setup_intr -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, which basically searches for it in the relevant directories as I understand, gets me:
Code:
/usr/src/sys/sys/bus.h:256: * These flags may be passed by drivers to bus_setup_intr(9) when
/usr/src/sys/sys/bus.h:564:int  bus_setup_intr(device_t dev, struct resource *r, int flags,
/usr/src/sys/kern/subr_bus.c:4816:bus_setup_intr(device_t dev, struct resource *r, int flags,
/usr/src/sys/kern/subr_bus.c:5196:      KOBJMETHOD(bus_setup_intr,      root_setup_intr),
/usr/src/sys/kern/subr_intr.c:1302:              * postpone calling bus_setup_intr on I/O IRQ(s) until
/usr/src/sys/dev/pci/pci_host_generic.c:680:    DEVMETHOD(bus_setup_intr,               bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_dw_mv.c:301:   if (bus_setup_intr(dev, sc->irq_res, INTR_TYPE_MISC | INTR_MPSAFE,
/usr/src/sys/dev/pci/hostb_pci.c:254:   DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_private.h:59:bus_setup_intr_t  pci_setup_intr;
/usr/src/sys/dev/pci/pci_pci.c:115:    DEVMETHOD(bus_setup_intr,                bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_pci.c:1261:    error = bus_setup_intr(dev, sc->pcie_irq, INTR_TYPE_MISC|INTR_MPSAFE,
/usr/src/sys/dev/pci/vga_pci.c:717:     DEVMETHOD(bus_setup_intr,       vga_pci_setup_intr),
/usr/src/sys/dev/pci/isa_pci.c:75:    DEVMETHOD(bus_setup_intr,         bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci.c:155: DEVMETHOD(bus_setup_intr,       pci_setup_intr),
/usr/src/sys/amd64/vmm/io/ppt.c:694:            error = bus_setup_intr(ppt->dev, ppt->msi.res[i],
/usr/src/sys/amd64/vmm/io/ppt.c:822:            error = bus_setup_intr(ppt->dev, ppt->msix.res[idx],
/usr/src/sys/amd64/vmm/amd/amdviiommu.c:143:    error = bus_setup_intr(dev, sc->event_res, INTR_TYPE_MISC | INTR_MPSAFE,
/usr/src/sys/x86/x86/legacy.c:90:       DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/x86/legacy.c:303:      DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/x86/nexus.c:110:static bus_setup_intr_t                nexus_setup_intr;
/usr/src/sys/x86/x86/nexus.c:154:       DEVMETHOD(bus_setup_intr,       nexus_setup_intr),
/usr/src/sys/x86/x86/mptable_pci.c:165: DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/isa/atrtc.c:500:               } else if ((bus_setup_intr(dev, sc->intr_res, INTR_TYPE_CLK,
/usr/src/sys/x86/isa/clock.c:598:               /* Dirty hack, to make bus_setup_intr to not enable source. */
/usr/src/sys/x86/isa/clock.c:600:               if ((bus_setup_intr(dev, sc->intr_res,
/usr/src/sys/x86/pci/pci_bus.c:660:     DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/pci/qpi.c:210: DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/pci/qpi.c:288: DEVMETHOD(bus_setup_intr,       bus_generic_setup_intr),
/usr/src/sys/x86/iommu/amd_drv.c:388:   error = bus_setup_intr(sc->iommu.dev, sc->irq_cmdev,
/usr/src/sys/x86/iommu/iommu_utils.c:599:       error = bus_setup_intr(dev, dmd->irq_res, INTR_TYPE_MISC,
Which goes like:
  1. At /usr/src/sys/sys/bus.h there is only a regular header for bus_setup_intr at /usr/src/sys/kern/subr_bus.c:4816, basically the next result.
  2. At /usr/src/sys/kern/subr_bus.c:4816 there is a bus_setup_intr implementation, which basically calls the BUS_SETUP_INTR at the parent device.
  3. At /usr/src/sys/kern/subr_bus.c:5196 setups the BUS_SETUP_INTR for root something, which is basically a panic.
  4. At /usr/src/sys/kern/subr_bus.c, there is also bus_generic_setup_intr, which is also a call to the parent's BUS_SETUP_INTR.
  5. At /usr/src/sys/dev/pci/pci_host_generic.c:680, /usr/src/sys/dev/pci/hostb_pci.c:254, /usr/src/sys/dev/pci/pci_pci.c:115, /usr/src/sys/dev/pci/isa_pci.c:75, /usr/src/sys/x86/x86/legacy.c:90, /usr/src/sys/x86/x86/legacy.c:303, /usr/src/sys/x86/x86/mptable_pci.c:165, /usr/src/sys/x86/pci/pci_bus.c:660, /usr/src/sys/x86/pci/qpi.c:210, and /usr/src/sys/x86/pci/qpi.c:288 do set calll to the parent's BUS_SETUP_INTR, by bus_generic_setup_intr.
  6. /usr/src/sys/dev/pci/vga_pci.c:717 should not be relevant as it sets vga_pci_setup_intr.
  7. Which gives only left /usr/src/sys/x86/x86/nexus.c, which setups nexus_setup_intr, and give some incognitas of how legacy is supposed do work if it will just inherint until reaches a panic.
Now at nexus_setup_intr at /usr/src/sys/x86/x86/nexus.c, it runs rman_activate_resource(9), now it calls intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.

This looks like an dead end expect for the vtable calls to pic_enable_intr and pic_enable_src. Now very clear how they are set.

vtpci_setup_intrs, now forwards​

This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.
Now in base to that I search with grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672:    DEVMETHOD(bus_alloc_resource,           pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257:   DEVMETHOD(bus_alloc_resource,   pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108:    DEVMETHOD(bus_alloc_resource,    pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719:     DEVMETHOD(bus_alloc_resource,   vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71:    DEVMETHOD(bus_alloc_resource,     isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource,   pci_alloc_resource),
which gets me
  1. pci_host_generic_core_alloc_resource calls, at MSI-X relevanet bus_generic_rman_alloc_resource, which implemented at /usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once again bus_alloc_resource, for what looks like the child.
  2. pci_hostb_alloc_resource looks like a infinite loop, calling itself irrelevanlty.
  3. pcib_alloc_resource looks like either calling to the parent or pcib_alloc_subbus which either calls itself or pcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls when SYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant case SYS_RES_MEMORY and the rest of the function the only new kind of calls are for pcib_grow_window and bus_generic_alloc_resource, in the case of the former there is only calls to pcib_alloc_nec_window and pcib_expand_window, they are probably not relevant.
  4. pcib_alloc_resource looks like calling itself and then calling to the parent.
  5. pci_alloc_resource looks like either caling to the parent or more usefully caling pci_alloc_multi_resource, which calls bus_generic_alloc_resource or resource_list_alloc, if not calling pci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, or pci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
The most common of the function that are left to check is bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.
Then there is resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.

But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9) manual page" href="https://man.freebsd.org/cgi/man.cgi?query=rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.

This looks like an dead end expect for the vtable calls to pic_enable_intr and pic_enable_src. Now very clear how they are set.

vtpci_setup_intrs, now forwards​

This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.
Now in base to that I search with grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672:    DEVMETHOD(bus_alloc_resource,           pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257:   DEVMETHOD(bus_alloc_resource,   pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108:    DEVMETHOD(bus_alloc_resource,    pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719:     DEVMETHOD(bus_alloc_resource,   vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71:    DEVMETHOD(bus_alloc_resource,     isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource,   pci_alloc_resource),
which gets me
  1. pci_host_generic_core_alloc_resource calls, at MSI-X relevanet bus_generic_rman_alloc_resource, which implemented at /usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once again bus_alloc_resource, for what looks like the child.
  2. pci_hostb_alloc_resource looks like a infinite loop, calling itself irrelevanlty.
  3. pcib_alloc_resource looks like either calling to the parent or pcib_alloc_subbus which either calls itself or pcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls when SYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant case SYS_RES_MEMORY and the rest of the function the only new kind of calls are for pcib_grow_window and bus_generic_alloc_resource, in the case of the former there is only calls to pcib_alloc_nec_window and pcib_expand_window, they are probably not relevant.
  4. pcib_alloc_resource looks like calling itself and then calling to the parent.
  5. pci_alloc_resource looks like either caling to the parent or more usefully caling pci_alloc_multi_resource, which calls bus_generic_alloc_resource or resource_list_alloc, if not calling pci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, or pci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
The most common of the function that are left to check is bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.
Then there is resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.

But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.&sektion=9&manpath=freebsd-release-ports">rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.

This looks like an dead end expect for the vtable calls to pic_enable_intr and pic_enable_src. Now very clear how they are set.

vtpci_setup_intrs, now forwards​

This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.
Now in base to that I search with grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672:    DEVMETHOD(bus_alloc_resource,           pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257:   DEVMETHOD(bus_alloc_resource,   pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108:    DEVMETHOD(bus_alloc_resource,    pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719:     DEVMETHOD(bus_alloc_resource,   vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71:    DEVMETHOD(bus_alloc_resource,     isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource,   pci_alloc_resource),
which gets me
  1. pci_host_generic_core_alloc_resource calls, at MSI-X relevanet bus_generic_rman_alloc_resource, which implemented at /usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once again bus_alloc_resource, for what looks like the child.
  2. pci_hostb_alloc_resource looks like a infinite loop, calling itself irrelevanlty.
  3. pcib_alloc_resource looks like either calling to the parent or pcib_alloc_subbus which either calls itself or pcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls when SYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant case SYS_RES_MEMORY and the rest of the function the only new kind of calls are for pcib_grow_window and bus_generic_alloc_resource, in the case of the former there is only calls to pcib_alloc_nec_window and pcib_expand_window, they are probably not relevant.
  4. pcib_alloc_resource looks like calling itself and then calling to the parent.
  5. pci_alloc_resource looks like either caling to the parent or more usefully caling pci_alloc_multi_resource, which calls bus_generic_alloc_resource or resource_list_alloc, if not calling pci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, or pci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
The most common of the function that are left to check is bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.
Then there is resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.

But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9)
 
Back
Top