After taking too much time reading FreeBSD source code, and hacking some bhyve, to try to understand how FreeBSD handles MSI-X interrupts I am a lot confused, and not sure what happens:
Firstly, I understand more or less than the relevant source file directories are
The path to
Trying to fnid
Now I run
Which goes like:
This looks like an dead end expect for the vtable calls to
This function did also called
Now in base to that I search with
which gets me
Then there is
But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9) manual page" href="https://man.freebsd.org/cgi/man.cgi?query=rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls
This looks like an dead end expect for the vtable calls to
This function did also called
Now in base to that I search with
which gets me
Then there is
But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.&sektion=9&manpath=freebsd-release-ports">rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls
This looks like an dead end expect for the vtable calls to
This function did also called
Now in base to that I search with
which gets me
Then there is
But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9)
Firstly, I understand more or less than the relevant source file directories are
/usr/src/sys/sys, /usr/src/sys/kern, /usr/src/dev/pci, additionaly /usr/src/sys/amd64 and /usr/src/sys/x86 because I am working in amd64, and finally /usr/src/sys/dev/virtio because is where the travel starts.The path to bus_setup_intr
- Go to
vt9p_attachat/usr/src/sys/dev/virtio/p9fs/virtio_p9fs.c, here the device attachment starts*, after several lines with several degrees of importance you get a call tovirtio_setup_intr, - whilh located at
/usr/src/sys/dev/virtio/virtio.ccalls, given the context,vtpci_legacy_setup_interruptsat/usr/src/sys/dev/virtio/pci/virtio_pci_legacy.c, which callsvtpci_setup_interruptsat/usr/src/sys/dev/virtio/pci/virtio_pci.c. - Now
vtpci_setup_interrupts, as the context is configured to use MSI-X which was checked with bhyve printf debuging, will first call tovtpci_alloc_intr_msix_pervqat ibidem, which after some variable settingvtpci_alloc_msixat ibidem, it callspci_msix_countat/usr/src/sys/dev/pci/pci.c, which is no setting registers, then calls pci_alloc_msixalso at/usr/src/sys/dev/pci/pci.c, which at the end writes the MSI-X enable bit in the relevant PCI capability, which by bhyve printf debuing goes before setting the PCI MSI-X Table Entry contents.- Thus going back to
vtpci_setup_interruptsat/usr/src/sys/dev/virtio/pci/virtio_pci.c, it will now call tovtpci_setup_intrsat ibidem, - now going backwards, because fordwards is even more confusing, it finalizes with a call to
vtpci_setup_msix_interruptsat ibidem, - which, continuing backwards, finalizes with a call to
vtpci_set_host_msix_vectors, which configurates the virtio contiguration vector and virtio address vector, which by bhyve prinf go after the MSI-X configuration, this the call is before this. - Continuing backwards at
vtpci_setup_msix_interruptsit callsvtpci_setup_pervq_msix_interruptsat ibidem, - now this function as well as the rest of
vtpci_set_host_msix_vectorsare basically filling arguments to call tobus_setup_intr.
Trying to fnid bus_setup_intr
Now I run grep -n bus_setup_intr -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, which basically searches for it in the relevant directories as I understand, gets me:
Code:
/usr/src/sys/sys/bus.h:256: * These flags may be passed by drivers to bus_setup_intr(9) when
/usr/src/sys/sys/bus.h:564:int bus_setup_intr(device_t dev, struct resource *r, int flags,
/usr/src/sys/kern/subr_bus.c:4816:bus_setup_intr(device_t dev, struct resource *r, int flags,
/usr/src/sys/kern/subr_bus.c:5196: KOBJMETHOD(bus_setup_intr, root_setup_intr),
/usr/src/sys/kern/subr_intr.c:1302: * postpone calling bus_setup_intr on I/O IRQ(s) until
/usr/src/sys/dev/pci/pci_host_generic.c:680: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_dw_mv.c:301: if (bus_setup_intr(dev, sc->irq_res, INTR_TYPE_MISC | INTR_MPSAFE,
/usr/src/sys/dev/pci/hostb_pci.c:254: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_private.h:59:bus_setup_intr_t pci_setup_intr;
/usr/src/sys/dev/pci/pci_pci.c:115: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci_pci.c:1261: error = bus_setup_intr(dev, sc->pcie_irq, INTR_TYPE_MISC|INTR_MPSAFE,
/usr/src/sys/dev/pci/vga_pci.c:717: DEVMETHOD(bus_setup_intr, vga_pci_setup_intr),
/usr/src/sys/dev/pci/isa_pci.c:75: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/dev/pci/pci.c:155: DEVMETHOD(bus_setup_intr, pci_setup_intr),
/usr/src/sys/amd64/vmm/io/ppt.c:694: error = bus_setup_intr(ppt->dev, ppt->msi.res[i],
/usr/src/sys/amd64/vmm/io/ppt.c:822: error = bus_setup_intr(ppt->dev, ppt->msix.res[idx],
/usr/src/sys/amd64/vmm/amd/amdviiommu.c:143: error = bus_setup_intr(dev, sc->event_res, INTR_TYPE_MISC | INTR_MPSAFE,
/usr/src/sys/x86/x86/legacy.c:90: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/x86/legacy.c:303: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/x86/nexus.c:110:static bus_setup_intr_t nexus_setup_intr;
/usr/src/sys/x86/x86/nexus.c:154: DEVMETHOD(bus_setup_intr, nexus_setup_intr),
/usr/src/sys/x86/x86/mptable_pci.c:165: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/isa/atrtc.c:500: } else if ((bus_setup_intr(dev, sc->intr_res, INTR_TYPE_CLK,
/usr/src/sys/x86/isa/clock.c:598: /* Dirty hack, to make bus_setup_intr to not enable source. */
/usr/src/sys/x86/isa/clock.c:600: if ((bus_setup_intr(dev, sc->intr_res,
/usr/src/sys/x86/pci/pci_bus.c:660: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/pci/qpi.c:210: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/pci/qpi.c:288: DEVMETHOD(bus_setup_intr, bus_generic_setup_intr),
/usr/src/sys/x86/iommu/amd_drv.c:388: error = bus_setup_intr(sc->iommu.dev, sc->irq_cmdev,
/usr/src/sys/x86/iommu/iommu_utils.c:599: error = bus_setup_intr(dev, dmd->irq_res, INTR_TYPE_MISC,
- At
/usr/src/sys/sys/bus.hthere is only a regular header forbus_setup_intrat/usr/src/sys/kern/subr_bus.c:4816, basically the next result. - At
/usr/src/sys/kern/subr_bus.c:4816there is abus_setup_intrimplementation, which basically calls theBUS_SETUP_INTRat the parent device. - At
/usr/src/sys/kern/subr_bus.c:5196setups theBUS_SETUP_INTRfor root something, which is basically a panic. - At
/usr/src/sys/kern/subr_bus.c, there is alsobus_generic_setup_intr, which is also a call to the parent'sBUS_SETUP_INTR. - At
/usr/src/sys/dev/pci/pci_host_generic.c:680,/usr/src/sys/dev/pci/hostb_pci.c:254,/usr/src/sys/dev/pci/pci_pci.c:115,/usr/src/sys/dev/pci/isa_pci.c:75,/usr/src/sys/x86/x86/legacy.c:90,/usr/src/sys/x86/x86/legacy.c:303,/usr/src/sys/x86/x86/mptable_pci.c:165,/usr/src/sys/x86/pci/pci_bus.c:660,/usr/src/sys/x86/pci/qpi.c:210, and/usr/src/sys/x86/pci/qpi.c:288do set calll to the parent'sBUS_SETUP_INTR, bybus_generic_setup_intr. /usr/src/sys/dev/pci/vga_pci.c:717should not be relevant as it setsvga_pci_setup_intr.- Which gives only left
/usr/src/sys/x86/x86/nexus.c, which setupsnexus_setup_intr, and give some incognitas of how legacy is supposed do work if it will just inherint until reaches a panic.
nexus_setup_intr at /usr/src/sys/x86/x86/nexus.c, it runs rman_activate_resource(9), now it calls intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.This looks like an dead end expect for the vtable calls to
pic_enable_intr and pic_enable_src. Now very clear how they are set. vtpci_setup_intrs, now forwards
This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.Now in base to that I search with
grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672: DEVMETHOD(bus_alloc_resource, pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257: DEVMETHOD(bus_alloc_resource, pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108: DEVMETHOD(bus_alloc_resource, pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719: DEVMETHOD(bus_alloc_resource, vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71: DEVMETHOD(bus_alloc_resource, isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource, pci_alloc_resource),
pci_host_generic_core_alloc_resourcecalls, at MSI-X relevanetbus_generic_rman_alloc_resource, which implemented at/usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once againbus_alloc_resource, for what looks like the child.pci_hostb_alloc_resourcelooks like a infinite loop, calling itself irrelevanlty.pcib_alloc_resourcelooks like either calling to the parent orpcib_alloc_subbuswhich either calls itself orpcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls whenSYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant caseSYS_RES_MEMORYand the rest of the function the only new kind of calls are forpcib_grow_windowandbus_generic_alloc_resource, in the case of the former there is only calls topcib_alloc_nec_windowandpcib_expand_window, they are probably not relevant.pcib_alloc_resourcelooks like calling itself and then calling to the parent.pci_alloc_resourcelooks like either caling to the parent or more usefully calingpci_alloc_multi_resource, which callsbus_generic_alloc_resourceorresource_list_alloc, if not callingpci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, orpci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.Then there is
resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9) manual page" href="https://man.freebsd.org/cgi/man.cgi?query=rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls
intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.This looks like an dead end expect for the vtable calls to
pic_enable_intr and pic_enable_src. Now very clear how they are set. vtpci_setup_intrs, now forwards
This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.Now in base to that I search with
grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672: DEVMETHOD(bus_alloc_resource, pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257: DEVMETHOD(bus_alloc_resource, pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108: DEVMETHOD(bus_alloc_resource, pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719: DEVMETHOD(bus_alloc_resource, vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71: DEVMETHOD(bus_alloc_resource, isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource, pci_alloc_resource),
pci_host_generic_core_alloc_resourcecalls, at MSI-X relevanetbus_generic_rman_alloc_resource, which implemented at/usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once againbus_alloc_resource, for what looks like the child.pci_hostb_alloc_resourcelooks like a infinite loop, calling itself irrelevanlty.pcib_alloc_resourcelooks like either calling to the parent orpcib_alloc_subbuswhich either calls itself orpcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls whenSYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant caseSYS_RES_MEMORYand the rest of the function the only new kind of calls are forpcib_grow_windowandbus_generic_alloc_resource, in the case of the former there is only calls topcib_alloc_nec_windowandpcib_expand_window, they are probably not relevant.pcib_alloc_resourcelooks like calling itself and then calling to the parent.pci_alloc_resourcelooks like either caling to the parent or more usefully calingpci_alloc_multi_resource, which callsbus_generic_alloc_resourceorresource_list_alloc, if not callingpci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, orpci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.Then there is
resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.&sektion=9&manpath=freebsd-release-ports">rman_get_flgas[/cmd], which by the looks of it, does not send writes to the MSI-X table, likewise with rman_activate_resource(9), now it calls
intr_lookup_source, that by the means of grep -n intr_lookup_source -R /usr/src/sys/sys /usr/src/sys/kern /usr/src/sys/dev/pci /usr/src/sys/amd64 /usr/src/sys/x86, looks like it is at /usr/src/sys/x86/x86/intr_machdep.c:249 and only checks an variable, so continuing at nexus_setup_intr, it calls intr_add_handler at ibidem, which firstly calls intr_event_add_handler at /usr/src/sys/kern/kern_intr.c, which also does not look it sends any kind of MSI-X Table write, following of an sx_xlock(9) locked call to intrcnt_uptname, which basically sets a variable. The otherwise no very relevant intr_add_handler, has some vtable calls to pic_enable_intr and pic_enable_src. Finally nexus_setup_intr finalizes with a call to the undocumented rman_set_irq_cooklie, which in principle should not make MSI-X table writes.This looks like an dead end expect for the vtable calls to
pic_enable_intr and pic_enable_src. Now very clear how they are set. vtpci_setup_intrs, now forwards
This function did also called vtpci_alloc_intr_resources at ibidem, which the only relevant call, that it has, is vtpci_alloc_interrupt, which calls bus_alloc_resource_any, which by it's implementation at /usr/src/sys/sys/bus.h only calls to bus_alloc_resource, which I could only assume is implemented at /usr/src/sys/kern/bus_if.m, but I am not sure.Now in base to that I search with
grep -n "bus_alloc_resource" -R /usr/src/sys/dev/pci/ | grep DEVMETHOD getting
Code:
/usr/src/sys/dev/pci/pci_host_generic.c:672: DEVMETHOD(bus_alloc_resource, pci_host_generic_core_alloc_resource),
/usr/src/sys/dev/pci/hostb_pci.c:257: DEVMETHOD(bus_alloc_resource, pci_hostb_alloc_resource),
/usr/src/sys/dev/pci/pci_pci.c:108: DEVMETHOD(bus_alloc_resource, pcib_alloc_resource),
/usr/src/sys/dev/pci/vga_pci.c:719: DEVMETHOD(bus_alloc_resource, vga_pci_alloc_resource),
/usr/src/sys/dev/pci/isa_pci.c:71: DEVMETHOD(bus_alloc_resource, isab_pci_alloc_resource),
/usr/src/sys/dev/pci/pci.c:166: DEVMETHOD(bus_alloc_resource, pci_alloc_resource),
pci_host_generic_core_alloc_resourcecalls, at MSI-X relevanetbus_generic_rman_alloc_resource, which implemented at/usr/src/sys/kern/subr_bus.c:4351, also looks like a dead end except for that callcs once againbus_alloc_resource, for what looks like the child.pci_hostb_alloc_resourcelooks like a infinite loop, calling itself irrelevanlty.pcib_alloc_resourcelooks like either calling to the parent orpcib_alloc_subbuswhich either calls itself orpcib_groc_subsub[cmd] which only writes to pci config space, thus should not be related to MSI-X table writes, nor the call to [cmd]bus_adjust_resources, which should be see later, but probably does not write becasue this is code for a bus and not a device?, there is some calls whenSYS_RES_IOPORT, but should not be relevant as MSI-X tables are in memory, in the (possibly) relevant caseSYS_RES_MEMORYand the rest of the function the only new kind of calls are forpcib_grow_windowandbus_generic_alloc_resource, in the case of the former there is only calls topcib_alloc_nec_windowandpcib_expand_window, they are probably not relevant.pcib_alloc_resourcelooks like calling itself and then calling to the parent.pci_alloc_resourcelooks like either caling to the parent or more usefully calingpci_alloc_multi_resource, which callsbus_generic_alloc_resourceorresource_list_alloc, if not callingpci_reserve_map, which does not looks like as do not write, and when write the comments mention about relocation of bars, orpci_assign_interrupt, which also does not look like working for write to PCI-X Tables.
bus_generic_alloc_resource, which is likely implemented at /usr/src/sys/kern/subr_bus.c, which is basically a call to the same it the parent.Then there is
resource_list_alloc, which probably has only one relevant call to BUS_ALLOC_RESOURCE, which also there is to check, but probably is not relevant.But there is a dead end, and looks of this to check.
There is two things to see about this, I may need some help, and there is a reason why the NASA's power of 10 rules do not like to use macros.(9)