From the locking subsystem point of view: if you are specifying O_NONBLOCK to flopen, it shall never deadlock from the locking point of view; it shall either succeed and lock the file, or return with the error indicator set. So from purely the locking point of view, the behavior you are seeing is a bug.
From the file system point of view: As far as I can remember, it should not ever be possible to hang an open call. There are atomic file system operations that need to temporarily block open (for example atomic renames), but those should finish in finite time. The only way to deadlock a file system does not apply to FreeBSD UFS an ZFS as far as I know, so again, this should not happen, except if there is a bug.
Could the cause be an IO error? For example, it is possible that the "system" (everything underneath your flopen call) had to start an IO to the disk to perform the operation (for example to the file system metadata), and that IO is simply not finishing, due to a problem in the IO stack (for example a firmware bug in your disk controller). If that's the case, you should find processes stuck in funny states (in Linux those are called D wait, I don't know how to diagnose this on FreeBSD, since it has never happened to me on FreeBSD), and you should be seeing error messages in system logs. On some Unix variants, there are watchdog timers that will abort stalled IOs after some period (often 90 or 300 seconds), so you might have to wait a few minutes. On other Unix variants, hardware-stalled IOs can take infinitely long (I once saw an example of an IO that was still pending after 4 days on a commercial Unix variant). Again, system logs would help diagnose this.
Now, the above statement that this "shall never happen" depends on an accurate definition of the word "deadlock": Theoretically you should always get eventual progress, but under unusual workloads, it might take a very long time. Nothing guarantees fairness, and your process might be the victim of starvation (the words fairness and starvation are terms of art in parallel systems, and have specific meanings). In practice, this will only happen if some other process is keeping the locking or file systems insanely busy. You can try to debug that with top or ps, and see whether there is another process that has gone berserk and is beating up the OS so much that it doesn't have any time to server your flopen request.