A try to look behind the story of Kangjie Lu/Quishi Wu vs. Greg Kroah-Hartman, also known as the "Hypocrit Commit" story

O

obsigna

Profile disabled
itwire.com

iTWire - Uni group slammed over submitting known buggy patches to Linux kernel

A group from the University of Minnesota has come in for a tongue-lashing from the normally mild-mannered Linux developer Greg Kroah-Hartman, the maintainer of the stable kernel. Kroah-Hartman blew up after the group submitted patches to the kernel which were known to be buggy. He said in a post...
itwire.com itwire.com

Intentionally buggy commits for fame—and papers [LWN.net]

lwn.net lwn.net
linuxreviews.org

Linux Kernel Developers Were Not Amused By Faulty Patches Sent By University of Minnesota Researchers

linuxreviews.org linuxreviews.org

I am not interested in Linux, it starts already that I dislike the name, and my dislike by far does not end at this point. I like FreeBSD, and once I read the story, the first question which came to my mind was, whether Mr. Wu tried to interfere into FreeBSD development as well. So, I searxed for the keywords FreeBSD "Quishi Wu".

According to what I found so far, he did not, at least not by using his real name. However, what I found is, that Quishi Wu seems to be a bare bone scientist, who wrote very well elaborated papers on security issue, which probably most of those who are referring to the issue as "Intentionally buggy commits for fame—and papers" didn't read, and even I they had, won't understand much beyond the abstract of it:
  • Detecting Missing-Check Bugs via Semantic- and Context-Aware Criticalness and Constraints Inferences [PDF]
  • Automatically Identifying Security Checks for Detecting Kernel Semantic Bugs [PDF]
  • Precisely Characterizing Security Impact in a Flood of Patches via Symbolic Rule Comparison [PDF]
  • Detecting Kernel Memory Leaks in Specialized Modules with Ownership Reasoning [PDF]
  • Understanding and Detecting Disordered Error Handling with Precise Function Pairing [PDF]
... and finally the incremented paper:
  • On the Feasibility of Stealthily Introducing Vulnerabilities in Open-Source Software via Hypocrite Commits [PDF]
I gave the links of the papers as listed on the GitHub-Repository - for convenience. The papers are cited in many locations in the SearX-results, though. From what I see, these are well renown and peer-reviewed ones. Some of the discussed issues involve FreeBSD as well.

My conclusion, there are scientists and there are the folks, the first ones know what they are doing, while the others believe that they know something.
 
I have been following this story on the surface only, without digging deep. I think what we have here is a group of academic computer security/resilience researchers, who have been studying error handling in kernels. A valid field of study, and they seem to be legit. They created a mechanism for automatically finding coding problems in kernels, and published about it and released the tool. They are not the only ones; I know of another group (at Stanford) that does similar work, and whose tools have become a commercial product of some note. Great.

Then these researchers went one step further: They wondered where these security-relevant kernel bugs come from, and the obvious answer is obvious: they are introduced by humans who write code (duh). So how difficult would it be to introduce them (whether inadvertently or deliberately)? Being of the researcher mind, they tried to introduce them. And here the story gets complicated. The process of reviewing and committing code is not longer in the realm of pure computer science, it is an engineering process, based on sociological factors such as trust and interpersonal relationships. This requires a skill set larger or different of those required for a normal CS researcher. They introduced a few good commits, then a few with deliberate vulnerabilities in them. Whether their good commits were actually useful or not, and whether their broken commits were actually accepted or not seems to be a matter of some debate. What they succeeded in doing is create a big brouhaha, and make public the fact that commits to the Linux kernel have an "interesting" way of being validated, based on a notion of "trust" that's a bit odd. I think right now, with emotions running high (even the most evil Linus personally got into the fray, in spite of the fact that he had banned himself for being psychologically not stable enough to lead a large project), it is too soon to evaluate what this all means.
 
I don't know about Linux, but I'm convinced that, at least, some of the FreeBSD code commits aren't reread or very lightly reread.
We got this recently: https://forums.freebsd.org/threads/...focused-on-wireguard-regarding-freebsd.79537/

Even I, at my little level, found several bugs in pmbr. They exist since 2007 I believe and, as far as I know, they never been corrected since*. Ok, that is not vulnerabilities.

You see, it's not like a C code deeply entangled in the kernel where to find a bug demands many knowledges. It's just what an assembly mnemonic does or does not. I think no one has never reread this code.
PR 233180

(*) They have been corrected and commited in 14-CURRENT, the jully 14th 2021.
 
Last edited:
Both the "wiregard in FreeBSD 13" story and your bug with coding errors in pmbr are qualitatively different. They are about the question "do bugs exist in the kernel", and they certainly do.

The question the authors of this research paper wanted to answer is: Can bugs be maliciously introduced? In the sense of: How difficult is it to exploit the system of (human) trust to add security vulnerabilities? I don't think we know the answer from the current "U Minn versus Linux" drama. But AFAIK there is no suspicion or hint that either Wiregard or bugs such as the one you tried to get fixed are intentionally introduced to be security vulnerabilities.
 
ralphbsz said:
The question the authors of this research paper wanted to answer is: Can bugs be maliciously introduced? ... I don't think we know the answer from the current "U Minn versus Linux" drama.
Click to expand...
Really? The authors managed to introduced malicious code, and the Linux kernel devs became only aware of this, because they were pointed to the published paper by a third party. The Linux heroes were f***ed without knowing it, which is of course the most embarrassing occasion of being f***ed, and because of this they're so upset.

I would have simply accepted the result, and work together with scientific research groups for developing tools which do automatic code analysis. The static analyzer of clang would be a really good start for this. This and then somebody does this for FreeBSD code snippets, and yes the analyzer almost always finds issues. However, it seems that this magnificant tool is not widely used.

In some of the cited papers of the research group of the University of Minnesota referred to this kind of tools, and it seems that they developed some tools/procedures for finding hard to debug concurrency problems.

No, the Linux heroes prefer to stay in the pout corner, and are asking for a refund in order to let them in again -- sort of U-Minn contracts Linux nannies for learning how to commit code without embarrassing anybody.

OK, f**k yourself then.
 
obsigna said:
Really? The authors managed to introduced malicious code, and the Linux kernel devs became only aware of this, because they were pointed to the published paper by a third party.
Click to expand...
According to another version of the story, of the 3 or 4 malicious commits the authors made, some were rejected because of unrelated problems, some were rejected because the bug they wanted to introduce was actually caught, and some were withdrawn by the authors, and none made it into production. The fact that multiple versions of the story exist tells me that I don't know what the truth is.

I would have .. work together with scientific research groups for developing tools which do automatic code analysis.
Click to expand...
Such tools exist, they have existed for a decade or more, and they have been used successfully to find bugs. For an example, search the web for "Dawson Engler model checking ext3 reiser", and you'll find a published paper. I don't know whether using such tools is part of the normal Linux kernel development workflow, but I know several commercial development environments where such tools are used. Matter-of-fact, in a previous job we used to use exactly the tool Dawson Engler's company sells, and I would get a few bugs per week in my e-mail. Most of the time, the automatic analyzer was "correct" in that it found real problems in the code, but also most of the time the real problems were not relevant, because they found problems in a code path that either can not be executed, or is only relevant in corner cases in which a bigger problem already exists.

I think your characterization of what happened is highly biased, and not everyone agrees with it. I don't pretend to know what the real situation is.
 
ralphbsz said:
Both the "wiregard in FreeBSD 13" story and your bug with coding errors in pmbr are qualitatively different. They are about the question "do bugs exist in the kernel", and they certainly do.

The question the authors of this research paper wanted to answer is: Can bugs be maliciously introduced? In the sense of: How difficult is it to exploit the system of (human) trust to add security vulnerabilities? I don't think we know the answer from the current "U Minn versus Linux" drama. But AFAIK there is no suspicion or hint that either Wiregard or bugs such as the one you tried to get fixed are intentionally introduced to be security vulnerabilities.
Click to expand...
My point is: the code isn't always reread and verified before to be committed. For example, no tool can find the bugs I was speaking of, as simple as they are, because a tool is not aware what the program is supposed to do.

No matter if bugs were intentionally inserted or not, they are bugs. I just showed that some of them could be easily avoided without the need of a kernel guru. I think most people aren't inclined to spend time on code verification because they feel it ins't as noble as writing it.
 
ralphbsz said:
Dawson Engler's company sells... ... because they found problems in a code path that either can not be executed, or is only relevant in corner cases in which a bigger problem already exists.
Click to expand...
  1. The work of Engler et.al. resulted into the open source project KLEE and can be used by anyone -- http://klee.github.io

  2. Code paths that can't be executed today are the security holes of tomorrow, those paths are to be removed from the code before deployment. I have zero tolerance for developers who tend to leave their brightest ideas (which usually come by the tenth beer) in the working tree within #if 0 ... #endif, or even better if (0 && ...) {...}. A single inattentive copying & pasting in the code may turn the brightest idea into the darkest nightmare.

  3. Guess what, corner cases are exploited by malware.
Independent of whether OSS developer's use clang's built-in static analyzer and/or KLEE and/or other analyzers, malware developers do.

I just found the presentation of David A. Ramos and Dawson Engler in 2015 about static code analyzing in general and KLEE in special: https://www.usenix.org/sites/default/files/conference/protected-files/sec15_slides_ramos.pdf

Here come some some statements:
  • Found 77 new bugs in BIND, OpenSSL, Linux kernel
  • Bugs are everywhere!
  • PATCHES
    • 99 little bugs in the code
    • 99 little bugs in the code
    • take one down and path it around.

    • 127 little bugs in the code
  • EVALUATION: PATCHES
    Discovered 10 new bugs (4 in BIND, 6 in OpenSSL)

  • EVALUATION: GENERAL BUG-FINDING
    20,000+ functions: BIND, OpenSSL, Linux kernel (~12 MLOC)
    Found 67 new bugs
    Linux kernel: exploitable AUTH_GSS leak in NFS SunRPC layer
    Linux kernel: leak of private kernel stack data via firewire ioctl
    Linux kernel: VMware VMCI driver: unchecked memcpy length (~Heartbleed)
    Linux kernel: CEPH distributed file system: division-by-zero (kernel FPE)

Now an old finding: TWO results of a run of clang's static analyzer on the libreoffice code base in 2012:

726 bugs

3 month later
95 bugs

Perfect, at least the libreoffice dev team utilizes automated analysis tools. The static analyzer of clang evolved very much since.

Here a simple use case on this Forums:
forums.freebsd.org

C - valgrind: memory gets used up but it is not showing the what or why

userx@FreeBSD.net:~ $ valgrind --tool=memcheck mhsetroot --17065:0:procself sysctll 1 userx@FreeBSD.net:~ $ valgrind --tool=memcheck --leak-check=yes mhsetroot --30861:0:procself sysctll 1 userx@FreeBSD.net:~/Downloads/mhsetroot-2.0 $ valgrind --tool=memcheck --leak-check=yes ./mhsetroot -s -o...
forums.freebsd.org forums.freebsd.org

The static analyzer comes for free with clang in FreeBSD -- goddamn it, just use it!!!!!

So, now elaborate again on why automated code analysis is useless.
 
I definitely didn't say that automated static analysis is useless. Matter-of-fact, I have been pushing for it in various settings (typically at work). But I also don't think that it is a panacea. To quote one of the two best books on software engineering: there is no silver bullet. If you want to write high-quality (resilient) code, then automatic analysis is one of the arrows in your quiver, but you also need many others.

Your characterization of dead code and corner cases doesn't match what I intended to say. Example 1: You are writing a function that implements how animals move (standard example from OO software engineering text books). Some walk, some swim, some fly, some crawl. How they do that depends on their length, weight, and number of appendages (a spider is not like a blue whale). An automated tool like Coverity might find a bug that can only occur if the animal is an elephant, and if the motion is flying (note: I did not say "code that explicitly implements flying elephants", I said a bug in a code path that would only happen if elephants could fly). For example: A variable is not initialized in the case of flying, but then only used in the case of elephants. While this is dangerous and sloppy (exactly as you described, because the code might change later), it is also not a bug. Similarly, the second case is that a tool like Coverity might find the lack of a malloc'ed memory being freed ... in a routine that is only called in the code path in which the system is already shutting down due to an assert having been fired. So it is not really a memory leak.

Here is what I would conclude, from my experience: Automatic analysis tools are useful, and sometimes find real bugs. But they also require a significant investment in time (and sometimes money) to set up and run, and they require a lot of time to babysit. That's because for every real bug they find, they'll also find many things that are nominally problems, but can never affect real users/customers. They are best used in conjunction with many other techniques, such as a reasonable management style, a user/customer focussed corporate culture, good engineering practices (such as explicitly creating requirements, designing before implementing), careful coding, hiring skilled engineers, a culture of openness, code reviews and inspection, and a thorough test regime run by independent testers and auditors. As I said, not a silver bullet.

The question brought on by the case the OP refers to is: Does Linux have the mindset of creating resilient and high-quality software? The U Minn group purportedly wanted to measure that. It is not clear to me what their contribution to our knowledge ultimately was.
 
I saw the story, as well, and I'm thinking, maybe UM researchers were treating this like a pen test (network penetration testing). Not coming in with intent to do damage to the Linux kernel's reputation for quality, but checking the watertightness of the development process. Linux devs never got a heads up about that kind of audit, so they understandably got angry when the development process was shown to have such cracks.

Reading stories of actual pen testing on the Internet (via Firefox's Pocket like Wired Magazine's (https://www.wired.com/story/inside-...ee-that-landed-two-white-hat-hackers-in-jail/)), quite a few of the professional pen testers got in trouble for their 'unauthorized messing around inside' just like the UM researchers did with their patches.
 
You must log in or register to reply here.
Back
Top