Absolutely agree, and I should have listed that as an exception. Matter-of-fact, I was looking at using Kivy directly to framebuffer on a Raspberry Pi a few months ago, and got it to kind of work once (and then the project got abandoned due to lack of time). Still, I didn't attempt to do graphic operations directly on the framebuffer; instead I used Kivy to draw some simple things.
Just for a historical note: Before the mid-80s, I already did some graphics, on micros (6809 with video output from RAM), personal machines (Commodore 2001, Apple II), minicomputers (VAXen), and mainframes (with Tek4014 attached). On the VAX we had a Grinnell frame buffer that was murder to use and unreliable (often would lock up the Unibus), and later a dedicated 11/750 with an Evans&Sutherland video output device. Then workstations started making things even more complex, initially MicroVAXen with their VWS windowing system. Programming any of those was an enormous amount of work, and no two things used the same interface. It was brutal. Then X showed up, and suddenly there was a light at the end of the tunnel: We could do most of our work with a single programming interface! Simplicity and convenience had been achieved. I remember taking a graphics library (UG = "Unified Graphics" out of SLAC, used by the TopDrawer graphical data analysis program), and writing an X backend for it; took a few weeks, but was a great investment, since I could stop maintaining and fixing the half dozen other backends.
And this is why I get upset when people like Spartrekus glibly want to throw great software (like X) out the window, just because of their prejudices. Yes, it can be fun as an educational exercise to learn how to do graphics directly. But just because it can be done doesn't mean that it should be done, nor does it mean that you should do it the hard way directly to the hardware. If he wants to learn to draw points, lines, markers, and figures, he should go to a good common denominator, and in Unix land that's something like OpenGL, Qt, or raw X.