Other Debugging and automated testing of C# mono programs?

aragats

Daemon

Reaction score: 524
Messages: 1,255

Does anybody know any useful tools on the subject?
I have a lang/mono application written in C# which runs fine, but it's not perfect yet.
Personally I wouldn't ever use any .NET stuff, but I have to deal with this now. Maybe sometime later I'll re-write it in plain C though.
 

ralphbsz

Daemon

Reaction score: 1,178
Messages: 1,907

What is the input and output and interaction of the program?

If it is text- or file based, then it doesn't really matter what language it is written in. The simplest example of such an application is the standard Unix filter, which reads from stdin and writes to stdout. In testing, you can treat it as a black box, and it doesn't matter at all what language it is written in. You just feed it a battery of test cases on its input, observe the output, and verify that the output is as expected. For a standard filter, this is actually quite simple to do with a shell scripts, a series of canned inputs and outputs, and diff.

If the program reads and writes other files, or has other state it stores somewhere, it gets a little harder, but it still doesn't change the basic paradigm: feed it input, observe the output, and keep track of state changes. If the program has GUI interaction (draws output in windows, requires keyboard or mouse input), then testing gets much harder to automate, since one has to write programs to pretend to use keyboard and mouse, and observe the pixels change.

But the important part is this: If you want to treat your program as a black box for the purpose of testing, then it doesn't matter what language it is written in, and C# and Mono make no difference at all.

There are lots of software testing frameworks around. Professional software testing people use and write those, and some are supposedly very useful and save work. When I'm acting as an amateur developer for stuff at home, I ignore those frameworks, and instead write simplistic shell scripts that just check the output of the program.
 
OP
OP
aragats

aragats

Daemon

Reaction score: 524
Messages: 1,255

It's conglomerate: stdout, files, sockets and GUI (-:
The reason I mentioned mono is because it introduces an extra layer, and, e.g. it's no obvious how to find and trace back a memory leak.
 

ralphbsz

Daemon

Reaction score: 1,178
Messages: 1,907

If you are interested in internals like memory leaks, you are doing white box testing: you are looking inside the program itself. Unfortunately, I have zero experience with C# and mono or .net.
 
OP
OP
aragats

aragats

Daemon

Reaction score: 524
Messages: 1,255

A regular C program you can run in a debugger, and here you see only a mono process running, so debuggers are useless...
 
OP
OP
aragats

aragats

Daemon

Reaction score: 524
Messages: 1,255

If the program has GUI interaction (draws output in windows, requires keyboard or mouse input), then testing gets much harder to automate, since one has to write programs to pretend to use keyboard and mouse, and observe the pixels change.
I successfully use x11/xdotool in a shell script for such purpose.
However, there exist other approaches to this. For example, some time ago I worked in a team of a big project where most C functions had Tcl-C bindings (see lang/tcl87). So, we could trigger any function calls by running Tcl scripts without actual interactions with GUI elements.
 

ralphbsz

Daemon

Reaction score: 1,178
Messages: 1,907

We did something similar about 20 years ago. We were writing a large system in Java; it was about 80 KLoC when I left, and in the meantime has grown to over 12 MLoC. We discovered that there is a pretty good Python implementation that's written in Java, and can be embedded into Java based programs. It allowed both calling Java-based code from the Python command line, and it allowed escaping out of Java and executing Python scripts. Because the Python implementation was able to see the Java "symbol table", we had direct access to objects and variables.

The first big advantage was the following. The system was to control complex hardware, but that usually means that actually using the software required access to specialized hardware, and if every "machine" costs many M$, that means that debugging and testing gets expensive and tedious. So we used a combination of Java and small Python scripts to write hardware "emulators". Those were usually pretty stupid, but good enough to get the whole software system to basically work on a laptop against emulated hardware. And because many of the emulators were just small scripts, we had the ability to make them more complex by quickly modifying the scripts (no recompile required!), for example injecting errors or malfunctions (like you order a stepper motor to go to 123.4 degrees, and it instead goes to 4.321 degrees).

The other advantage was that a lot of the "testing" (in the sense of quality control testing) could be done in white-box mode (with full access to the software), by testing functional units from Python scripts. And by hooking Python scripts into the communication to remote computers and to the GUI interface, we were able to automate a lot of testing.

Finally, one more advantage: It meant that the whole complex software system (which has to be complex, given the real-world complexity of the hardware and the problem we're trying to solve) had simple to use interfaces. For example, manufacturing technicians who really weren't skilled enough to do programming were able to reach into the system and hack up small scripts (often written horribly badly, but functional) to exercise the hardware during assembly. It gave a big and complex thing a simple-to-use interface.

Was this all a lot of work? Absolutely. But if you have a team of ~100 software engineers, who will spend many hundreds of person-years into a system, then investing in infrastructure like this is good business.

(Sorry for the off-topic nature of this post; it's sort of background info to the on-topic question.)
 
OP
OP
aragats

aragats

Daemon

Reaction score: 524
Messages: 1,255

Thanks!
unitrunker , do you know, is there a way to create and run regression tests in monodevelop?
Of course, it will be hard to use monodevelop in an embedded system, will have to assume that an application debugged in a desktop will behave the same way on a different platform.
 

unitrunker

Active Member

Reaction score: 66
Messages: 184

My main interest is microservices written as WebApi in C#.

As an example, I have a solution containing multiple projects - some depending on others. One project is exclusively for unit testing. I used nuget to install NUnit and monodevelop's IDE to create an empty unit testing project. I added a test class and methods. The tests magically appear in the Unit Testing side panel of the IDE with red/green highlights to show which passed or failed.

Example unit test method:

C#:
        [Test]
        public void TestAuthenticateValidate()
        {
            var auth = new Bob.AuthController();
            var request = new Contract.Authenticate.Request();
            request.Login = "Alice";
            request.Pass = "sprinkles";
            var access = auth.Put(request);
            Assert.IsNotNull(access);
            Assert.IsNotNull(access.Token);
            Assert.IsTrue(access.Success);
            var scope = auth.Put(access.Token);
            Assert.IsNotNull(scope);
            Assert.IsTrue(scope.Success);
        }
 

unitrunker

Active Member

Reaction score: 66
Messages: 184

Above is a unit test and not a regression test.

Regression tests - in the context of REST services can be as simple as a POSIX script with lots of calls to fetch. I don't know what external interface your app provides. Regression testing options would be very different for a GUI.
 
Top