stosb instruction on local variable problem

[arvin]

I compile the following code

#include <stdint.h>


static __attribute__((always_inline))
void
stosb ( char *dst, unsigned long len, char val)
{
	__asm__ volatile (

		"pushfq; "
		"cld; "
		"rep stosb; "
		"popfq;"

		: "=c" (len), "=D" (dst)
		: "c" (len), "a" (val), "D" (dst)
		: "memory" );
};


int
main ()
{
	uint64_t value_64;

	stosb ((void*) &value_64, sizeof(value_64), 1 );
}

with gcc (version 4.2.1 20070719 [FreeBSD]) and the -O3 flag set on a
FreeBSD System (8.1-RELEASE FreeBSD 8.1-RELEASE #0).

When I run it I always get a trace trap:

% ./build/bin/test_memory

zsh: trace trap (core dumped)  ./build/bin/test_memory

Any suggestion what the problem might be? Btw. if I remove the always_inline
attribute and hence the stosb function isn't inlined anymore everything works
fine.

Thanks in advance

 

[Beastie]

The di register or equivalent must hold the destination "dst" and the cx register or equivalent must hold the number of times "len" rep will repeat the stos instruction. Is that what you are doing? Because otherwise, you are probably overwriting the wrong memory location.

 

[arvin]

Yes exactly. The expression

: "c" (len), "a" (val), "D" (dst)

tells the compiler to store len into $cx, val into $ax and dst into $di register,
if I understood right.

 

[arvin]

[SOLVED] stosb instruction on local variable problem

Hi Guys

We could actually find and fix the bug. The problem was that gcc was using due to optimization (O3 flag) the red zone below the stack pointer to hold value_64. But since our asm code was also using this region by pushing the EFLAGS register into it and then afterwards poping again, the EFLAGS register was not actually restored but overwritten by the value in %eax. In our case this was setting the TF (Trace Flag) and this was causing the trace trap signal.

Arvin