I got it running
))
finaly i got it running...
i had to increase valuses for theese sysctl values
kern.ipc.semmns=1024
kern.ipc.semmni=512
I havent tested to fine tune theese values, may be they are too high (defaults are semnns=60 and semmni=10).
So here are all steps in case someone need to get sybase running on freebsd. May be some of the steps are not nessesery but I think they doesnt hurt.
1.1 kldload linux
1.2 add linprocfs to /etc/fstab and mount it
1.3 install emulation/linux_base-fc4
2.1 download linux version of sybase developers edition from sybase (they will provide you with key), i have downloaded 10.0.1 version, because my database currently is on this version
http://www.sybase.com/detail?id=1055872
2.2 untargz the file
2.3 edit file setup and replace #!/bin/sh with !/compat/linux/bin/sh
2.4 find string /bin/tar in the file and replace it /usr/bin/tar`
(optional 2.5.... edit every occurance of #!/bin/ and replace it with #!/compat/linux/bin in all files)
3.1 sysctl compat.linux.osrelease=2.6.16
3.2 execute setup and install (on step with the key enter the dev key or real key, if dev key is entered there is prompt on every start of dbeng10/dbsrv10)
I have deselected all components except sql engine/server/client and tools
3.3 there were express bug fix for sql anyhere 10.0.1 I had downloaded it and installed it like in previous steps 2.2-3.2
4.1 cd to the directory where installed sqlanywhere (I installed in /compat/linux/opt/sqlanywhere)
4.2 find every occurance in files of #!/bin and again replace it with #!/compat/linux/bin
5.1 return the defalt value for compat.linux.osrelease - sysctl compat.linux.osrelease=2.4.2
5.2 edit /boot/loader.conf and add theese:
kern.ipc.semmns=1024
kern.ipc.semmni=512
5.3 reboot
6.1 execute
/compat/linux/bin/sh
6.2 execute source /compat/linux/opt/sqlanywhere/bin32/sa_config.sh
6.3 run whatever needed from sqlanywhere (dbeng dbsrv10 dbstop dbisqlc ... etc.)
Thats it.
Of course, it would be good to have rc scripts for running and stopping server (especially stopping) it

And may be running as different (non root) user.
p.s. tested on freebsd 7.1-stable i386 (sources and world from 11.2.2009)