Port www/webkit-gtk3 freezing on compilation

W

wisdown

Hey guys,

Someone already got stuck when compiling this port?
For me it doesn't matter if I use the FreeBSD native compiler (Clang) or if I force gcc49 for example. Always the compilation freezes on:

Code: 
CXX Spurce/WebKit2/unix/Programs_WebKitPluginProcess-PluginMainUnix.o

For freeze, I mean nothing else works, the HD stops read/write, I can switch between consoles, but the consoles don't let me use them, only switch, then I reboot the frozen system.

There any place where I can check the compiler log for try investigate this issue?

My make.conf:

Code: 
OVERRIDE_LINUX_BASE_PORT=c6
OVERRIDE_LINUX_NONBASE_PORTS=c6

# Set the Processor
CPUTYPE?=core2

# Force gcc49 as default compiler
# .if !empty(.CURDIR:M/usr/ports/*) && exists (/usr/local/bin/gcc49)
# CC=gcc49
# CXX=g++49
# CPP=cpp49
# .endif

Thanks in advance.

PS.: I have compiled entire system using the FreeBSD 10 default compiler without problems, just have did a try with gcc49 for see if would be possible avoid this situation.
 
Maybe, it is too less swap space or /var/tmp, www/webkit-gtk3 is big program needs a lot of cache.

By the way (I know it is commented out): I wonder about the first .endif line. You have two endif. Should result in an error message. The first one is not necessairely.
 
talsamon said:
Maybe, it is too less swap space or /var/tmp, www/webkit-gtk3 is big program needs a lot of cache.
Click to expand...

Probably is not about swap space, I have did a slice of 32GB for it, and got this warning on every reboot:

Code: 
warning: total configured swap (8388608 pages) excedes maximum recommended amount (6070688 pages)

Reading blogs and this forum, people say I can ignore this warning.

After do the slice, I have checked twice using:


Code: 
dd if=/dev/zero of=/dev/ada0p4 bs=4096
dd if=/dev/random of=/dev/ada0p4 bs=4096

No errors found (and after random parameter, no more errors about checksum), by the way, I'm using an encrypted swap, should this be the problem?
All other ports (I'm installing one by one trying get gnome3) have compiled without errors

After check my zroot pool, there no errors too:

zdb zroot

(So slow, this one reminds me of the scandisk from Windows)
http://docs.oracle.com/cd/E23823_01/html/816-5166/zdb-1m.html

I have no idea how to check an encrypted swap slice, since zpool status doesn't show the slice.

My zroot has 100 GB free (this includes /var/tmp)

Where is the .endif file?
Inside of /usr/ports/www/webkit-gtk3/work I have not found it.
 
Ah got it, theres an typo too, going to try compile again without those lines.

I have one slice for swap with 32GB and one for cache with 8 GB (have read somewhere the recommendation for cache is at least double of RAM).

If doesn't work, going to try to swap the slices too.
(32 GB for cache and 8 GB for SWAP)

After you point me about space, and when I was monitoring it too trying found how to fix, I have noticed the cache slice get full and swap only use around 2 GB then freeze.

Will post the result of those tests after few hours.

Thanks for try help.
 
I have swaped the slices, so instead have those slices:

32 GB SWAP
8 GB ZFS cache

I changed for:

32 GB ZFS cache
8 GB SWAP

The warn on dmesg about exced swap size disappear.

But, this port still freeze on same step.
 
Thanks for the reply fulano.

I have cleared my make.conf and got no success compiling www/webkit-gtk3, freezing at same point.

Have compiled with success:

www/webkit-gtk2
www/webkit-qt4
www/webkit-qt5

Currently compiling: www/webkit2-gtk3

And more then other 800 packages, doing one by one with no problems like this one (sometimes dependence missing, then i fixed checking they with pkg_libchk).
Seems the issue happen only with www/webkit-gtk3.

talsamon,

You right, change the slices get more room for zfs cache not solve the problem, the compilation stuck at same file.

Any other idea guys?
 
1) change in the port and make rm -rf and fetch it with portsnap extract [U]www/webkit2-gtk3[/U] new.
2) www/webkit-gtk3 has only two options debug and fulldebug, they are off. This means there nearly no difference to use a package. Try to install it with pkg install [U]www/webkit2-gtk3[/U].
 
According to Portsmon, both webkit2-gtk3-2.6.6 and webkit-gtk3-2.4.9 are compiling fine.

The ZFS cache is best when it's a seperate device. If ZFS isn't satisfying the request via the in RAM ARC, then hitting disk for either L2ARC or the actual file is going to take the same amount of time. If your disk needs to jump back and forth between L2ARC on one partition and the data on another partition on the same disk you may end up with worse performance then if ZFS was just able to read from the data partition.
 
webkit2-gtk3-2.6.6 and webkit-gtk3-2.4.9 are compiling fine.
Click to expand...

I don't think so. I don't need www/webkit-gtk3, but I tried to compile it (2.6.6). Fails with gcc and clang with this message, It seems an other error as above:
Code: 
Linking CXX static library ../../lib/libGObjectDOMBindings.a
gmake[4]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
[ 83%] Built target GObjectDOMBindings
gmake[3]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
Makefile:136: recipe for target 'all' failed
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
===> Compilation failed unexpectedly.
 
talsamon said:
I don't think so. I don't need www/webkit-gtk3, but I tried to compile it (2.6.6). Fails with gcc and clang with this message, It seems an other error as above:
Code: 
Linking CXX static library ../../lib/libGObjectDOMBindings.a
gmake[4]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
[ 83%] Built target GObjectDOMBindings
gmake[3]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
Makefile:136: recipe for target 'all' failed
gmake[2]: *** [all] Error 2
gmake[2]: Leaving directory '/usr/ports/www/webkit2-gtk3/work/webkitgtk-2.6.6'
===> Compilation failed unexpectedly.
Click to expand...
That may be a problem specific to your environment. I just installed both ports on a new install a few hours ago and they both compiled and installed fine on my machine. :)
 
talsamon,

Trying pkg install www/webkit-gtk3 got 2 extras packages:

devel/boehm-gc
devel/boehm-gc-threaded

I have accepted, then removed after with:

pkg delete boehm-gc-7.4.2_3
pkg delete boehm-gc-threaded-7.4.2_3

Then:

cd /usr/ports/devel/boehm-gc
make DISABLE_VULNERABILITIES=YES deinstall clean rmconfig fetch-recursive config-recursive install clean
cd /usr/ports/devel/boehm-gc-threaded
make DISABLE_VULNERABILITIES=YES deinstall clean rmconfig fetch-recursive config-recursive install clean

After this I have tested with:

pkg_libchk

Everything seems fine.

For next I have tried:

pkg install www/webkit2-gtk3

Then for my surprise says need remove:

lang/guile2
devel/autogen

So my guess is probaly those two would be making some kinda of conflict (not sure).

And say need install too:

lang/x10

My second guess is this one probably is an missing dependency where ports does not install before.


junovitch,

Thanks for the reply, I don't have an benchmark test or something like that for test the current setup, and cannot install an second HDD or SSD for better performance, this install I'm doing on an old notebook Core 2 Duo with 4 GB RAM, but, I can say empirically using an slice from inner tracks in HDD for cache gives me a better compilation time, better than devel/ccache does.
 
protocelt said:
That may be a problem specific to your environment. I just installed both ports on a new install a few hours ago and they both compiled and installed fine on my machine.
Click to expand...

So actually we have two environments with problem and not an isolated situation...
Probably, if more people try recompile we should see more errors, but since the usually I have see is compile one time and forget, people will hold the first compiled version and aren't testing other versions.
 
This is insane. I am working on QA for some PRs and added www/webkit-gtk3 to my build list. I kicked this off overnight and it took nearly 6 hours for that one package. I only let it finish in my 8.4-RELEASE amd64 Poudriere jail. I removed it since I have to finish the other work.

Code: 
[00:55:52] ====>> [03][00:00:00] Starting build of www/webkit-gtk3
[06:50:04] ====>> [03][05:54:12] Finished build of www/webkit-gtk3: Success
 
protocelt you are right. It compiles with poudriere fine. The Enviroment "problem" I think I had no chance to find. But it seems no problem, all other ports (including LibreOffice) compiles since the last installation fine. (And as junovitch said, I think it's better to use a package for www/webkit2-gtk3, it needs a lot of time to compile, especially with poudriere).
 
Sorry for the delay on answer guys, was busy with some personal problems, but, was letting the notebook compiling the programs, so gonna share my final result (and yes, got GNOME enabled).

And thanks for all support, gonna answer in order of replies:

junovitch,

This is insane. I am working on QA for some PRs and added www/webkit-gtk3 to my build list. I kicked this off overnight and it took nearly 6 hours for that one package. I only let it finish in my 8.4-RELEASE amd64 Poudriere jail. I removed it since I have to finish the other work.
Click to expand...

In other forums some people sayed samething, they are able to compile without problems. Then I have noticed mostly people able to compile all, are the one runing "old versions" of FreeBSD, didn't see this working on latest and updated release.
I have tried use the version 9.3, but unfortunaly this version seems lack on drivers for my old notebook, the 10.1 seems work better on detect hardware, is missing some drivers of course, but detect more than 9.3, by the way, with CLANG seems is the source of problems compiling (we can't have all, need choose one).

From hardware the major problem for me is don't have nouveau working, I have read about this in process of be ported, while this not happen, I gonna deal with the lagged driver of Nvidia (Nvidia 7900GS Video), this notebook seems be problematic on Linux too, the only time I have get this working nice was on Ubuntu 10.0 using nouveau... An short briefing:

http://www.linlap.com/toshiba_satellite_p105

My dmesg:

Code: 
Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.1-RELEASE-p10 #0: Wed May 13 06:54:13 UTC 2015
  root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
VT: running with driver "vga".
CPU: Intel(R) Core(TM)2 CPU  T7200  @ 2.00GHz (2006.63-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x6f6  Family = 0x6  Model = 0xf  Stepping = 6
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0xe3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM>
  AMD Features=0x20100800<SYSCALL,NX,LM>
  AMD Features2=0x1<LAHF>
  VT-x: HLT,PAUSE
  TSC: P-state invariant, performance statistics
real memory  = 4294967296 (4096 MB)
avail memory = 3068743680 (2926 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <TOSQCI TOSQCI00>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
ioapic0: Changing APIC ID to 1
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Cuse4BSD v0.1.36 @ /dev/cuse
random: <Software, Yarrow> initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80d94ca0, 0) error 19
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
acpi0: <TOSQCI TOSQCI00> on motherboard
acpi0: Power Button (fixed)
unknown: I/O range not supported
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 450
Event timer "HPET1" frequency 14318180 Hz quality 440
Event timer "HPET2" frequency 14318180 Hz quality 440
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
acpi_ec0: <Embedded Controller: GPE 0x17> port 0x62,0x66 on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0x2000-0x207f mem 0xd1000000-0xd1ffffff,0xc0000000-0xcfffffff,0xd0000000-0xd0ffffff irq 16 at device 0.0 on pci1
nvidia0: <GeForce Go 7900 GS> on vgapci0
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: child nvidia0 requested pci_enable_io
vgapci0: Boot video device
hdac0: <Intel 82801G HDA Controller> mem 0xd2500000-0xd2503fff irq 22 at device 27.0 on pci0
pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pcib2: failed to allocate initial I/O port window: 0-0xfff
pcib2: failed to allocate initial memory window: 0-0xfffff
pcib2: failed to allocate initial prefetch window: 0-0xfffff
pci2: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection 7.4.2> irq 16 at device 0.0 on pci2
em0: Using an MSI interrupt
em0: Ethernet address: 00:16:36:de:84:aa
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
pcib3: failed to allocate initial I/O port window: 0-0xfff
pcib3: failed to allocate initial memory window: 0-0xfffff
pcib3: failed to allocate initial prefetch window: 0-0xfffff
pci3: <ACPI PCI bus> on pcib3
wpi0: <Intel(R) PRO/Wireless 3945ABG> irq 17 at device 0.0 on pci3
pcib4: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pcib4: failed to allocate initial I/O port window: 0-0xfff
pcib4: failed to allocate initial memory window: 0-0xfffff
pcib4: failed to allocate initial prefetch window: 0-0xfffff
pci4: <ACPI PCI bus> on pcib4
uhci0: <Intel 82801G (ICH7) USB controller USB-A> port 0x1800-0x181f irq 23 at device 29.0 on pci0
uhci0: LegSup = 0x2c00
usbus0 on uhci0
uhci1: <Intel 82801G (ICH7) USB controller USB-B> port 0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: LegSup = 0x2c00
usbus1 on uhci1
uhci2: <Intel 82801G (ICH7) USB controller USB-C> port 0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: LegSup = 0x2c00
usbus2 on uhci2
uhci3: <Intel 82801G (ICH7) USB controller USB-D> port 0x1860-0x187f irq 16 at device 29.3 on pci0
uhci3: LegSup = 0x2c00
usbus3 on uhci3
ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xd2504000-0xd25043ff irq 23 at device 29.7 on pci0
usbus4: waiting for BIOS to give up control
usbus4: timed out waiting for BIOS
usbus4: EHCI version 1.0
usbus4 on ehci0
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pcib5: failed to allocate initial memory window: 0xd2100000-0xd21fffff
pci10: <ACPI PCI bus> on pcib5
cbb0: <PCI-CardBus Bridge> irq 17 at device 4.0 on pci10
cardbus0: <CardBus bus> on cbb0
pccard0: <16-bit PCCard bus> on cbb0
pci10: <serial bus, FireWire> at device 4.1 (no driver attached)
pci10: <mass storage> at device 4.2 (no driver attached)
sdhci_pci0: <Generic SD HCI> irq 17 at device 4.3 on pci10
sdhci_pci0: 1 slot(s) allocated
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH7M SATA150 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x18b0-0x18bf at device 31.2 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
acpi_acad0: <AC Adapter> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
acpi_lid0: <Control Method Lid Switch> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model Synaptics Touchpad, device ID 0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcefff,0xcf000-0xcffff,0xdf000-0xdffff,0xe0000-0xe17ff on isa0
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
random: unblocking device.
usbus0: 12Mbps Full Speed USB v1.0
fuse-freebsd: version 0.4.4, FUSE ABI 7.8
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
vboxdrv: fAsync=0 offMin=0x1f8 offMax=0x540
hdacc0: <Conexant CX20549 (Venice) HDA CODEC> at cad 0 on hdac0
hdaa0: <Conexant CX20549 (Venice) Audio Function Group> at nid 1 on hdacc0
pcm0: <Conexant CX20549 (Venice) (Analog)> at nid 16 and 18,20 on hdaa0
pcm1: <Conexant CX20549 (Venice) (Front Digital)> at nid 19 on hdaa0
unknown: <Conexant CX20549 (Venice) HDA CODEC Modem Function Group> at nid 2 on hdacc0 (no driver attached)
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 12Mbps Full Speed USB v1.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
usbus4: 480Mbps High Speed USB v2.0
ugen4.1: <Intel> at usbus4
uhub4: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus4
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
ada0 at ata0 bus 0 scbus0 target 0 lun 0
ada0: <ST9320421AS SD13> ATA-8 SATA 2.x device
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <HL-DT-ST DVDRAM GSA-4084N KQ09> Removable CD-ROM SCSI-0 device
cd0: Serial Number KYB6CDE5804
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
ada0: Serial Number 5TJ0DE5P
ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada0: 305245MB (625142448 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad0
SMP: AP CPU #1 Launched!
uhub4: 8 ports with 8 removable, self powered
Enter passphrase for ada0p6: ugen2.2: <STMicroelectronics> at usbus2
GEOM_ELI: Device ada0p6.eli created.
GEOM_ELI: Encryption: AES-CBC 256
GEOM_ELI:  Integrity: HMAC/SHA512
GEOM_ELI:  Crypto: software
Trying to mount root from zfs:zroot/ROOT/default []...
GEOM_ELI: Device ada0p4.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
warning: total configured swap (8388608 pages) exceeds maximum recommended amount (6038448 pages).
warning: increase kern.maxswzone or reduce amount of swap.
WARNING: attempt to domain_add(bluetooth) after domainfinalize()
pid 1148 (vmmouse_detect), uid 0: exited on signal 10 (core dumped)

Backing to main problem, In VMWare using FreeBSD 9.3 I was able to compile this without problems, too. So in the end for get this package, I have did like talsamon suggests:

pkg install www/webkit2-gtk3

Now, with graphical enviroment up, and 1796 packages installed (mostly compiled by myself), I have an list with the packages where people using FreeBSD 10.1 and current ports would have problems compilig:

devel/libftdi/
lang/mono/
mail/thunderbird/
security/gnupg/
security/pinentry/
security/libksba/
www/firefox/
www/libxul/
www/w3m/

For those , the only way for get it (at least for me) is using the pkg install...
From list, mono for example give an nice hint saying port system never have the mostly updated patches, and because it, the option is get the compiled version:

It is typically recommended that BSD users NOT download the standard source tarball from the Mono download page and attempt to build/install it but use their ports system instead. The Mono ports typically contain platform specific patches which have either not been merged upstream or to work around idiosyncrasies in the system.

Source:
http://www.mono-project.com/docs/about-mono/supported-platforms/bsd/
Click to expand...

talsamon,

Found it: It compiles with make install clean -I/usr/local/include/cairo/cairo-gl.h. Why this is so, I don't know.
Click to expand...

I have tested this alternative and got no luck too, then back to compiled version from pkg.

Maybe it works with Poudriere, but at least for me, it doesn't work, I have no idea why can't compile it, and on my try it freeze on same step.

About Gnome3

I have read about missing apllications on:

https://wiki.freebsd.org/Gnome

And for make more close of my old experience on Linux, have installed an Debian on VirtualBox, and then installed the default applications.

Found some other problems in Gnome3 like Settings / System - Date & Time

The changes aren't saving, so after, setup my current location and close it, when I open again, the default is back:

BRT (London, United Kingdom)

This is an bug on Arch Linux too:

https://bbs.archlinux.org/viewtopic.php?id=140862

I gonna test it more, before decide what to do.

Once more time, thank you all!!!
 
You must log in or register to reply here.
Back
Top