ASRock J4205-ITX / Intel Apollo Lake

Greetings folks!

As this is my third post in this forum, i would like to take the chance to leave my compliments for this great community. You are very helpful and I hope you continue to be! :)

Pre-story:

I bought a ASRock J4205-ITX which is based on Intels "new" Apollo Lake architecture. The reason why i choose this system is because of its power consumption. It takes only 19 watts with two spinning HDDs. Imho this is a perfect base for a "green" private 24/7 Homeserver. Although I am a Linux professional, I chose FreeBSD as the operating system for two reasons: first, I think ZFS mirroring is the best choice for keeping my data safe in relation to integrity an redundancy, and FBSD seems to offer the best balance between technical integration and license compatibility. Second, I would like to look over the edge in relation to other Unix-like systems.

My configuration:
Code:
FreeBSD 11.0-RELEASE-p7 #0: Fri Jan 27 21:00:51 CET 2017
root@FreeBSD01:/usr/obj/usr/src/sys/GENERIC amd64
FreeBSD clang version 3.8.0 (tags/RELEASE_380/final 262564) (based on LLVM 3.8.0)

CPU: Intel(R) Pentium(R) CPU J4205 @ 1.50GHz (1497.66-MHz K8-class CPU)

  Origin="GenuineIntel"  Id=0x506c9  Family=0x6  Model=0x5c  Stepping=9

  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>

  Features2=0x4ff8ebbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,RDRAND>

  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>

  AMD Features2=0x101<LAHF,Prefetch>

  Structured Extended Features=0x2294e283<FSGSBASE,TSCADJ,SMEP,ERMS,NFPUSG,MPX,PQE,RDSEED,SMAP,CLFLUSHOPT,PROCTRACE,SHA>

  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>

  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr

  TSC: P-state invariant, performance statistics

real memory  = 8589934592 (8192 MB)

My issues:

After installing FBSD 11 from scratch I noticed three major problems:

1.: System time skews rapidly. The system is not able to keep its time in sync when using its "best quality" time counter TSC. I noticed after several hours of running a time difference of about 150+ seconds when setting it by ntpdate at system start.
2.: System response is very very slow. Friends, I know this is not the fastest machine on the market, and in fact this is a "low cost" category, but I cannot accept that a machine with 4 CPU cores even lags by typing commands into the shell. :eek:
3.: The network connection tears off when traffic goes up to high loads. This seems to be a general problem with kernels standard Realtek-NIC driver: keywords "re0 watchdog interrupts" which has already been discussed in the network-area.

Searching solutions:

To point 1: time keeps in sync when switching the counter
kern.timecounter.hardware from TSC to ACPI-fast:

Code:
kern.timecounter.choice: TSC(1000) ACPI-fast(900) HPET(950) i8254(0) dummy(-1000000)
kern.timecounter.hardware: ACPI-fast
I can live with that "solution" by placing "kern.timecounter.hardware=ACPI-fast" into the sysctl.conf

To point 2: after the (good) experience with stepping back in the quality line of the kernels chosen options, i tried to use the HPET-timer instead of default LAPIC

Code:
kern.eventtimer.timer: HPET
kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) HPET3(440) HPET4(440) i8254(100) RTC(0)
Unfortunately, this has not changed a bit. Response times of the system are still bad at all. :confused:
To be constructive: seemingly there was a similar problem in Linux-Kernel which is described here: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1606147. Is there someone out there who can check this issue in relation to the FBSD-Kernel? :)

To point 3: The problem is resolved by compiling Realtek's latest driver as a module into the kernel. I have issued some heavy load-tests, and it works pretty good.

Footer:
The easiest solution might be changing the OS, but giving up is not an option for me. And put your hand on your heart: this is not fun. ;)

Maybe there are other people out there who have the same problems, or perhaps even solutions?

Best regards,
Marc
 
Last edited by a moderator:
Hi Marc,

I have the same issue regarding very poor performance with the J4205-ITX. The status of my current tests are:

1) poor performance is related to energy management
2) Intel Speed step has no influence on performance
3) C-state have the most major influence

I did several tests on meassuring the computing time of "freebsd-update fetch", results:
1) C-States up to C3 -> time: about 27minutes
2) C-States up to C1 -> time: about 7 minutes
3) C-States off: time: 40 seconds

I think the OS changes to fast to C-States below C0. But I didn't found the switch where I can control the idle-time that causes to change to C1, C2 or C3.

My current setting is to disable C-States in mainboard setup until I found the right config for freebsd. With turned off C-States the system ist really fast,
no delay during typing commands in local console.

I have to check your issues about network problems and time skews.

Can anyone else help to find a good solution?

Thanks,

Reinhard
 
Thank you Reinhard for your detailed description!

Good to see that I am not alone with this.

As of your findings I checked the behavior directly on my system: the result is that a simple


sysctl hw.acpi.cpu.cx_lowest=C1


already improved the response time noticeable! o_O

To take this thought further, I deactivated powerd directly and checked the response time again. To my surprise, the opposite of my expectation occurred: the response time turned out significantly worse.

According to the factual situation I totally agree with you that power management is the cause.

Other/more ideas will be appreciated! :)
 
After some further tests with good old "ubench" (running several times each) I got the following results:


ubench -t 20 (@sysctl hw.acpi.cpu.cx_lowest=C3)


Code:
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>

http://www.phystech.com/download/ubench.html

FreeBSD 11.0-RELEASE-p7 FreeBSD 11.0-RELEASE-p7 #0: Fri Jan 27 21:00:51 CET 2017     root@FreeBSD01:/usr/obj/usr/src/sys/GENERIC amd64

Ubench CPU:   640439
Ubench MEM:   681781
--------------------
Ubench AVG:   661110



ubench -t 20 (@sysctl hw.acpi.cpu.cx_lowest=C1)


Code:
Unix Benchmark Utility v.0.3
Copyright (C) July, 1999 PhysTech, Inc.
Author: Sergei Viznyuk <sv@phystech.com>

http://www.phystech.com/download/ubench.html

FreeBSD 11.0-RELEASE-p7 FreeBSD 11.0-RELEASE-p7 #0: Fri Jan 27 21:00:51 CET 2017     root@FreeBSD01:/usr/obj/usr/src/sys/GENERIC amd64

Ubench CPU:  1126752
Ubench MEM:   658215
--------------------
Ubench AVG:   892483
This is a substantial increase of CPU performance only by limiting C-States!

Which is also interesting: When powerd is running in foreground verbose (powerd -v) it seems to doesn't really respond to the load situation when sysctl hw.acpi.cpu.cx_lowest is C3. Current freq stays @ 800 MHz constantly.

After changing hw.acpi.cpu.cx_lowest to C1, powerd adjust the clock speed much faster (from 800 Mhz to 1501 MHz). This seems to be a vicious circle...

Tomorrow im going to install my power meter again to measure the influence of the settings to the systems power consumption.
 
Hi Marc,

as far as I found out is that powerd only adjusts the CPU frequency. C-states are handled by the kernel. The most influence is disabling or
limiting C-states. But I do not really understand the mechanism.

- does the kernel change too fast to Cx? e.g. not taking CPU load correct into account?
- does CPU always have same c-state on all cores? usage of c-states as reported by sysctl does not confirm that
- how long is CPU/core in sleep state? What trigger causes wake-up? Timer (hardware, software)?

lets continue to find out the root cause and solution :)
 
Sorry, I was a little bit busy for the weekend. :)

My findings:

After some tests with my power meter I can say that the difference in power consumption between sysctl hw.acpi.cpu.cx_lowest=C1 -> 18.0 watts and sysctl hw.acpi.cpu.cx_lowest=C3 17.7 is only 0.3 watts @ASRock J4205-ITX. The thermal difference in CPU temperature is therefore practically non-existent ( dev.cpu.0.temperature: 39.0C).

When c-state is set to default (C3), sysctl -a | grep cx_ says

hw.acpi.cpu.cx_lowest: C3
dev.cpu.3.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.3.cx_usage_counters: 0 0 1632

dev.cpu.3.cx_usage: 0.00% 0.00% 100.00% last 46609us
dev.cpu.3.cx_lowest: C3
dev.cpu.3.cx_supported: C1/1/1 C2/2/50 C3/3/150
dev.cpu.2.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.2.cx_usage_counters: 7 9 1823

dev.cpu.2.cx_usage: 0.38% 0.48% 99.12% last 4711us
dev.cpu.2.cx_lowest: C3
dev.cpu.2.cx_supported: C1/1/1 C2/2/50 C3/3/150
dev.cpu.1.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.1.cx_usage_counters: 0 0 1256

dev.cpu.1.cx_usage: 0.00% 0.00% 100.00% last 76713us
dev.cpu.1.cx_lowest: C3
dev.cpu.1.cx_supported: C1/1/1 C2/2/50 C3/3/150
dev.cpu.0.cx_method: C1/mwait/hwc C2/mwait/hwc C3/mwait/hwc
dev.cpu.0.cx_usage_counters: 0 0 3389

dev.cpu.0.cx_usage: 0.00% 0.00% 100.00% last 39516us
dev.cpu.0.cx_lowest: C3
dev.cpu.0.cx_supported: C1/1/1 C2/2/50 C3/3/150


You can see clearly that the kernel sends the CPU (cores) down to C3 asap or perhaps unsuitably fast. The "wakeup" then takes too much time, so the system response is very sluggish. And this is even the reason why powerd cannot adjust the clock speed in appropriate time, because powerd is only another cpu-dependent process.

If powerd is now killed, the problem increases because the cores stays at its minimum c-state and at its down-trimmed frequency (800 MHz). o_O

My personal conclusion is that for the moment all apollo lake-systems should run with sysctl hw.acpi.cpu.cx_lowest=C1 in sysctl.conf for as long as this behavior has not yet been finally analyzed at kernel level. This leaves two logical possibilities: either there is a bug in the kernel itself or the hardware is not capable to execute the transitions between the c-states in an acceptable time.

Are there kernel-hackers present? :rolleyes:

Best regards,
Marc
 
You probably need to run -CURRENT and you probably also need to run powerdxx instead of powerd. There are snapshots available if you want to try, keep in mind that they have lots of debugging enabled by default. Also, this is one of many reasons why you want to avoid Realtek NICs.
 
Although It's not the exact same board but I can confirm that I get the same results on a J3455B-ITX (little bit slower CPU) otherwise the same Apollo Lake SoC + other findings:
  • 11-RELEASE is really unresponsive
  • 11-STABLE snapshots behaved the same as 11-R but booted in contrast to 12-CURRENT
  • 12-CURRENT snapshots as of 20170210 hang at "Timecount "HPET" frequency 19200000 Hz quality 950" and I gave up waiting after 10 minutes.
Where would be the best place to report such findings?

I currently switched "back" to Debian unstable on it since the Linux kernel in stable/jessie is also not behaving correctly (unstable, bit sluggish) however modern Linux kernels do run pretty fine (4.9). Linux also needed some time to get these SoCs properly supported. With a passive power supply I could get just below 10W using one 2.5" SSD. The fact that these boards have 4 cores, support VT and happily accept 16GB RAM makes them pretty nice for 24/7 lab stuff when ECC is not absolutely required.
 
So, and here we go...

My new kernel is now on duty and i can report the following observations. Before booting the patched kernel, I have commented my "workaround settings" in sysctl.conf, so the kernel may run whit its default settings:

Code:
#kern.eventtimer.timer=HPET
#kern.timecounter.hardware=HPET
#hw.acpi.cpu.cx_lowest=C1


After booting the patched kernel, the system time stays in sync with my ntp-timeserver. As far as the delays on the console are concerned, I also notice an improvement. This is a noticeable progress.

Unfortunately ubench shows the same cpu performance reduction when hw.acpi.cpu.cx_lowest is at its self-chosen state, which is now surprisingly C2 instead of C3. So i will return to hw.acpi.cpu.cx_lowest=C1 in sysctl.conf.
 
I bought ASRock J3455M
disabling C-States on bios make system running fast
but my Wi-Fi card reject to work
tunning kern.timecounter.hardware and kern.eventtimer.timer not help
it shows only ath0: device timeout in log

I think problem is really on timers, when I run this system in Dom0 XEN, wi-fi starts to work

FreeBSD Hitahca 11.0-RELEASE-p8 FreeBSD 11.0-RELEASE-p8 #0: Wed Feb 22 06:12:04 UTC 2017 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64
CPU: Intel(R) Celeron(R) CPU J3455 @ 1.50GHz (1497.60-MHz K8-class CPU)
ath0: <Atheros 9287> mem 0x91000000-0x9100ffff irq 21 at device 0.0 on pci4
ath0: [HT] enabling HT modes
ath0: [HT] enabling short-GI in 20MHz mode
ath0: [HT] 1 stream STBC receive enabled
ath0: [HT] 1 stream STBC transmit enabled
ath0: [HT] 2 RX streams; 2 TX streams
ath0: AR9287 mac 384.2 RF5133 phy 15.15
ath0: 2GHz radio: 0x0000; 5GHz radio: 0x00c0

kern.timecounter.hardware: XENTIMER
kern.eventtimer.timer: XENTIMER


I was try to rebuild kernel/word in STABLE but got compile errors
 
My situation.
ASRock: J3455B-ITX, the latest BIOS (as on 07.2017) BIOS/UEFI v. P1.20.
J3455 it is also Intel Apollo Lake (read member msi, Feb 26, 2017).

C-States must be disabled in BIOS.

1)
11.0-RELEASE-p0 and p10 - OK
11.1-RELEASE-p0 - hangs at Timecount HPET
In verbose mode boot - detailed info:
...
routing MSI-X IRQ 263 to local APIC 4 vector 49
Assigning MSI-X IRQ 256 to local APIC 0 vector 52
... stop

Any combination (16 in total) of:
kern.timecounter - HPET, ACPI-fast, i8254, TSC
kern.eventtimer - LAPIC, HPET, i8254, RTC
did not work.

Turn off HPET table in BIOS and kern.timecounter/kern.eventtimer LAPIC/ACPI-fast: no effect.

2)
I followed the message about MSI / MSIX.
I completely forbid all instances of msix eg.:
hw.ixl.enable_msix = 0
hw.ix.enable_msix = 0
hw.em.enable_msix = 0
hw.igp.enable_msix = 0
hw.pci.enable_msix = 0
machdep.disable_msix_migration = 1

11.1 - system hangs at Timecount HPET


3)
Incidentally - I connected the ethernet cable (so far: I did the tests without the network).

11.1 - it works OK :)


4)
I have investigated which settings determine good startup.
The only setting is:
machdep.disable_msix_migration = 1
https://reviews.freebsd.org/D6947
https://lists.freebsd.org/pipermail/freebsd-stable/2016-August/085186.html

Of course, it must be connected to the network - just for a moment after the start.

Probably a bigger problem with the acpi ALASKA AMI chipset.


Maybe my observations will help someone solve the problem.
Maybe some hints?
Where can I report this problem?

Best regards,
zjk
 
  • Thanks
Reactions: Oko
Has anyone been able to solve the problems with Apollo Lake /Asrock j4205/3455?
Because i am sitting in front of one and it is driving me nuts.
 
For FreeBSD 11.1:

With 11.1 FreeBSD: does not work UEFI boot (with sc or vt console). You need to use the "old style" freebsd-boot - that is in the computer BIOS: turn off UEFI, enable CMS, add a freebsd-boot partition scheme to your disk.

1. Disable C-states in BIOS.
2. Add: machdep.disable_msix_migration = 1 - in /boot/loader.conf
3. Attach network

It's not a solution - it's a workaround - but my 3455 machines are working fine as a NASes for a few days.

zjk
 
Last edited:
Seems like the whole issue is related to some ACPI stuff. I am using gigabyte GA-J3455N-D3H board, but having same issues - system hangs at Timecount HPET when I'm trying to boot up FreeBSD 11.1 from USB flash drive.

Is there someone who's working with this issue on kernel-level? I'd like to check out any available diff's and patch the kernel by myself. I don't really like "workaround" solutions...

Regards, Xdevelnet

P.S. By the way, I haven't any success with linux, o_O. I heard that something was done in 4.8 linux kernel to fix this issue, but I can't find "what exactly" was changed. I just wondering if I can repeat that success with FreeBSD.
 
For "ready" disk images or USB drives: version 11.1 does not work UEFI boot (with sc or vt console). You need to use the "old style" freebsd-boot - that is in the computer BIOS: turn off UEFI, enable CMS, add a freebsd-boot partition scheme to your disk.. However, I do not know if it concerns GPT or MBR, because I only use GPT.

On the other hand - I do not see the special impact of the settings that allow the 11.1 system to boot, to its performance. I had problems with re0, but it was enough to increase the number of threads with nfs server / client.

Instead - because I use the console - there is always a serious problem with video output switching and their resolutions in sc or vt mode. But it's already for another forum thread.

zjk
 
Last edited:
Hi folks,

just want to add another variant of zjks walkaround if it doesn't work for you (as it didn't do for me)

add these to the loader.conf:
Code:
machdep.disable_msix_migration=1
hint.hpet.0.clock=0
hint.hpet.0.per_cpu=0

do the rest as mentioned by zjk above.
 
Hi folks,

just want to add another variant of zjks walkaround if it doesn't work for you (as it didn't do for me)

add these to the loader.conf:
Code:
machdep.disable_msix_integration=1
hint.hpet.0.clock=0
hint.hpet.0.per_cpu=0

do the rest as mentioned by zjk above.

And should not be:
machdep.disable_msix_migration = 1
?
:)
ZJK
 
And should not be:
machdep.disable_msix_migration = 1
?
:)
ZJK

of course, i'll edit. thank you!
btw UEFI does work if hpet is disabled this way (at least with current BIOS firmware)

should this be reported as a problem via freebsd bugtracker?
 
after playing around and boiling it down it seems that just disabling hpet is enough for a stable boot

Code:
hint.hpet.0.clock=0
 
I'll describe it more clearly: UEFI works "differently", it completely blocks cs or vt consoles - sometimes ssh works (computer starts "correctly" - but without consoles).

In any case: hint.hpet.0.clock = 0 does not solve my startup problem (with UEFI).

:(
zjk
 
HPET not need at all
Code:
kern.timecounter.choice: ACPI-fast(900) HPET(950) i8254(0) TSC(1000) dummy(-1000000)
kern.timecounter.hardware: TSC

kern.eventtimer.choice: LAPIC(600) HPET(550) HPET1(440) HPET2(440) HPET3(440) HPET4(440) i8254(100) RTC(0)
kern.eventtimer.timer: LAPIC
 
Back
Top