high cpu load with sftp-server

When our sftp server has more than 5 simultaneous connections the CPU usage jumps up 50% and rises exponentially with each connection. This is a busy server that sometimes has up to 50 simultaneous connections. Is their a kernel tweak or parameter that can be set to avoid this? We recently upgraded our sftp server to Freebsd 7.1 - but had this issue with 6.2 and 7.0 also. We don't seem to have this problem on a similar server running Centos 5.2.

I should also add...

We have 8 CPUs...

CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (2660.02-MHz K8-class CPU)
usable memory = 4281409536 (4083 MB)
avail memory = 4112084992 (3921 MB)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
For example, this is the result of doing an ls on a directory containing 8,000 files:

Number of connections total time to perform ls command cpu usage per process
1 19 sec 80-99%
2 41 sec 80-99%
3 65 sec 80-99%
4 89 sec 80-99%