SSH client always using port 10000

ronaldlees

ronaldlees

I suddenly have a problem where the local ssh clients are always using local port 10000 when connecting to remote sshd servers (remote servers are all on port 22). What might that be?
 
derekschrock said:
Look for Port in /etc/ssh/config or ~/.ssh/config
Check to see if ssh is an alias or shell function type ssh
Click to expand...

I see no "Port" specified in the config files, and type ssh =/usr/sbin/ssh.

What's odd is that I have two different computers doing the exact same thing.
 
Out of the blue, it's random again (random local ports). So, the "-vv" output probably won't show us what was wrong. What is particularly strange is that this happened to two different machines (one Linux, one BSD) - and was consistent over the past day or so (a good number of connections too).
 
SirDice said:
Port is to specify a destination port, not a source port. As far as I know ssh(1) simply uses whatever random port the OS gives it. What does sysctl net.inet.ip.portrange output?
Click to expand...

Code: 
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
net.inet.ip.portrange.randomized: 1
net.inet.ip.portrange.reservedlow: 0
net.inet.ip.portrange.reservedhigh: 1023
net.inet.ip.portrange.hilast: 65535
net.inet.ip.portrange.hifirst: 49152
net.inet.ip.portrange.last: 65535
net.inet.ip.portrange.first: 10000
net.inet.ip.portrange.lowlast: 600
net.inet.ip.portrange.lowfirst: 1023

SirDice OK, I see the 10000 now. I would expect the port would have been randomized within the range. It definitely was not randomized throughout the day yesterday.
 
Yes, that at least explains the port 10000. But you are right, I would have expected the ports to be more random too. If you make two ssh(1) connections in short succession do they both use source port 10000?
 
SirDice said:
Yes, that at least explains the port 10000. But you are right, I would have expected the ports to be more random too. If you make two ssh(1) connections in short succession do they both use source port 10000?
Click to expand...

Indeed - I saw that happen a couple times. At least I think I did. You know how it is, when you're first looking at something like this, and then you've cleared it off, and later ask yourself "Did I really see that?"
 
ronaldlees said:
Indeed - I saw that happen a couple times. At least I think I did. You know how it is, when you're first looking at something like this, and then you've cleared it off, and later ask yourself "Did I really see that?"
Click to expand...

One possible explanation is that you already have many outgoing connections that have reserved many ports and the randomization algorithm doesn't find a suitable source port and it falls back to the first one available.
 
kpa said:
One possible explanation is that you already have many outgoing connections that have reserved many ports and the randomization algorithm doesn't find a suitable source port and it falls back to the first one available.
Click to expand...

I can't say that I had sufficient connections to imagine the supply could have been exhausted :)

The paranoid in me thinks about what a static source port might facilitate (like a tunnel back to me via an existing outgoing connection). But, I didn't see any evidence of anything like that in the configuration on my machine, so no reason to think so.

It hasn't happened yet today AFAIK. It's a mystery.
 
Perhaps a randomless random generator hitting a state which always returns 0 (the first entry in the TCP Port list)? If so, this seems to go back to normal after rebooting, or cause an event that regenerates the random generator seed. I've seen this kind of 'stuck' behavior while implement my own random number generator algorythm.
 
ronaldlees said:
The paranoid in me thinks about what a static source port might facilitate (like a tunnel back to me via an existing outgoing connection). But, I didn't see any evidence of anything like that in the configuration on my machine, so no reason to think so.
Click to expand...

With the exception of a man-in-the-middle scenario, predictable ports should not make a tunnel to a different endpoint possible, as while they might be able to spoof an incoming packet, the outgoing packets won't reach them. Note that they need to successfully predict not just the port, but also the TCP sequence numbers (which should start from a random point). They would also face the near impossible challenge of actually breaking the SSH crypto protections which should resist even someone with true man-in-the-middle access to your TCP stream (as long as you always verify remote keys).

Personally, I wouldn't lose a great deal of sleep over it, unless I saw some other odd behaviour that seemed to relate to it. Just check that you are up to date with security fixes, and chalk it up to one of those occasional random oddities.
 
fossette said:
Perhaps a randomless random generator hitting a state which always returns 0 (the first entry in the TCP Port list)? If so, this seems to go back to normal after rebooting, or cause an event that regenerates the random generator seed. I've seen this kind of 'stuck' behavior while implement my own random number generator algorythm.
Click to expand...

Yikes! I would hope the RNG is not that bad! I did run dieharder (in ports as math/dieharder ) - and it gave me a decent evaluation on the random number generator (although the author indicates that it's a non-exhaustive evaluation). Given the disclaimer, does anyone know of similar tools I could use to test the RNG, to get a second opinion?
 
Write a program that creates a large number of sockets and writes down the source ports the OS assigned to the sockets with its randomizer.
 
Murph said:
Note that they need to successfully predict not just the port, but also the TCP sequence numbers (which should start from a random point).
Click to expand...
Tricky to do indeed. Just contemplating here, but suppose the OPs random generator is somehow borked and it's causing source ports to not be random wouldn't that same lack of randomness affect the sequence numbers too? If one supposed random number isn't random, the others won't be either?
 
kpa said:
Write a program that creates a large number of sockets and writes down the source ports the OS assigned to the sockets with its randomizer.
Click to expand...

Good idea. Do we even know whether or not the system RNG is used with port number allocation?
 
SirDice said:
Tricky to do indeed. Just contemplating here, but suppose the OPs random generator is somehow borked and it's causing source ports to not be random wouldn't that same lack of randomness affect the sequence numbers too? If one supposed random number isn't random, the others won't be either?
Click to expand...

I find it hard to believe that the RNG would do this, so I'm leaning towards your suggestion that "the OPs random generator is somehow borked" - probably via exploit or something, and that I should do a drop-kick on my current instances, reinstall the OS, and start over.

Code: 
$ sysctl kern.random
kern.random.sys.harvest.swi: 1
kern.random.sys.harvest.interrupt: 1
kern.random.sys.harvest.point_to_point: 1
kern.random.sys.harvest.ethernet: 1
kern.random.sys.seeded: 1
kern.random.yarrow.slowoverthresh: 2
kern.random.yarrow.slowthresh: 128
kern.random.yarrow.fastthresh: 96
kern.random.yarrow.bins: 10
kern.random.yarrow.gengateinterval: 10
kern.random.live_entropy_sources:
kern.random.active_adaptor: yarrow
kern.random.adaptors: yarrow,dummy
Code: 
net.inet.ip.portrange.randomtime: 45
net.inet.ip.portrange.randomcps: 10
What's the implication of the last two lines (from net.inet.ip.portrange?)
 
ronaldlees said:
What's the implication of the last two lines (from net.inet.ip.portrange?)
Click to expand...

On 10.3:
Code: 
# sysctl -d net.inet.ip.portrange
net.inet.ip.portrange: IP Ports
net.inet.ip.portrange.randomtime: Minimum time to keep sequental port allocation before switching to a random one
net.inet.ip.portrange.randomcps: Maximum number of random port allocations before switching to a sequental one
net.inet.ip.portrange.randomized: Enable random port allocation
net.inet.ip.portrange.reservedlow: 
net.inet.ip.portrange.reservedhigh: 
net.inet.ip.portrange.hilast: 
net.inet.ip.portrange.hifirst: 
net.inet.ip.portrange.last: 
net.inet.ip.portrange.first: 
net.inet.ip.portrange.lowlast: 
net.inet.ip.portrange.lowfirst:

Those descriptions suggest to me that it flips back and forth between sequential and random, presumably for performance reasons (random is expensive, compared to sequential). I would not rush to any conclusions about the need to burn your system to the ground, certainly not without much more compelling evidence of a compromise. I don't know the precise details underneath that behaviour, but the above certainly could explain something like you are seeing, and sequential could quite possibly result in the same port number appearing repeatedly in this sequence {open, close, open, close, …}, and an incrementing port number in {open, open, open, …, close, close, close}.
 
Ahh, here we are it is documented in TFM, ip(4):
Ports are allocated at random within the specified port range in order to
increase the difficulty of random spoofing attacks. In scenarios such as
benchmarking, this behavior may be undesirable. In these cases,
net.inet.ip.portrange.randomized can be used to toggle randomization off.
If more than net.inet.ip.portrange.randomcps ports have been allocated in
the last second, then return to sequential port allocation. Return to
random allocation only once the current port allocation rate drops below
net.inet.ip.portrange.randomcps for at least
net.inet.ip.portrange.randomtime seconds. The default values for
net.inet.ip.portrange.randomcps and net.inet.ip.portrange.randomtime are
10 port allocations per second and 45 seconds correspondingly.
Click to expand...
 
I wonder why UDP doesn't prefer more randomized source port selection, it is much more vulnerable to forgery *) by an active attacker than TCP which has as noted the sequence numbers as extra protection.


*) For example DNS used to use fixed source port 53 also for any outgoing queries, this was hopelessly insecure given how the DNS protocol works.
 
kpa said:
I wonder why UDP doesn't prefer more randomized source port selection, it is much more vulnerable to forgery *) by an active attacker than TCP which has as noted the sequence numbers as extra protection.


*) For example DNS used to use fixed source port 53 also for any outgoing queries, this was hopelessly insecure given how the DNS protocol works.
Click to expand...

Of course the random UDP source port applies only to UDP based protocols that have been designed in such a way or have been fixed to work that way. It's common especially on game protocols that use UDP as transport to just use a single UDP port for both directions, ingoing and outgoing. This is because it's much easier to program a game server with just a single UDP listening port. For the same reason we still have the NTP protocol that uses only one UDP listening port 123 regardless of direction.
 
kpa said:
For the same reason we still have the NTP protocol that uses only one UDP listening port 123 regardless of direction.
Click to expand...

Only the NTP server port number is fixed. The protocol (v3 and v4, at least) and reference implementation handles random client port numbers just fine, same as DNS. My public stratum 2 server sees around half of its clients using random source ports, currently 5,701 using port 123 out of 10,460 clients, with the random ports mostly being evenly spread across the full range other than a spike of 153 using port 1024.

Games which are strictly client<-->server have no need to use a fixed client port. Where they also have peer to peer comms, they often use fixed UDP ports as a way to pierce NAT and stateful firewalls. If 2 peers both start a UDP conversation simultaneously (having obtained the remote peer's address from a central server), that will open sufficient NAT and firewall states at both ends to allow them to talk to each other when random inbound UDP would normally be blocked or dropped.
 
Murph said:
My public stratum 2 server sees around half of its clients using random source ports, currently 5,701 using port 123 out of 10,460 clients, with the random ports mostly being evenly spread across the full range other than a spike of 153 using port 1024.
Click to expand...

Oops, wrong query used there, that was only the more active recent clients (minimum 50 packets). Looking at all recent clients, 8,019 out of 42,091 use port 123, with the remainder spread fairly evenly around the range (and spikes of 273 on port 1024, 72 on 32769, 56 on 2052, and 42 on 1025; with smaller spikes giving strong evidence of sequential port allocation starting at bases of 1024, 2048, 3072, 32768, and 49152; and evidence of a subset of the population restricting their random port choice to the official high range of 49152–65535 and others to the upper half of 32768–65535).
 
You must log in or register to reply here.
Back
Top