This is what I mean too.
Serial console is for direct wire access or serial over LAN.
Serial can give access to BIOS, emergency reboot, etc...
OpenSSH is for remote access, it superceeds telnet and serial console.
OpenSSH is meant for remote access.
Quoting your answer, you are doing WAN: "I've forwarded those TCP ports from outside world to my server"
OK, it technicaly works, but is probably highly insecure.
Don't use serial console over IP unless it gives you very special features, like access to BIOS, emergency reboot, etc ... Everything else can be done with OpenSSH.
The first answer was about serial gateway.
A serial gateway is an SSH server with several serial console clients.
It can be build easily using a Beaglebone or bought on eBay.
If you are doing port redirections overs Internet (insecure WAN), you can only redirect to an SSH port OR to a serial gateway (SSH port).