How to make top/systat constantly running on a spare virtual console?

Bobi B.

Bobi B.

Greetings, All. Is it possible to have top(1) or systat(1) constantly running on a spare virtual console, like ttyv8? In the best case they will also accept keyboard input when virtual console is active, so top/ systat toggles could be changed.

Here is my case. A testing server misbehaved half an hour ago, stopped doing whatever it is usually doing (there is a queue of tasks that left waiting), virtual console login stopped working -- one can press Enter on login: prompt and console will print prompt again, but entering root will make console hang. sshd(8) stopped accepting remote connections. Since machine is forwarding log events to a remote server I can see it generates couple of kernel: pid XXX (getty), uid 0: exited on signal 11 messages every 10 seconds; basically an hour ago processes started persistently dumping core (including cron), but nothing else interesting in syslog. I would like to have a diag tool left running, so I'll be able to analyse it if it happens again.

Can you suggest any other tool that might come in handy, or another way to approach such a problem? Thank you!
 
Bobi B.

This sounds to me like it could be a SIGSEGV signal(3). I would make sure that all running processes have (correct) read/write memory access. Also, if your server starts failing after a while (to me) that also indicates a memory problem. What are you running on that server? You've probably seen this already.
 
This is a server-grade Supermicro machine running in a conditioned room, couple of years old, being fine so far. It is running fine right now, as well (restarted it yesterday). Machine is working as a testing IPTV server, doing some live & offline transcoding, some recording & streaming, basically a mix between testing and staging machine. Mostly constant I/O load, low & occasional higher CPU load.

Having said this, I suspect it is more of a software problem, hence I'm looking into ways to locate where the problem is, if it happens to manifest itself again. I would hate this to happen on a production machine. And if it is a my mistake I prefer to find this now, so I can fix it.
 
This is maybe a silly suggestion, but it happened to me more than a few times... have you checked the disk is not full ? ;) ... i will look into you original question shortly
 
Nicola Mingotti said:
This is maybe a silly suggestion, but it happened to me more than a few times... have you checked the disk is not full ? ;) ... i will look into you original question shortly
Click to expand...
Nope; rudimentary monitoring were still working, showing neither of partitions/filesystems were full. However this would be visible in the syslog.

There were also no excess CPU or memory utilisation.
 
Bobi B. to your original question here are my considerations

-] if you don't require interaction with "top" (or systat) you could
simply forward their output on the local network. like in the code below.

-] if you require interaction with "top" etc. i guess the best is option
is to keep a "ssh" connection to the controlled machines always open.
It must be possible to tap into the "/dev/pts/X" connected
to ssh. I remember i saw something similar in the Expect book, but i can't find the book !
... I tried to "tap into" with "cu" directly, but ok, too much rudimentary ... if i find a clean solution to this I will tell you, it is an interesting thing.

Code: 
------------ client ---------------
#!/usr/bin/env ruby

require 'socket'
host = 'localhost'
port = '12345'

TCPSocket.open(host, port) do |s|
  while line = s.gets
    puts line.chop
  end
end
-----------------------------------------

Code: 
----------- server ----------------
#!/usr/bin/env ruby

require 'socket'

server = TCPServer.open(12345)
loop {
  client = server.accept
  out = `top -n`
  client.puts(out)
  client.close
}
---------------------------------------
 
did you check the memory of the server? It's smell like a bad memory or CPU problem (overheating/unstable voltage PSU problems)

If the getty is terminated then also the program that you run at the tty will be terminated but you can try this:

gettytab(5)

Edit your/etc/gettytab and add on the bottom the following line

Code: 
MyApp|console|My application:\
    :ht:np:sp#9600:lo=/usr/bin/systat

edit your /etc/ttys and change some of the ttyv to the following for example:

Code: 
ttyv1 "/usr/libexec/getty console" cons25 on secure

When you switch to console 1 (ctrl+alt+f2) and type any name without password it will start the systat. If the program (systat) exit it will show you login prompt again.

The credits goes to Matthew
https://lists.freebsd.org/pipermail/freebsd-questions/2004-March/042040.html

ps.
maybe it will be easy just to loging on several consoles and leave the programs running on them.
 
VladiBG said:
did you check the memory of the server? It's smell like a bad memory or CPU problem (overheating/unstable voltage PSU problems)
Click to expand...
Not checked yet, however this is not a new machine and everything is running fine after restart, following that weird freeze.

VladiBG said:
If the getty is terminated then also the program that you run at the tty will be terminated but you can try this:

gettytab(5)

Edit your/etc/gettytab and add on the bottom the following line

Code: 
MyApp|console|My application:\
    :ht:np:sp#9600:lo=/usr/bin/systat

edit your /etc/ttys and change some of the ttyv to the following for example:

Code: 
ttyv1 "/usr/libexec/getty console" cons25 on secure

When you switch to console 1 (ctrl+alt+f2) and type any name without password it will start the systat. If the program (systat) exit it will show you login prompt again.

The credits goes to Matthew
https://lists.freebsd.org/pipermail/freebsd-questions/2004-March/042040.html
Click to expand...
I'll definitely dig into this, thanks!

VladiBG said:
ps.
maybe it will be easy just to loging on several consoles and leave the programs running on them.
Click to expand...
I thought of it, as well. Will probably just log-in and leave few utilities running all the time. But were looking for a way to automate this, if possible.
 
Nicola Mingotti said:
Bobi B. to your original question here are my considerations

-] if you don't require interaction with "top" (or systat) you could
simply forward their output on the local network. like in the code below.

-] if you require interaction with "top" etc. i guess the best is option
is to keep a "ssh" connection to the controlled machines always open.
It must be possible to tap into the "/dev/pts/X" connected
to ssh. I remember i saw something similar in the Expect book, but i can't find the book !
... I tried to "tap into" with "cu" directly, but ok, too much rudimentary ... if i find a clean solution to this I will tell you, it is an interesting thing.

Code: 
------------ client ---------------
#!/usr/bin/env ruby

require 'socket'
host = 'localhost'
port = '12345'

TCPSocket.open(host, port) do |s|
  while line = s.gets
    puts line.chop
  end
end
-----------------------------------------

Code: 
----------- server ----------------
#!/usr/bin/env ruby

require 'socket'

server = TCPServer.open(12345)
loop {
  client = server.accept
  out = `top -n`
  client.puts(out)
  client.close
}
---------------------------------------
Click to expand...
We've a small, "proprietary" monitoring agent running on those servers providing some metrics via HTTP. I could see the machine "looks" fine, but not much under the hood.

top >/dev/ttyv8 </dev/ttyv8 works; not sure this is the right way, tho; top is still owned by the TTY it have been started under (i.e. ssh session). Is there a utility to start a program on another TTY?
 
Bobi B. said:
top >/dev/ttyv8 </dev/ttyv8 works; not sure this is the right way, tho; top is still owned by the TTY it have been started under (i.e. ssh session). Is there a utility to start a program on another TTY?
Click to expand...

Reading man pages i found only that a new pts should be created with mknod . Then probabily we can use the strategy you say: top >/dev/pts/X </dev/pts/X ... But i still i did not check well; i have found another way.

The solution i propose you is using the tool screen, wich was introduced to me by aragats a few months ago. Actually i never used it because its shortcuts interfere with my "emacs" / "mg" habits, but in this case it seems a good solution.

So, I write you a procedure wich can of course be automatized a lot, but i guess you can do it as well as i can;) So here is the bare-bone

---------------------------------------------------------------------------------------------
1] let's call C your desktop and S the server you want to monitor

2] You will be typing in C; S is remote and considered not nearby, but it has
sshd active and you can log into it. Indeed, it is supposed you start this
monitoring procedure before the server S goes critical and forbids you to login.

3] Open an xterm and type screen
Now you will see a shell, but this sheel is under the control of "screen".
foo@client> ssh bar@server
bar@server> top

4] Now press <C-a C-c> this will open a new window into "screen", with a new shell
foot@client> ssh bar@server
bar@server> systat

5] Now press <C-a d> this will detach "screen" from your current xterm.
You can also kill the "xterm" after it. This is important because if we
would have liked to keep many "xterm" always open for monitoring there
would have been no need to implement this solution.

6] Open a new xterm

7] "screen" should be still alive, look for its PID (i write precedure for completeness
i know this is knid of too basics to say;) )
foo@client> ps aux | grep screen
In my case the pid is 23529

8] Connect "screen" program to the old instance of "screen" you run before
foo@client> screen -r 23529

9] You will see now "top" or "stat" running in S

10] You can change the view from "top" to "stat" pressing <C-a "> and selecting
the appropriate window in the list it appears.

11] To quit the "screen" press <C-a d>

12] You can kill "screen" in the usual way kill 23529
--------------------------------------------------------------------------------------

In this way you are always monitoring "S" and you are always logged in there.

... of course ssh may fall; but this is another story;)
 
Nicola Mingotti said:
Reading man pages i found only that a new pts should be created with mknod.
Click to expand...
That's not entirely correct. While creating a new /dev entry with mknod will work, your changes won't retain after a reboot:
Code: 
peter@zefiris:/home/peter $ mount | grep dev
devfs on /dev (devfs, local, multilabel)
/dev is usually mounted as a virtual file system which means that any changes you make won't retain and will get reset right after a reboot. That's because specific devfs rules are followed when populating the filesystem, see /etc/defaults/devfs.rules.

Another issue is that the system is managed through devd(8) which can dynamically create (or remove) devices.

So although mknod will work for that one session, you really want to check either devfs(5) or devd.conf(5) if you want to add device entries.
 
Today I stumbled on man 8 watch and I remembered this thread.
I think this tool solves nicely Bobi B. original question.

Procedure:

1] I am typing in a xterm (konsole)
$> tty
/dev/pts/2

2] I want to run top in a virtual terminal, let's say ttyv3 so
$> sudo watch -W /dev/ttyv3
[do the login]
$> tty

/dev/ttyv3
$> top

3] I quit the watch command pressing Ctrl+G

4] I check where i am
$> tty
/dev/pts/2

5] Now i go to see the virtual terminal pressing Ctrl+Alt+F4
=> I see top running

6] I go back to XWindows Ctrl+Alt+F9
[attention, here i must wait about 15 seconds till the XWindows becomes fully operative]

bye
n.
 
Thank You Bobi B. for posting this question.
Similarly in my mind I was thinking about my 8 unused virtual terminals
Opening a separate program in each.
I really liked the idea of ytree on one virtual terminal and pftop in another virtual terminal.
With each console requiring a login this does become way more complicated.
To a point that I think Tmux instances with each command would be better.
Tmux instances for bhyve has worked out superbly for me. Clt-B and W is nice.
 
You must log in or register to reply here.
Back
Top