BASH: Format Monitor Output

[IT_Architect]

I receive a stream from a monitor, but I need to format it to store it in a database to graph it. There are three segments to the data; key, value, and status. The status needs to be translated into a number. (status number to code table shown. below) I have a method that uses two reads, the first one that stores the key + value, and a second that stores key + converted status. I don't know if it's possible, but I'd like to do it with one read, and store key + value + converted status on one line in the output to the file.

Status number to code table
#0 #us: unspecified
#0 #ok: ok
#1 #nc: non-critical
#2 #cr: critical
#3 #nr: non-recoverable

What I receive:

System Temp      | 48 degrees C      | cr
+1.5V            | 1.51 Volts        | ok
+5V              | 5.15 Volts        | nc
+5VSB            | 5.12 Volts        | ok
+12V             | 12.19 Volts       | ok
-12V             | -12.00 Volts      | ok
+3.3V            | 3.26 Volts        | ok

What I have:

v_System_Temp:48
v_+1.5V:1.51
v_+5V:5.15
v_+5VSB:5.12
v_+12V:12.19
v_-12V:-12.00
v_+3.3V:3.26
s_System_Temp:2
s_+1.5V:0
s_+5V:1
s_+5VSB:0
s_+12V:0
s_-12V:0
s_+3.3V:0

What I want:

System_Temp:48:2
+1.5V:1.51:0
+5V:5.15:1
+5VSB:5.12:0
+12V:12.19:0
-12V:-12.00:0
+3.3V:3.26:0

Is this possible?

Thanks!

 

[expl]

What are you using to store the data in database?

I think string manipulation in shell script is very inefficient and complicated. The format you are talking can be achieved with couple of lines of python/perl/ruby and they can be used then to store the data directly to the database instead of using file as a place for temporary holding of the data.

 

[IT_Architect]

I cannot store it directly in the database or I surely would. It polls and needs to pull from a file. I do not know python, perl, or ruby, but I do PHP which uses PCRE in 5.3+.

 

[wblock@]

I receive a stream from a monitor, but I need to format it to store it in a database to graph it. There are three segments to the data; key, value, and status. The status needs to be translated into a number. (status number to code table shown. below) I have a method that uses two reads, the first one that stores the key + value, and a second that stores key + converted status. I don't know if it's possible, but I'd like to do it with one read, and store key + value + converted status on one line in the output to the file.

Please show the code you have at present. What programming languages are allowed? Does "one read" mean one read of the input file, or one read of the input line?

 

[IT_Architect]

Please show the code you have at present. What programming languages are allowed? Does "one read" mean one read of the input file, or one read of the input line?

The monitoring machine has perl and php.

This is what I have now in BASH.

ipmitool -H $1 -U "$2" -P "$3" -L USER sdr | sed 's/|/:/g' | sed 's/[[:space:]]//g'\
 | sed 's/^.*:ns//g' | sed '/^$/d' | sed 's/\(.*\).../\1/' | sed -e 's/\([[:alpha:]]\)*$//g'\
 | sed 's/^/ipmi_v_/g' > /tmp/ipmi_$1.txt

ipmitool -H $1 -U "$2" -P "$3" -L USER sdr | sed 's/|/:/g' | sed 's/[[:space:]]//g'\
 | sed 's/^.*:ns//g' | sed '/^$/d' | sed 's/:.*:/:/g' | sed 's/:us/:0/g' | sed 's/:ok/:0/g'\
 | sed 's/:nc/:1/g' | sed 's/:cr/:2/g' | sed 's/:nr/:3/g' | sed 's/^/ipmi_s_/g' >> \
 /tmp/ipmi_$1.txt

Normally I'm pulling from APIs, they are always key:data, and they supply perl scripts to do that. This one is different because it's key:data:status, which is actually better in a way because then I don't need to set up and maintain rules for what the limits are, so the manufacturer can change them without requiring any maintenance on my part. However, no scripts.

 

[expl]

Here is PCRE regex: ^(\+*\-*[0-9]+\.*[0-9]*[VSB]*|\w+\s*\w+)\s*\|\s*(\+*\-*[0-9]+\.*[0-9]*).*\|\s*(\w+)\s*$

Here is python output for it, where regex is instance of the regex above, where string its your input from section "What I receive:" with new lines.

>>> regex.findall(string)
[(u'System Temp', u'48', u'cr'), 
(u'+1.5V', u'1.51', u'ok'), 
(u'+5V', u'5.15', u'nc'), 
(u'+5VSB', u'5.12', u'ok'), 
(u'+12V', u'12.19', u'ok'), 
(u'-12V', u'-12.00', u'ok'), 
(u'+3.3V', u'3.26', u'ok')]

I get a neat array of lists. Every array member represents a new line entry, every list member represents a field in the table. So all you need to do is convert the 3rd field from string to integer.

I guess this can be ported to PHP :].

 

[IT_Architect]

Here is PCRE regex: ^(\+*\-*[0-9]+\.*[0-9]*[VSB]*|\w+\s*\w+)\s*\|\s*(\+*\-*[0-9]+\.*[0-9]*).*\|\s*(\w+)\s*$...guess this can be ported to PHP :].

It's better than I could have done. I'll take a look.

Thanks!

 

[wblock@]

Doing a substitution that's the same for each line will fail when one line is different. And it will be. And as we've seen, you quickly end up with regexes that are hard to maintain. I did it in Perl, it works, but it will fail:

perl -pe 's/\s+\|/:/g; s/:\s+/:/g; s/:(us|ok)/:0/g; s/:nc/:1/g; s/:cr/:2/g; s/:nr/:3/g; s/ Volts//g;'

Write a loop to handle it, checking for the field name and then reading and parsing the rest of the line. That can be done in Perl, or awk(1), or PHP, which is just a funny kind of Perl. Or even, if you feel the need, in sh(1). (Not bash, but generic sh, which is portable.)

 

[IT_Architect]

This code:

egrep -i "$key" $FilePath | cut -d '|' -f$column | sed 's/[[:space:]]//g' | sed 's/\t//g' | sed -e 's/\([[:alpha:]]\)*$//g'

on this data:

...
+5V | 5.15 Volts | nc
+5VSB | 5.12 Volts | ok
...

produced

5.155.12

which is a problem for the +5V check, which is why the monitor would not accept it, and worked perfectly for the +5VSB check. The same situation existed for the +3.3V checks. Since the first column is the key, I would have also had the same problems checking the status, which in turn would have caused multiple triggers to fire for a single bad status. Since the solution I presented in my following post was invalid, I deleted the post. Not only couldn't I get this to work right, I had another script that just had to do some math and logic with the results. I learned how incredibly weak the shell programs are at that. From now on, I will not consider shell scripts outside of running commands and returning statuses. They are a huge time waster for more than that.