Thursday, July 20, 2006

bc and awk

I had to calculate sum of a particular column from a huge data file. So I wrote a small bc script like this

----sum.bc-----
sum = 0
while(1)
{
c = read()
if ( c == q )
{
break
}
sum += c
}
sum
quit
----sum.bc-----

and invoked it like this

cat data.dat | awk -F '"' '{print $24}END{print "q"}' | bc -q -l sum.bc

This worked perfectly on my home system which is a linux. But when I did the same in Solaris which had a older bc that dude didn't understand my modern lingo. So I had to abandon the bc script.

So I thought I will just produce the entire sum string with numbers and "+" signs and pump it to bc, like this.

cat data.dat | awk -F '"' '{print $24}' | awk '{printf "%s+",$1}END{print "0"}' | bc -l

Again this worked with my small sample file but bc ran out of internal space when I fed a huge data file. Finally I had to abandon the specialized tool for calculation and fall back on my usual file processor awk to take over the job of the calculator.

cat data.dat | awk -F '"' '{print $24}' | awk '{s += $1}END{print s}'

Well..well ...I fed more than a Gig of data to this small wonder in that old dinky Solaris machine and it worked like a charm.

No comments: