Traverse a file row by row using awk

[Member24]

Hi, I recently started working using awk. I am facing a problem to traverse a file row by row.
I have a file test.txt which has 2 fields filename and action.
test.txt
--------
/tmp/a.txt|Action1
/tmp/b.txt|Action2
/tmp/c.txt|Action3

I need to check for the existence of file ($1) in the above file, if true copy the action ($2) to a temporary file /usr/temp.txt and i am using this file as input to another script. Below is the awk i am using,

awk -F"|" '{
if (!system("test -f " $1))
{
printf "%s exists",$1
printf "%s", $2 > /usr/temp.txt
system("") -- Executing a script which takes temp.txt as input
system("rm -f " $1)
}
else
{
print $1 " Not exists"
}
}' /tmp/test.txt

The problem i am facing here is, if a.txt and c.txt are there in test.txt file, it is copying the Action1 and Action3 into the file temp.txt. I want only one Action at a time.

This is how i want. Check for first file existence if true copy the action to a file and go to next line. if not true go to next line. and so on..

Please help.

[JeoSaurus]

Hi Member24!

It looks like you're running into this feature of awk output redirection:

print items > output-file
This type of redirection prints the items into the output file output-file. The file name output-file can be any expression. Its value is changed to a string and then used as a file name (see section Expressions). When this type of redirection is used, the output-file is erased before the first output is written to it. Subsequent writes to the same output-file do not erase output-file, but append to it.

(Found here: http://www.chemie.fu-berlin.de/chemnet/use/info/gawk/gawk_7.html)

BUT it looks like there is a simple solution. Here's another excerpt from that page:

Redirecting output using `>', `>>', or `|' asks the system to open a file or pipe only if the particular file or command you've specified has not already been written to by your program, or if it has been closed since it was last written to.

So! It looks like you can use the "close()" command to have awk start anew with the next line that it writes to the file, and overwrite what's already there. I copied your script and modified it a bit for this test. Here's the result:

## The script WITHOUT the close()
awk -F"|" '{
   if (!system("test -f " $1)) {
      print $1 " exists\n"
      print $2 > "action.txt"
      system("cat action.txt")
   } else {
      print $1 " Not exists"
   }
}' list.txt

## In the output you can see that every time we cat action.txt, it
## has been appended to, not overwritten.

/tmp/a.txt exists

Action1

/tmp/b.txt exists

Action1
Action2

/tmp/c.txt exists

Action1
Action2
Action3

## WITH the close()

awk -F"|" '{
   if (!system("test -f " $1)) {
      print $1 " exists\n"
      print $2 > "action.txt"
      system("cat action.txt")
      close("action.txt")
   } else {
      print $1 " Not exists"
   }
}' list.txt

## We can see in the output that when action.txt is closed after
## after the action, it is overwritten before the next action

/tmp/a.txt exists

Action1

/tmp/b.txt exists

Action2

/tmp/c.txt exists

Action3

I hope this helps!

[Member24]

Hi Gromit. Thanks for the response. I will try this. Actually i am using the output file action.txt as an input to microstrategy command manager. This is the command. system("mstrcmdmgr -n '"$PROJECT_NAME'" -u '"USER_NAME"' -p '"$PASSWORD"' -f action.txt -o '"LOG_FILE'""). How do i use close() here? Also i want check the exit status of this it. $? doesn't seem to be working in awk. Is there any way to do this?

[Member24]

close() is working fine. But i am facing problem in creating the file with date action_`date +%Y-%m-%d` and redirecting $2 to the file within the awk. Also how do i see the exit status of previous executed command?

[Member24]

I could resolve the both file creation with date and finding exit status of previous executed command in awk. I am stuck with msending mail with 'mutt' command. I want to change the sender's address. Please note, i habe to use mutt with in awk. Please suggest me.

[Member24]

This is the command i am trying to execute in awk. system("/usr/bin/ksh/send_mail.ksh -s "subject" -f '"FROM_EMAIL_ID"' -t '"TO_EMAIL_ID"' -m "message""). FROM_EMAIL_ID and TO_EMAIL_ID are environment variables which are declared in parameter_general.ksh. send_mail.ksh has from address and to address and is using mutt command to send emails. The above command is working in other shell scripts but when i use it in awk with system() call, it is not working. It says from address and to address are not passed. Please help.

[JeoSaurus]

Hi Member24!

Looks like you've made great progress!

I *think* the problem you're having now is the way that AWK handles environment variables. You may be able to use the ENVIRON array to pull that data into awk. Something like this might work:

system("/usr/bin/ksh/send_mail.ksh -s "subject" -f "ENVIRON["FROM_EMAIL_ID"]" -t "ENVIRON["TO_EMAIL_ID"]" -m "message"")

OR you could try importing the variables up front, with the awk command, before starting the script:

awk -v FROM="$FROM_EMAIL_ID" -v TO="$TO_EMAIL_ID" 'script starts here...
...
system("/usr/bin/ksh/send_mail.ksh -s "subject" -f "FROM" -t "TO" -m "message"")
...

I hope this helps!

[Member24]

Hi Gromit,

Thanks for your suggestion. I identified that the problem is not in environment variables. When i removed quotes to subject and message it worked. subject=$3, message=$4. system("/usr/bin/ksh/send_mail.ksh -s subject -f '"FROM_EMAIL_ID"' -t '"TO_EMAIL_ID"' -m message"). But in the email i dint $3 and $4 values instead i got 'subject' as subject and 'message' as body. I tried giving $3 and $4 directly. Still it dint work. I tried with quotations and without quotations. still no use. Please suggest me.

[Member24]

Hi Gromit,

I resolved the problem. Issue is, i should enclose $3 value which is coming from lookup file in double quotes("") since it is subject. In my awk i assigned $3 to a variable and used it in the system call. subject=$3, message=$4, system("/usr/bin/ksh/send_mail.ksh -s "subject" -f '"FROM_EMAIL_ID"' -t '"TO_EMAIL_ID"' -m "message"").

I really appreciate your help..!! Thanks :)

[JeoSaurus]

Good catch! Glad we could help :)