Remove the Last Field and Redirect to File

[abed1986]

Hello everyone,

I am trying to read in a folder path, delimited by / and ending in COSC, and then output the last field to a file. It is supposed to match all directories that end in a course name, for example, COSC-101 or COSC201.

I am using the following lines, but it doesn't succeed:

#!/bin/sh

read coursePath

echo $coursePath | grep -w "COSC$" >/dev/null 2>&1 | grep -oE "[^/]+$" > course

Thanks in advance for any help that can be offered

[histrungalot]

#!/bin/sh
read coursepath
echo $coursepath | sed -ne 's/.*COSC[^0-9]*\(.*\)/\1/p'

[abed1986]

Hello histrungalot,

Thanks for this workaround.

Do you know of a way to do this with grep? I forgot to specify before, but I am not allowed to use sed or awk in my assignment.

[abed1986]

In my assignment, a folder that represents a course is supposed to include files, each of which represents the marks of the students in that course for individual assignments. Each line consists of a student id, and a mark out of 100, separated by a single tab.

For example, you could have the file hw1 in the directory COSC-101, with the following contents:

st150 54
st173 82
st632 94
st234 100

And, also another file called hw2, with the following contents:

st632 81
st234 97
st150 71
st173 74

One of the requirements of the assignment is to average the marks across all "assignments" (i.e. files in the course directory) for every student id.

I am starting off by just attempting to get all the marks for each student into one file each.

I am trying to use a for loop to read the contents of all files, and for every unique student id create a new file (named with the format <studentid>marks) with all their marks:

cd COSC101
for file in ./*
do
cat $file | while read line; do
echo $line >> `cut -f1`marks
echo $line | cut -f2 >> `echo $line | cut -f1`marks
done
done

As you might expect, this doesn't work out too well. I just get a bunch of files that are named with a combination of both the student id and their marks, and I cannot cat them either.

Any assistance would be greatly appreciated

[histrungalot]

And this has to be done using a shell script?
If yes, any shell in particular?

[histrungalot]

Each shell is different in what level of file IO you can do and what type of string processing is available.
Here is a start for /bin/sh. If using a different shell I might do it differently.

#!/bin/sh
for file in ./*
do
     while read line;
     do
         student=`echo $line | cut -f1 -d' '`
         grade=`echo $line | cut -f2 -d' '`
         echo "$student got grade $grade"
     done < $file
done

[abed1986]

And this has to be done using a shell script?
If yes, any shell in particular?

It needs to be done using the Bourne shell only.

I am going to work on trying to get a grade total and average for each student id, and post back here if I face any obstacles.

Thanks again

[ravi89]

I am starting off by just attempting to get all the marks for each student into one file each.

I had used single space as a delimiter.

#!/bin/bash
for file in *.txt
do
        cat $file >>infile
done

for sid in `uniq infile | cut -d" " -f1 `
do
         grep "$sid" infile|cut -d" " -f2 >"$sid"
done
rm infile

[abed1986]

Hello ravi89,

I can only use the Bourne shell for my assignment. Thanks for posting that though, I am sure I will have to use the Bash shell in my class at some point.

I am more than a little disappointed, but after a lot of trying I was not able to sum the grade totals for each student.

It seems that the Bourne shell is really limited in this area, as you are stuck with using the expr command.

Here is my script so far:

#!/bin/sh

read coursePath

course=`echo $coursePath | rev | cut -d/ -f1 | rev`

if [ $course = "COSC" ]
then
	echo "COSC is here"
	echo $course


if [ -d "COSC" ]
then
	rm -rf COSC 2>&1
fi
mkdir COSC
cd COSC
mkdir marksbyid

cat << EOF > hw1
cse67	94
cse78	43
cse12	88
EOF

cat << EOF > hw2
cse78	94
cse12	79
cse67	81
EOF

for file in ./*
do
	while read line;
	do
		id=`echo $line | cut -f1 -d' '`
		mark=`echo $line | cut -f2 -d' '`
		echo "$id got mark $mark"

		if [ ! -f ./marksbyid/$id ]
		then
			touch ./marksbyid/$id
			echo $mark > "./marksbyid/$id"
		else
			echo $mark >> "./marksbyid/$id"
		fi

	done < $file
done

#cd marksbyid
#for file in ./*
#do
#done


else
	echo "COSC is not here"
fi

cd ..
rm -rf COSC 2>&1

In the course directory, I create a "marksbyid" folder with files named by student id, which contain the marks for each student by id.

I tried to use the following to sum the marks in the marksbyid directory:

sh
sh-4.1$ sum=0
sh-4.1$ for file in ./*
> do
> while read line;
> do
> sum=`expr $sum + $line`
> done
> echo $sum
> echo "Total: $sum" >> `ls file`
> done
exit

But this will give errors because of the use of $line (and possibly for other reasons).

[histrungalot]

#!/bin/sh

sum=0
num=0
while read line;
do
   sum=$((sum+line))
   num=$((num+1))
done<< EOF
50
50
50
100
100
100
EOF
avg=$((sum/num))
echo "Average: $avg"

Output:

$ ./a.sh 
Average: 75
$

[abed1986]

Hello again,

I have been able to complete all my assignment's requirements, except for one.

First, a summary: This bourne shell script takes files from a course directory that contain nothing but student ids and marks (separated by whitespace), and outputs their average, along with information about them from /etc/passwd.

I have been stuck on a little thing, but have not been able to crack it. I need to keep a newline between each student's information, however, there should be no newline after the last student's information.

Here is my script:

#!/bin/sh

read coursePath

cd "$coursePath"
mkdir /tmp/marksbyid

for file in ./*
    do
        while read line;
        do
            id=`echo $line | cut -f1 -d' '`
            mark=`echo $line | cut -f2 -d' '`
            #echo "$id got mark $mark"

            if [ ! -f /tmp/marksbyid/$id ]
            then
                touch /tmp/marksbyid/$id

                if [ "X$mark" != "Xexcused" ]
                then
                    echo $mark > "/tmp/marksbyid/$id"
                elif [ -z $mark ]
                then
                    echo 0 > "/tmp/marksbyid/$id"
                fi
            else
                if [ "X$mark" != "Xexcused" ]
                then
                    echo $mark >> "/tmp/marksbyid/$id"
                elif [ -z $mark ]
                then
                    echo 0 > "/tmp/marksbyid/$id"
                fi
            fi

    done < $file
done

for file in /tmp/marksbyid/*
do
    sum=0
    n=0

    while read line;
    do
        sum=$((sum+line))
        n=$((n+1))
    done < $file

    avg=$((sum/n))
    echo "$avg" >> "`ls $file`"
    #echo "`ls $file` Average: $avg"
done

cd /tmp/marksbyid
echo -e "$coursePath info\n" 
for file in ./*
do
    id=`ls $file | tr -d ./`
    #info=`cat /etc/passwd | grep $id`
    info=`egrep -i "^${id}" /etc/passwd`

    if [ $? -eq 0 ]
    then
        login=`echo $info | cut -d: -f1`
        name=`echo $info | cut -d: -f5`
        home=`echo $info | cut -d: -f6`
        shell=`echo $info | cut -d: -f7 | cut -d/ -f5`
        avg=`tail -1 $file`

        echo "login: $login"
        echo "name: $name"
        echo "home directory: $home"
        echo "login shell: $shell"
        echo "average mark: $avg"
        echo -e "\n"
    fi
done

rm -rf /tmp/marksbyid
exit

I have not been able to figure out how to omit the last newline from appearing below the final student. Also, it seems like there are two newlines in between each student instead of one, which I cannot figure out.

Any help is much appreciated

[histrungalot]

The two newlines are coming from the echo -e "\n". echo always puts a newline after the print. So use echo to only have one between students.
To not have newline at the end move the newline to the beginning of the printing for loop and only print the newline if this is not the first time thru the loop.

#!/bin/sh

# Fake test data
listing="This is a test"

# Create a variable to indicate if newline
# should be printed, want to print newline 
# before the print out of the data
firstTime=1
for file in $listing
do
   # Check to see if newline should be printed
   # don't want to print it if first time thru loop
   if [ $firstTime -eq 0 ]
   then
      # print newline
      echo
   fi
   # Change the print variable to print all 
   # of the time now
   firstTime=0
   echo "login: $file"
   echo "name: $file"
   echo "home directory: $file"
   echo "login shell: $file"
   echo "average mark: $file"
   #echo -e "\n"   <------------- Remove this
done

Posted output but it didn't show the newline white space so I took it out.

[abed1986]

I made the suggested changes, but for some strange reason there are still two newlines between the 2nd last and last student.

Here is the relevant portion of the script:

first=1
for file in ./*
do

    if [ $first -eq 0 ]
    then
        echo
    fi
    first=0

    id=`ls $file | tr -d ./`
    #info=`cat /etc/passwd | grep $id`
    info=`egrep -i "^${id}" /etc/passwd`

    if [ $? -eq 0 ]
    then
        login=`echo $info | cut -d: -f1`
        name=`echo $info | cut -d: -f5`
        home=`echo $info | cut -d: -f6`
        shell=`echo $info | cut -d: -f7 | cut -d/ -f5`
        avg=`tail -1 $file`

        echo "login: $login"
        echo "name: $name"
        echo "home directory: $home"
        echo "login shell: $shell"
        echo "average mark: $avg"
    fi
done

[histrungalot]

I think it should look like:

first=1
for file in ./*
do
    id=`ls $file | tr -d ./`
    #info=`cat /etc/passwd | grep $id`
    info=`egrep -i "^${id}" /etc/passwd`

    if [ $? -eq 0 ]
    then
        # You only want to do the newline if egrep was successful
        if [ $first -eq 0 ]
        then
           echo
        fi
        first=0
        login=`echo $info | cut -d: -f1`
        name=`echo $info | cut -d: -f5`
        home=`echo $info | cut -d: -f6`
        shell=`echo $info | cut -d: -f7 | cut -d/ -f5`
        avg=`tail -1 $file`

        echo "login: $login"
        echo "name: $name"
        echo "home directory: $home"
        echo "login shell: $shell"
        echo "average mark: $avg"
    fi
done

See if that fixes the current newline issuse. If for some reason the egrep failed you would have printed a newline which is not what you want to do.

Do you ever have if [ $? -eq 0 ]fail?

[abed1986]

I posted that at school before I headed off for class, and while going to class I realized that was exactly the reason why that was happening! Thanks for pointing it out though.

There are a couple of non-existent student ids thrown in to the mark files, and those will not match anything in /etc/passwd.

Now I am stuck on one last requirement that necessitates that my output be ordered by first name (i.e. the fifth delimited column in /etc/passwd). The problem is in the way that I structured my program.

I stripped each student id's marks into its own separate file named by student id, which I then use to egrep using the file name as a filter, but that will result in them being ordered in whatever way they were originally ordered.

The assignment is due tomorrow, and it seems to me that I have no way of resolving this without drastically overhauling my program. It seems I have backed myself into a corner here, unfortunately.

[histrungalot]

This is the best I can do without using sed or awk and not modifying the whole structure of the script.
I tested it with something I created so I think it will work for you.

#!/bin/sh

read coursePath

cd "$coursePath"
mkdir /tmp/marksbyid

for file in ./*
    do
        while read line;
        do
            id=`echo $line | cut -f1 -d' '`
            mark=`echo $line | cut -f2 -d' '`
            name=`egrep -i "^${id}" /etc/passwd | cut -d: -f5`
            #echo "$id got mark $mark"
            if [ ! -f /tmp/marksbyid/$id ]
            then
                touch /tmp/marksbyid/$id

                if [ "X$mark" != "Xexcused" ]
                then
                    echo $mark > "/tmp/marksbyid/$id"
                elif [ -z $mark ]
                then
                    echo 0 > "/tmp/marksbyid/$id"
                fi
            else
                if [ "X$mark" != "Xexcused" ]
                then
                    echo $mark >> "/tmp/marksbyid/$id"
                elif [ -z $mark ]
                then
                    echo 0 > "/tmp/marksbyid/$id"
                fi
            fi

    done < $file
done

for file in /tmp/marksbyid/*
do
    sum=0
    n=0

    while read line;
    do
        sum=$((sum+line))
        n=$((n+1))
    done < $file

    avg=$((sum/n))
    echo "$avg" >> "`ls $file`"
    #echo "`ls $file` Average: $avg"
done

cd /tmp/marksbyid
echo -e "$coursePath info\n" 

#------------ Sorting --------------
# Get a list of name with ids
listing=""
for file in ./*
do
   id=`ls $file | tr -d ./`
   info=`egrep -i "^${id}" ../pas`
   if [ $? -eq 0 ]
   then
      name=`echo $info | cut -d: -f2`
      listing=`echo "${listing}$name:$id;"`
   fi
done

# Sort the list of name and ids
listing=`echo $listing | tr ';' '\n' | sort -t\: -k1,1 | tr ':' ' '`
#------------ Sorting --------------

other=0
first=1
for file in $listing
do
    # Every other file is the name so only look at 
    # every other string in the listing variable
    if [ $other -eq 1 ] 
    then
      other=0
      #id=`ls $file | tr -d ./`
      id=$file
      #info=`cat /etc/passwd | grep $id`
      info=`egrep -i "^${id}" /etc/passwd`

      if [ $? -eq 0 ]
      then
        # You only want to do the newline if egrep was successful
        if [ $first -eq 0 ]
        then
           echo
        fi
        first=0
        login=`echo $info | cut -d: -f1`
        name=`echo $info | cut -d: -f5`
        home=`echo $info | cut -d: -f6`
        shell=`echo $info | cut -d: -f7 | cut -d/ -f5`
        avg=`tail -1 $file`

        echo "login: $login"
        echo "name: $name"
        echo "home directory: $home"
        echo "login shell: $shell"
        echo "average mark: $avg"
      fi
    else
      other=1
    fi
done

rm -rf /tmp/marksbyid
exit

[abed1986]

Thanks as always histrungalot for helping me out with this.

I had to modify your last code a little bit to fit my specific assignment requirements, but I learned a lot doing so, so kudos.