943,724 Members | Top Members by Rank

Ad:
You are currently viewing page 1 of this multi-page discussion thread
Aug 25th, 2009
0

A shell script that makes backup copies of changed files (also involving CRON)

Expand Post »
Hey

Ive been asked (and I may add this to my system as well) to make a shell script that backups to all files in x location to another location in z. The thing is that the backup must only be made if one file has been changed. Example:

/x contains:

a (1 byte)
b (1 byte)
c (1 byte)

/y contains:

nothing

Ill put this script into CRON's file and everytime (say 3) it will execute. 3 arrives and there is nothing in /y and it will copy all files in /x to /y. To compensate space I think a good idea (this I will do to my script for my system) is gzip it all up. Anyways the current state right now is.

/x contains:

a (1 byte)
b (1 byte)
c (1 byte)

/y contains:

a (1 byte)
b (1 byte)
c (1 byte)

Again 3 arrives and /y is the same as /x so nothing changes and nothing happens. But now, I open c in /x and type something inside now it is:

/x contains:

a (1 byte)
b (1 byte)
c (2 bytes)

This changes it so now when the script excutes at 3 it has to delete everything in /y and rewrite it all over again (regardless that the other files havent changed).

How do I do this? I imagine something like doing a ls -a on /x and using cut to cut certain columns and seeing if it equals /y's files/file sizes but im not sure how to make a filename and file size relation and a bit humiliating but I dont know how to use cut either (I know something but still)

If someone could help thank you very much.
Similar Threads
Reputation Points: 49
Solved Threads: 1
Posting Whiz in Training
riahc3 is offline Offline
298 posts
since May 2008
Aug 26th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Why don't you just use rsync? This is a significant undertaking when the wheel has already been invented.
Featured Poster
Reputation Points: 1749
Solved Threads: 735
Senior Poster
sknake is offline Offline
3,948 posts
since Feb 2009
Aug 27th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Click to Expand / Collapse  Quote originally posted by sknake ...
Why don't you just use rsync? This is a significant undertaking when the wheel has already been invented.
Thanks but I rather use a self made shell script. Any tips?
Reputation Points: 49
Solved Threads: 1
Posting Whiz in Training
riahc3 is offline Offline
298 posts
since May 2008
Aug 27th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Yeah .. look at the rsync source for ideas
Featured Poster
Reputation Points: 1749
Solved Threads: 735
Senior Poster
sknake is offline Offline
3,948 posts
since Feb 2009
Aug 28th, 2009
1

Re: A shell script that makes backup copies of changed files (also involving CRON)

Click to Expand / Collapse  Quote originally posted by sknake ...
Yeah .. look at the rsync source for ideas
I rather not "rip" code from another utility.

Id just like to make this using a shell script. Please no more comments suggesting rsync. Thank you
Reputation Points: 49
Solved Threads: 1
Posting Whiz in Training
riahc3 is offline Offline
298 posts
since May 2008
Aug 28th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Ok well build a file index of your source directories and destination directories. Then build a list of source/dest files that do not exist in either folders and delete/add them as necessary. For files that do exist in both directories you should use an md5sum and filesize check depending on how careful you want to be. Collision rates are very low (unless malicious and intentional) for md5sums so you could probably get away with just do that.

You could use ls and parse the columns, or use du or stat :
bash Syntax (Toggle Plain Text)
  1. root@svn:/root/backup# du -b backup.sh | sed 's/\([0-9]*\)\(.*\)/\1/'
  2. 4252

Just because the size is the same doesn't mean that it has not changed so also compare the md5 stamp:
bash Syntax (Toggle Plain Text)
  1. root@svn:/root/backup# md5sum backup.sh | sed 's/\([a-Z0-9]*\)\(.*\)/\1/'
  2. 241cca25e509409c1fbd6f1d46ec94e6

Now just automate all of that and you're good to go!
Featured Poster
Reputation Points: 1749
Solved Threads: 735
Senior Poster
sknake is offline Offline
3,948 posts
since Feb 2009
Aug 29th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Click to Expand / Collapse  Quote originally posted by sknake ...
Ok well build a file index of your source directories and destination directories. Then build a list of source/dest files that do not exist in either folders and delete/add them as necessary. For files that do exist in both directories you should use an md5sum and filesize check depending on how careful you want to be. Collision rates are very low (unless malicious and intentional) for md5sums so you could probably get away with just do that.

You could use ls and parse the columns, or use du or stat :
bash Syntax (Toggle Plain Text)
  1. root@svn:/root/backup# du -b backup.sh | sed 's/\([0-9]*\)\(.*\)/\1/'
  2. 4252

Just because the size is the same doesn't mean that it has not changed so also compare the md5 stamp:
bash Syntax (Toggle Plain Text)
  1. root@svn:/root/backup# md5sum backup.sh | sed 's/\([a-Z0-9]*\)\(.*\)/\1/'
  2. 241cca25e509409c1fbd6f1d46ec94e6

Now just automate all of that and you're good to go!

Thank you for helping

Lets analize everything.
du lists with details everything in the current folder, -b showing it in bytes and sed cuts off all the chars listed. That I understand.

md5sum would actually show me the md5 of backup.sh right? Not the actual files in the backed up file/folder/etc. And why would I cut off those chars if md5 includes letters and numbers?

Also reexplain please the part about "build a file index of your source directories and destination directories. Then build a list of source/dest files that do not exist in either folders and delete/add them as necessary."

would I have to do something like:

Shell Scripting Syntax (Toggle Plain Text)
  1. ls /x >> /x/xlist
  2. ls /y >>/x/ylist
  3. if xlist -ne ylist then
  4. ....

Is that what you ment?
Reputation Points: 49
Solved Threads: 1
Posting Whiz in Training
riahc3 is offline Offline
298 posts
since May 2008
Aug 29th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

>>md5sum would actually show me the md5 of backup.sh right? Not the actual files in the backed up file/folder/etc. And why would I cut off those chars if md5 includes letters and numbers?

The expression I gave for the md5 didn't cut off anything, it took the entire hash. Just like du you will need to run md5sum on every file to get the file size & check sum.

As far as your concept for builing the directories -- yes, that is how you would go about it. However ls prints out a lot of crap you don't need for this so you would be better off using find /src/dir -type d to make that directories with mkdir -p and then use find /src/dir -type f to get the files.
Featured Poster
Reputation Points: 1749
Solved Threads: 735
Senior Poster
sknake is offline Offline
3,948 posts
since Feb 2009
Aug 30th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

Click to Expand / Collapse  Quote originally posted by sknake ...
As far as your concept for builing the directories -- yes, that is how you would go about it. However ls prints out a lot of crap you don't need for this so you would be better off using find /src/dir -type d to make that directories with mkdir -p and then use find /src/dir -type f to get the files.
I believe /src/dir is the current directory so I would have to change it; Is there any variable in Linux that shows the current directory? Could I maybe use pwd?
Reputation Points: 49
Solved Threads: 1
Posting Whiz in Training
riahc3 is offline Offline
298 posts
since May 2008
Aug 30th, 2009
0

Re: A shell script that makes backup copies of changed files (also involving CRON)

.... you have a long road ahead of you

Take your pick
bash Syntax (Toggle Plain Text)
  1. sk@sk:~$ pwd
  2. /home/wheel/sk
  3. sk@sk:~$ echo ${PWD}
  4. /home/wheel/sk
Featured Poster
Reputation Points: 1749
Solved Threads: 735
Senior Poster
sknake is offline Offline
3,948 posts
since Feb 2009

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in Shell Scripting Forum Timeline: Change Layout
Next Thread in Shell Scripting Forum Timeline: How can i display time 5mins ago on Solaris 10





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC