944,070 Members | Top Members by Rank

Ad:
Dec 13th, 2006
0

'GREP' usage on huge files, any Limitation?

Expand Post »
grep -v "connected" filename > newfile

With regards to the use of grep and then output to a file as it was done in the sample above, are there any limitation to its use especially when the file is big? I have experience record truncation when it is output to a file. Anyone ever experience this before? How can this problem be resolved?
Similar Threads
Reputation Points: 10
Solved Threads: 0
Newbie Poster
crab68 is offline Offline
4 posts
since Dec 2006
Dec 13th, 2006
0

Re: 'GREP' usage on huge files, any Limitation?

Record truncation? Not normal behavior unless the record has embedded ascii nul characters. Lack of disk space or exceeding enabled quotas will also cause the output file to truncate.

grep has a line length limit of 2048 characters.
There also is a concept of largefiles, files which are so big a signed 32 bit file pointer cannot access them > 2.4GB.

Which of these things applies to your case?
Last edited by jim mcnamara; Dec 13th, 2006 at 10:51 am.
Reputation Points: 62
Solved Threads: 10
Junior Poster
jim mcnamara is offline Offline
179 posts
since May 2004
Dec 14th, 2006
0

Re: 'GREP' usage on huge files, any Limitation?

Record truncation? Not normal behavior unless the record has embedded ascii nul characters. Lack of disk space or exceeding enabled quotas will also cause the output file to truncate.

grep has a line length limit of 2048 characters.
There also is a concept of largefiles, files which are so big a signed 32 bit file pointer cannot access them > 2.4GB.

Which of these things applies to your case?
The file size is about 1.2GB. The recond was truncated when it was run in the script but when it was manually run later, the records in the file did not get truncated. Thus it is an intermittent problem. It could be due to disk space but I can't verify.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
crab68 is offline Offline
4 posts
since Dec 2006
Dec 14th, 2006
0

Re: 'GREP' usage on huge files, any Limitation?

The way disk i/o in unix works is that data is parked in an in-memory cache in the kernel - it is not guaranteed to be written to disk when the write() system call is invoked. Every 30 seconds or so the syncer daemon issues a sync command. This forces the kernel to write everything in the kernel buffer to disk.

What you are seeing is an incompleted write operation - for whatever reason. Common reasons are - a signal was sent to the process that terminated it, write() or sync failed because something else filled up the
disk (maybe a temp file) and then that file went away, disk errors caused a fatal error. If it's an nsf mounted disk then the network also becomes an issue. What errors do you see in the log?
Reputation Points: 62
Solved Threads: 10
Junior Poster
jim mcnamara is offline Offline
179 posts
since May 2004
Dec 18th, 2006
0

Re: 'GREP' usage on huge files, any Limitation?

The way disk i/o in unix works is that data is parked in an in-memory cache in the kernel - it is not guaranteed to be written to disk when the write() system call is invoked. Every 30 seconds or so the syncer daemon issues a sync command. This forces the kernel to write everything in the kernel buffer to disk.

What you are seeing is an incompleted write operation - for whatever reason. Common reasons are - a signal was sent to the process that terminated it, write() or sync failed because something else filled up the
disk (maybe a temp file) and then that file went away, disk errors caused a fatal error. If it's an nsf mounted disk then the network also becomes an issue. What errors do you see in the log?
There was no tracking of error message in the script. WIll probably need write a program to do the task of dividing the files into two.
Reputation Points: 10
Solved Threads: 0
Newbie Poster
crab68 is offline Offline
4 posts
since Dec 2006

This thread is more than three months old

No one has posted to this discussion for at least three months. Please let old threads die and do not reply to them unless you feel you have something new and valuable to contribute that absolutely must be added to make the discussion complete. Otherwise, please start a new thread in this forum instead.
Message:
Previous Thread in Shell Scripting Forum Timeline: shell script prob
Next Thread in Shell Scripting Forum Timeline: putting result of a sql query in a file in shell scripting





About Us | Contact Us | Advertise | Acceptable Use Policy
Forum Index | Build Custom RSS Feed


Follow us on Twitter


© 2011 DaniWeb® LLC