•
•
•
•
What is DaniWeb IT Discussion Community?
You're currently browsing the Shell Scripting section within the Software Development category of DaniWeb, a massive community of 427,816 software developers, web developers, Internet marketers, and tech gurus who are all enthusiastic about making contacts, networking, and learning from each other. In fact, there are 3,836 IT professionals currently interacting right now! Registration is free, only takes a minute and lets you enjoy all of the interactive features of the site.
Please support our Shell Scripting advertiser: Programming Forums
Views: 2861 | Replies: 4
 |
•
•
Join Date: Dec 2006
Posts: 4
Reputation: 
Rep Power: 0
Solved Threads: 0
Newbie Poster
'GREP' usage on huge files, any Limitation? grep -v "connected" filename > newfile
With regards to the use of grep and then output to a file as it was done in the sample above, are there any limitation to its use especially when the file is big? I have experience record truncation when it is output to a file. Anyone ever experience this before? How can this problem be resolved?
With regards to the use of grep and then output to a file as it was done in the sample above, are there any limitation to its use especially when the file is big? I have experience record truncation when it is output to a file. Anyone ever experience this before? How can this problem be resolved?
•
•
Join Date: May 2004
Posts: 177
Reputation:
Rep Power: 5
Solved Threads: 9
Junior Poster
Record truncation? Not normal behavior unless the record has embedded ascii nul characters. Lack of disk space or exceeding enabled quotas will also cause the output file to truncate.
grep has a line length limit of 2048 characters.
There also is a concept of largefiles, files which are so big a signed 32 bit file pointer cannot access them > 2.4GB.
Which of these things applies to your case?
grep has a line length limit of 2048 characters.
There also is a concept of largefiles, files which are so big a signed 32 bit file pointer cannot access them > 2.4GB.
Which of these things applies to your case?
Last edited by jim mcnamara : Dec 13th, 2006 at 9:51 am.
•
•
Join Date: Dec 2006
Posts: 4
Reputation:
Rep Power: 0
Solved Threads: 0
Newbie Poster
•
•
•
•
Originally Posted by jim mcnamara 
Record truncation? Not normal behavior unless the record has embedded ascii nul characters. Lack of disk space or exceeding enabled quotas will also cause the output file to truncate.
grep has a line length limit of 2048 characters.
There also is a concept of largefiles, files which are so big a signed 32 bit file pointer cannot access them > 2.4GB.
Which of these things applies to your case?
The file size is about 1.2GB. The recond was truncated when it was run in the script but when it was manually run later, the records in the file did not get truncated. Thus it is an intermittent problem. It could be due to disk space but I can't verify.
•
•
Join Date: May 2004
Posts: 177
Reputation:
Rep Power: 5
Solved Threads: 9
Junior Poster
The way disk i/o in unix works is that data is parked in an in-memory cache in the kernel - it is not guaranteed to be written to disk when the write() system call is invoked. Every 30 seconds or so the syncer daemon issues a sync command. This forces the kernel to write everything in the kernel buffer to disk.
What you are seeing is an incompleted write operation - for whatever reason. Common reasons are - a signal was sent to the process that terminated it, write() or sync failed because something else filled up the
disk (maybe a temp file) and then that file went away, disk errors caused a fatal error. If it's an nsf mounted disk then the network also becomes an issue. What errors do you see in the log?
What you are seeing is an incompleted write operation - for whatever reason. Common reasons are - a signal was sent to the process that terminated it, write() or sync failed because something else filled up the
disk (maybe a temp file) and then that file went away, disk errors caused a fatal error. If it's an nsf mounted disk then the network also becomes an issue. What errors do you see in the log?
•
•
Join Date: Dec 2006
Posts: 4
Reputation:
Rep Power: 0
Solved Threads: 0
Newbie Poster
•
•
•
•
Originally Posted by jim mcnamara
The way disk i/o in unix works is that data is parked in an in-memory cache in the kernel - it is not guaranteed to be written to disk when the write() system call is invoked. Every 30 seconds or so the syncer daemon issues a sync command. This forces the kernel to write everything in the kernel buffer to disk.
What you are seeing is an incompleted write operation - for whatever reason. Common reasons are - a signal was sent to the process that terminated it, write() or sync failed because something else filled up the
disk (maybe a temp file) and then that file went away, disk errors caused a fatal error. If it's an nsf mounted disk then the network also becomes an issue. What errors do you see in the log?
There was no tracking of error message in the script. WIll probably need write a program to do the task of dividing the files into two.
|
•
•
•
•
•
•
•
•
DaniWeb Shell Scripting Marketplace
•
•
•
•
Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
Linear Mode
- Including music on a webpage (HTML and CSS)
- How do I work around this? - virtual templates (C++)
- need a good graphics program (Graphics and Multimedia)
- System Processes in XP? NetTsk.exe? (Windows NT / 2000 / XP / 2003)
Other Threads in the Shell Scripting Forum
- Previous Thread: shell script prob
- Next Thread: putting result of a sql query in a file in shell scripting
