Ad:

C Discussion Thread
Unsolved
Views: 74674

Page 1
2
3
4
Next

You are currently viewing page 1 of this multi-page discussion thread

Jul 12th, 2004

File parsing in 'C'

Expand Post »

I have a bcp output file from sybase that i need to parse using 'c' and write to a new file.

A sample line from bcp o/p file is below

9890000501:74667:0:6::2:0000:0:6:0:5:0:0:0:0:0:0:0:::9890000501:1:1:0::::::::0:0:3::::::::::\202^B:\202^B^D:0:0
:1:0:0:0:0:0:0:0:0:0::::0:0:0:::::::::0::0:0:0:0:

The file has ":" has the delimitor.
I need to create a new file with a single line string as something like this

CUST:NAME=9890000501,HT=74667&0&6,LT=PT-0&PQ-5;

There will be thousands of subscribers.I also will need to get data from different bcp output files to get the data to be filled in this string.

The 'c' program needs to be fast.

I'm a novice with 'c'.

The only way i know is read char by char and check for ":" and then store the value into a char array.
I would have as many variables as there are in the string!!

What would be the fast way to do this?

Is there any function that would in one shot give me the values separated by a delimitor of my choice?

Which is the best way to build up the final string?

Can i create the fixed parts of the string and then just fill up values that need to be taken from the bcp output file??

please HELP!!!

Similar Threads

What is the best language for Text file Parsing? in Computer Science
Keep getting "reached end of file while parsing" in Java
Parsing a log file using regular expressions in C#
File parsing going awry in C
xml file parsing in c++ in C++

reuben12

Reputation Points: 11

Solved Threads: 0

Newbie Poster

Offline

16 posts
since Jul 2004

Permalink

Jul 12th, 2004

Re: File parsing in 'C'

Check out 'strtok()'; it will parse a string, stopping on one or more tokens; in this case your ':'.

The strings you parse could be referenced by an array of string pointers:

C Syntax (Toggle Plain Text)
const char* foundThings[30]; // or however big
 
while (going)
{
    foundThings[n] = strtok( inputString, ":" );
    if (foundThings[n] == NULL)
        going = false;
    else
        n++;
}
const char* foundThings[30]; // or however big

while (going)
{
    foundThings[n] = strtok( inputString, ":" );
    if (foundThings[n] == NULL)
        going = false;
    else
        n++;
}

now all parsed strings are in the foundThings[] array.

Chainsaw

Reputation Points: 36

Solved Threads: 11

Posting Pro in Training

Offline

436 posts
since Jun 2004

Permalink

Jul 12th, 2004

Re: File parsing in 'C'

WOW! thats neat.
Thank you!!

A couple of things!!

1) I read the man page for strtok.It says that it returns a (char *) pointer back.

Suppose i have a fixed line in which i want to fill up values from foundthing array.

for example ,

CUST:NAME=<value from array>,TP=<next value from array>&<next value from array>;

This values would be of variable length.

In perl i would write it in a single line with the '.' operator.

Is there a similar way here or i should store the fixed strings in a char array and do 'strcat' with each value.

The ouput file would have thousands of lines as above.

Example,

char S1[100]="CUST:NAME=";
strcat (S1,foundThing[0]);
strcat(S1,",TP=");
strcat(S1,foundThing[1]);

and so on.

Is there a faster way to do this work since efficiency is very important(reason to choose 'c' over perl)

reuben12

Reputation Points: 11

Solved Threads: 0

Newbie Poster

Offline

16 posts
since Jul 2004

Permalink

Jul 12th, 2004

Re: File parsing in 'C'

You may want to try fgets/sscanf. Maybe something like this.

C Syntax (Toggle Plain Text)
#include <stdio.h>
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char substr[32], *ptr = line;
		 int n;
		 fputs(line, stdout);
		 while ( *ptr )
		 {
			if ( sscanf(ptr, "%31[^:]%n", substr, &n) == 1 )
			{
			   ptr += n;
			   puts(substr);
			}
			else
			{
			   puts("---empty field---");
			}
			++ptr;
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}
#include <stdio.h>
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char substr[32], *ptr = line;
		 int n;
		 fputs(line, stdout);
		 while ( *ptr )
		 {
			if ( sscanf(ptr, "%31[^:]%n", substr, &n) == 1 )
			{
			   ptr += n;
			   puts(substr);
			}
			else
			{
			   puts("---empty field---");
			}
			++ptr;
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}

Then, if this is too slow, I'd look into other faster techniques.

Dave Sinkula

Team Colleague

Reputation Points: 2780

Solved Threads: 312

long time no c

Offline

4,790 posts
since Apr 2004

Permalink

Jul 12th, 2004

Re: File parsing in 'C'

Well, generally the fastest way to do it is to loop through the string in a while loop, not relying on standard string routines. Some string library functions may be implemented in assembler on some platforms, generally printf/scanf/strtok and the like aren't.

the non-standard-library parser could be something like this:

C Syntax (Toggle Plain Text)
 
// parse the source line into tokens[], returns the number of tokens found.
// note that some tokens may be empty.
int ParseLine( char* theSourceLine, const char* tokens[], int maxTokens )
{
    currentToken = 0;
    tokens[currentToken] = theSourceLine;
 
    while (*theSourceLine)
    {
        if (*theSourceLine == ':')
        {
            *theSourceLine = 0;  // null terminate this string
            currentToken++;
            if (currentToken >= maxTokens) return currentToken;  // reached the limit; maybe return -1?
            tokens[currentToken] = theSourceLine;  // next token starts here
        }
        theSourceLine++;
    }
    return currentToken;
}
// parse the source line into tokens[], returns the number of tokens found.
// note that some tokens may be empty.
int ParseLine( char* theSourceLine, const char* tokens[], int maxTokens )
{
    currentToken = 0;
    tokens[currentToken] = theSourceLine;

    while (*theSourceLine)
    {
        if (*theSourceLine == ':')
        {
            *theSourceLine = 0;  // null terminate this string
            currentToken++;
            if (currentToken >= maxTokens) return currentToken;  // reached the limit; maybe return -1?
            tokens[currentToken] = theSourceLine;  // next token starts here
        }
        theSourceLine++;
    }
    return currentToken;
}

And then, if it were me, I'd have an array of the static strings to build up the final string:

C Syntax (Toggle Plain Text)
static const char* finalStringConstants[MAX_FINAL_CONSTANTS] =
{
    "CUST:NAME=",
    "HT=",
    "LT=",
    <etc>
};
 
char outputLine[500];  // make it big enough
outputLine[0] = 0;      // null terminate it to start
for (i = 0; i < MAX_FINAL_CONSTANTS; i++)
{
    strcat(outputLine, finalStringConstants[i]);
    strcat(outputLine, tokens[i] );
}
static const char* finalStringConstants[MAX_FINAL_CONSTANTS] =
{
    "CUST:NAME=",
    "HT=",
    "LT=",
    <etc>
};

char outputLine[500];  // make it big enough
outputLine[0] = 0;      // null terminate it to start
for (i = 0; i < MAX_FINAL_CONSTANTS; i++)
{
    strcat(outputLine, finalStringConstants[i]);
    strcat(outputLine, tokens[i] );
}

If that's not fast enough, you could store the token lengths when you are parsing them and use memcpy() rather than strcpy() in the final loop.

Chainsaw

Reputation Points: 36

Solved Threads: 11

Posting Pro in Training

Offline

436 posts
since Jun 2004

Permalink

Jul 12th, 2004

Re: File parsing in 'C'

My highlighing in red:

Quote ...

Rob Pike, a leading expert on applying the C programming language, offers the following "rules" in Notes on Programming in C as programming maxims (but they can be easily viewed as points of a Unix philosophy):
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.

Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.

Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)

Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.

Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

Rule 6. There is no Rule 6.

Pike's rules 1 and 2 restate Donald Knuth's famous maxim, "Premature optimization is the root of all evil." Ken Thompson rephrased Pike's rule 4 as "When in doubt, use brute force." Rule 5 was previously stated by Fred Brooks in The Mythical Man-Month.

http://en.wikipedia.org/wiki/Unix_philosophy

Dave Sinkula

Team Colleague

Reputation Points: 2780

Solved Threads: 312

long time no c

Offline

4,790 posts
since Apr 2004

Permalink

Jul 13th, 2004

Re: File parsing in 'C'

Thanks for the solutions!!

Can we use sprintf() to format the string at the end instead of strcat()?

I can use multiple "%s" to pick from the static string and the parsed output strings
and write to a buffer in one shot.

This buffer can be written to the file.

Is this the best way to go??

reuben12

Reputation Points: 11

Solved Threads: 0

Newbie Poster

Offline

16 posts
since Jul 2004

Permalink

Jul 13th, 2004

Re: File parsing in 'C'

Do you something mean like this?

C Syntax (Toggle Plain Text)
#include <stdio.h>
 
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char name[16], ht[8], output[128];
		 int  a,b,c,d;
		 if ( sscanf(line, "%15[^:]:%7[^:]:%d:%d::%*d:%*d:%*d:%*d:%d:%d",
					 name, ht, &a, &b, &c, &d) == 6 )
		 {
			snprintf(output, sizeof output,
					 "CUST:NAME=%s,HT=%s,&%d&%d,LT=PT-%d&PQ-%d",
					 name, ht, a, b, c, d);
			puts(output);
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}
 
/* my output
CUST:NAME=9890000501,HT=74667,&0&6,LT=PT-0&PQ-5
*/
#include <stdio.h>

int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char name[16], ht[8], output[128];
		 int  a,b,c,d;
		 if ( sscanf(line, "%15[^:]:%7[^:]:%d:%d::%*d:%*d:%*d:%*d:%d:%d",
					 name, ht, &a, &b, &c, &d) == 6 )
		 {
			snprintf(output, sizeof output,
					 "CUST:NAME=%s,HT=%s,&%d&%d,LT=PT-%d&PQ-%d",
					 name, ht, a, b, c, d);
			puts(output);
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}

/* my output
CUST:NAME=9890000501,HT=74667,&0&6,LT=PT-0&PQ-5
*/

Sure -- but the fields in each record would need to be all filled or all empty in the same places. And the format string can get a bit unwieldy -- unless you are only looking for the first several fields out of it.

Dave Sinkula

Team Colleague

Reputation Points: 2780

Solved Threads: 312

long time no c

Offline

4,790 posts
since Apr 2004

Permalink

Jul 14th, 2004

Re: File parsing in 'C'

I guess sscanf wont work since i have fields with variable length delimited by ":"

%31[^:]

will it expect minimum 31 chars..i dont understand it

Can i use fprintf() to write straight to the file??

Since it is variable length fields i will have to clear the buffer for each record and memset would be expensive!!

I believe it wont write "\0" after each string ..right?

Using fprintf(), can i use "\n" for the succeding fields to be written to the next line?

I want the lines in my file to be separated by a carriage return and a line feed.
Is it the same as "\n"?????

reuben12

Reputation Points: 11

Solved Threads: 0

Newbie Poster

Offline

16 posts
since Jul 2004

Permalink

Jul 14th, 2004