DaniWeb Forum Index > Software Development > C

C RSS

File parsing in 'C'

Last »

•

Join Date: Jul 2004

Posts: 16

Reputation: reuben12 is an unknown quantity at this point

Solved Threads: 0

reuben12

Offline

Newbie Poster

File parsing in 'C'

Jul 12th, 2004

I have a bcp output file from sybase that i need to parse using 'c' and write to a new file.

A sample line from bcp o/p file is below

9890000501:74667:0:6::2:0000:0:6:0:5:0:0:0:0:0:0:0:::9890000501:1:1:0::::::::0:0:3::::::::::\202^B:\202^B^D:0:0
:1:0:0:0:0:0:0:0:0:0::::0:0:0:::::::::0::0:0:0:0:

The file has ":" has the delimitor.
I need to create a new file with a single line string as something like this

CUST:NAME=9890000501,HT=74667&0&6,LT=PT-0&PQ-5;

There will be thousands of subscribers.I also will need to get data from different bcp output files to get the data to be filled in this string.

The 'c' program needs to be fast.

I'm a novice with 'c'.

The only way i know is read char by char and check for ":" and then store the value into a char array.
I would have as many variables as there are in the string!!

What would be the fast way to do this?

Is there any function that would in one shot give me the values separated by a delimitor of my choice?

Which is the best way to build up the final string?

Can i create the fixed parts of the string and then just fill up values that need to be taken from the bcp output file??

please HELP!!!

•

Join Date: Jun 2004

Posts: 436

Reputation: Chainsaw is an unknown quantity at this point

Solved Threads: 11

Chainsaw

Offline

Unprevaricator

Re: File parsing in 'C'

Jul 12th, 2004

Check out 'strtok()'; it will parse a string, stopping on one or more tokens; in this case your ':'.

The strings you parse could be referenced by an array of string pointers:

 Help with Code Tags 
 C Syntax (Toggle Plain Text) 
const char* foundThings[30]; // or however big
 
while (going)
{
    foundThings[n] = strtok( inputString, ":" );
    if (foundThings[n] == NULL)
        going = false;
    else
        n++;
}
const char* foundThings[30]; // or however big

while (going)
{
    foundThings[n] = strtok( inputString, ":" );
    if (foundThings[n] == NULL)
        going = false;
    else
        n++;
}

now all parsed strings are in the foundThings[] array.

•

Join Date: Jul 2004

Posts: 16

Reputation: reuben12 is an unknown quantity at this point

Solved Threads: 0

reuben12

Offline

Newbie Poster

Re: File parsing in 'C'

Jul 12th, 2004

WOW! thats neat.
Thank you!!

A couple of things!!

1) I read the man page for strtok.It says that it returns a (char *) pointer back.

Suppose i have a fixed line in which i want to fill up values from foundthing array.

for example ,

CUST:NAME=<value from array>,TP=<next value from array>&<next value from array>;

This values would be of variable length.

In perl i would write it in a single line with the '.' operator.

Is there a similar way here or i should store the fixed strings in a char array and do 'strcat' with each value.

The ouput file would have thousands of lines as above.

Example,

char S1[100]="CUST:NAME=";
strcat (S1,foundThing[0]);
strcat(S1,",TP=");
strcat(S1,foundThing[1]);

and so on.

Is there a faster way to do this work since efficiency is very important(reason to choose 'c' over perl)

•

Join Date: Apr 2004

Posts: 4,335

Reputation: Dave Sinkula has a brilliant future

Solved Threads: 236

Dave Sinkula Dave Sinkula is offline

Offline

long time no c

Re: File parsing in 'C'

Jul 12th, 2004

You may want to try fgets/sscanf. Maybe something like this.

 Help with Code Tags 
 C Syntax (Toggle Plain Text) 
#include <stdio.h>
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char substr[32], *ptr = line;
		 int n;
		 fputs(line, stdout);
		 while ( *ptr )
		 {
			if ( sscanf(ptr, "%31[^:]%n", substr, &n) == 1 )
			{
			   ptr += n;
			   puts(substr);
			}
			else
			{
			   puts("---empty field---");
			}
			++ptr;
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}
#include <stdio.h>
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char substr[32], *ptr = line;
		 int n;
		 fputs(line, stdout);
		 while ( *ptr )
		 {
			if ( sscanf(ptr, "%31[^:]%n", substr, &n) == 1 )
			{
			   ptr += n;
			   puts(substr);
			}
			else
			{
			   puts("---empty field---");
			}
			++ptr;
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}

Then, if this is too slow, I'd look into other faster techniques.

"One of the methods used by statists to destroy capitalism consists in establishing controls that tie a given industry hand and foot, making it unable to solve its problems, then declaring that freedom has failed and stronger controls are necessary." --Ayn Rand

•

Join Date: Jun 2004

Posts: 436

Reputation: Chainsaw is an unknown quantity at this point

Solved Threads: 11

Chainsaw

Offline

Unprevaricator

Re: File parsing in 'C'

Jul 12th, 2004

Well, generally the fastest way to do it is to loop through the string in a while loop, not relying on standard string routines. Some string library functions may be implemented in assembler on some platforms, generally printf/scanf/strtok and the like aren't.

the non-standard-library parser could be something like this:

 Help with Code Tags 
 C Syntax (Toggle Plain Text) 
 
// parse the source line into tokens[], returns the number of tokens found.
// note that some tokens may be empty.
int ParseLine( char* theSourceLine, const char* tokens[], int maxTokens )
{
    currentToken = 0;
    tokens[currentToken] = theSourceLine;
 
    while (*theSourceLine)
    {
        if (*theSourceLine == ':')
        {
            *theSourceLine = 0;  // null terminate this string
            currentToken++;
            if (currentToken >= maxTokens) return currentToken;  // reached the limit; maybe return -1?
            tokens[currentToken] = theSourceLine;  // next token starts here
        }
        theSourceLine++;
    }
    return currentToken;
}
// parse the source line into tokens[], returns the number of tokens found.
// note that some tokens may be empty.
int ParseLine( char* theSourceLine, const char* tokens[], int maxTokens )
{
    currentToken = 0;
    tokens[currentToken] = theSourceLine;

    while (*theSourceLine)
    {
        if (*theSourceLine == ':')
        {
            *theSourceLine = 0;  // null terminate this string
            currentToken++;
            if (currentToken >= maxTokens) return currentToken;  // reached the limit; maybe return -1?
            tokens[currentToken] = theSourceLine;  // next token starts here
        }
        theSourceLine++;
    }
    return currentToken;
}

And then, if it were me, I'd have an array of the static strings to build up the final string:

 Help with Code Tags 
 C Syntax (Toggle Plain Text) 
static const char* finalStringConstants[MAX_FINAL_CONSTANTS] =
{
    "CUST:NAME=",
    "HT=",
    "LT=",
    <etc>
};
 
char outputLine[500];  // make it big enough
outputLine[0] = 0;      // null terminate it to start
for (i = 0; i < MAX_FINAL_CONSTANTS; i++)
{
    strcat(outputLine, finalStringConstants[i]);
    strcat(outputLine, tokens[i] );
}
static const char* finalStringConstants[MAX_FINAL_CONSTANTS] =
{
    "CUST:NAME=",
    "HT=",
    "LT=",
    <etc>
};

char outputLine[500];  // make it big enough
outputLine[0] = 0;      // null terminate it to start
for (i = 0; i < MAX_FINAL_CONSTANTS; i++)
{
    strcat(outputLine, finalStringConstants[i]);
    strcat(outputLine, tokens[i] );
}

If that's not fast enough, you could store the token lengths when you are parsing them and use memcpy() rather than strcpy() in the final loop.

•

Join Date: Apr 2004

Posts: 4,335

Reputation: Dave Sinkula has a brilliant future

Solved Threads: 236

Dave Sinkula Dave Sinkula is offline

Offline

long time no c

Re: File parsing in 'C'

Jul 12th, 2004

My highlighing in red:

•

Rob Pike, a leading expert on applying the C programming language, offers the following "rules" in Notes on Programming in C as programming maxims (but they can be easily viewed as points of a Unix philosophy):
Rule 1. You can't tell where a program is going to spend its time. Bottlenecks occur in surprising places, so don't try to second guess and put in a speed hack until you've proven that's where the bottleneck is.

Rule 2. Measure. Don't tune for speed until you've measured, and even then don't unless one part of the code overwhelms the rest.

Rule 3. Fancy algorithms are slow when n is small, and n is usually small. Fancy algorithms have big constants. Until you know that n is frequently going to be big, don't get fancy. (Even if n does get big, use Rule 2 first.)

Rule 4. Fancy algorithms are buggier than simple ones, and they're much harder to implement. Use simple algorithms as well as simple data structures.

Rule 5. Data dominates. If you've chosen the right data structures and organized things well, the algorithms will almost always be self-evident. Data structures, not algorithms, are central to programming.

Rule 6. There is no Rule 6.

Pike's rules 1 and 2 restate Donald Knuth's famous maxim, "Premature optimization is the root of all evil." Ken Thompson rephrased Pike's rule 4 as "When in doubt, use brute force." Rule 5 was previously stated by Fred Brooks in The Mythical Man-Month.

http://en.wikipedia.org/wiki/Unix_philosophy

•

Join Date: Jul 2004

Posts: 16

Reputation: reuben12 is an unknown quantity at this point

Solved Threads: 0

reuben12

Offline

Newbie Poster

Re: File parsing in 'C'

Jul 13th, 2004

Thanks for the solutions!!

Can we use sprintf() to format the string at the end instead of strcat()?

I can use multiple "%s" to pick from the static string and the parsed output strings
and write to a buffer in one shot.

This buffer can be written to the file.

Is this the best way to go??

•

Join Date: Apr 2004

Posts: 4,335

Reputation: Dave Sinkula has a brilliant future

Solved Threads: 236

Dave Sinkula Dave Sinkula is offline

Offline

long time no c

Re: File parsing in 'C'

Jul 13th, 2004

Do you something mean like this?

 Help with Code Tags 
 C Syntax (Toggle Plain Text) 
#include <stdio.h>
 
int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char name[16], ht[8], output[128];
		 int  a,b,c,d;
		 if ( sscanf(line, "%15[^:]:%7[^:]:%d:%d::%*d:%*d:%*d:%*d:%d:%d",
					 name, ht, &a, &b, &c, &d) == 6 )
		 {
			snprintf(output, sizeof output,
					 "CUST:NAME=%s,HT=%s,&%d&%d,LT=PT-%d&PQ-%d",
					 name, ht, a, b, c, d);
			puts(output);
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}
 
/* my output
CUST:NAME=9890000501,HT=74667,&0&6,LT=PT-0&PQ-5
*/
#include <stdio.h>

int main(void)
{
   const char filename[] = "file.txt";
   FILE *file = fopen(filename, "r");
   if ( file )
   {
	  char line [ BUFSIZ ];
	  while ( fgets(line, sizeof line, file) )
	  {
		 char name[16], ht[8], output[128];
		 int  a,b,c,d;
		 if ( sscanf(line, "%15[^:]:%7[^:]:%d:%d::%*d:%*d:%*d:%*d:%d:%d",
					 name, ht, &a, &b, &c, &d) == 6 )
		 {
			snprintf(output, sizeof output,
					 "CUST:NAME=%s,HT=%s,&%d&%d,LT=PT-%d&PQ-%d",
					 name, ht, a, b, c, d);
			puts(output);
		 }
	  }
   }
   else
   {
	  perror(filename);
   }
   return 0;
}

/* my output
CUST:NAME=9890000501,HT=74667,&0&6,LT=PT-0&PQ-5
*/

Sure -- but the fields in each record would need to be all filled or all empty in the same places. And the format string can get a bit unwieldy -- unless you are only looking for the first several fields out of it.

•

Join Date: Jul 2004

Posts: 16

Reputation: reuben12 is an unknown quantity at this point

Solved Threads: 0

reuben12

Offline

Newbie Poster

Re: File parsing in 'C'

Jul 14th, 2004

I guess sscanf wont work since i have fields with variable length delimited by ":"

%31[^:]

will it expect minimum 31 chars..i dont understand it

Can i use fprintf() to write straight to the file??

Since it is variable length fields i will have to clear the buffer for each record and memset would be expensive!!

I believe it wont write "\0" after each string ..right?

Using fprintf(), can i use "\n" for the succeding fields to be written to the next line?

I want the lines in my file to be separated by a carriage return and a line feed.
Is it the same as "\n"?????