Problem with fgets not reading properly

[JohnMcPherson]

Hello, I am having a problem with the fgets statement not reading data correctly. It reads most of the data I need without a problem, but three lines of data are not read correctly. The code in question follows:

      strcpy(passw,"P7");
      fscanf(fip, "%3s", ckod); /* card # 164 */
      fprintf(ff16,"\n\n%12s%12s\n", "card# 164 :", "Password ");
      printf("passwd is %s\n", ckod);
      if(strcmp(passw,ckod)!=0)
      {
        printf("\n%64s\n%80s\n", "E R R O R !", "Password is incorrect. Please resubmit.");
        exit(164);
      }

      /***** Cards 165 - 174 *****/

      fgets(str, 82, fip);
      fprintf(ff16, "\nLIST OF SOLVENCY SURCHARGE RATES\n");
      fgets(str, 82, fip);
      if (sscanf(&str[1], "%d", &baseYearSurcharge) < 1)
      {
        fprintf(ff16, "Failed to correctly read the screen input for base year\n");
      }
      else
      {
        fprintf(ff16, "\nIS SURCHARGE APPLIED TO THE BASE YEAR ( YES = 1, NO = 0 ) = %d\n", baseYearSurcharge);
      }
      fgets(str, 82, fip);
      if (sscanf(&str[1], "%lf  %lf", &solvpctchkinitial, &solvpctchkmaximum) < 2)
      {
        fprintf(ff16, "Failed to correctly read the screen input for floor and ceiling\n");
      }
      else
      {
        fprintf(ff16, "\nINITIAL PERCENTAGE = %lf, MAXIMUM PERCENTAGE = %lf\n", solvpctchkinitial, solvpctchkmaximum);
        surchargeCheckPercent = solvpctchkinitial;
      }
      /* Read tax rates from the file. */
      /* Always read 9 lines, with 9 fields each, but only the first (knri) are valid. */
      for (iLine = 0; iLine < 9; ++iLine)
      {
        /* Read the next line into str, an array at least 81 characters in size. */
        fgets(str, 82, fip);
        fprintf(ff16, "Read in line is %s\n", str); /* DEBUG only, remove when solved */
        iOffset = 0;
        for (iField = 0; iField < 9; ++iField)
        {
          if ((iLine * 9) + iField > knri)
          {
              /* If we are past the last significant rate, stop processing, but keep reading. */
              break;
          }
          else if (sscanf(str + iOffset, "%lf%n", &solvsurchgrt[(iLine * 9) + iField], &iNewOffset) < 1)
          {
            /* Alert the user that we have bad input in line (iLine + 1) */
            fprintf(ff16, "Unable to read a valid number at position %d of line %s.\n", iOffset+1, str);
            /* stop reading this line and try the next. */
            break;
          }
          else
          {
            /* Add on the length of the last field so that we start reading the next number after that */
            iOffset += iNewOffset;
            fprintf(ff16, "\n SOLVENCY SURCHARGE RATE = %lf\n", solvsurchgrt[jj-1]);
            fprintf(ff16, " SURCHARGE RATE IS STORED IN ELEMENT %d OF THE LIST OF SURCHARGES\n", jj);
            jj++;
          }
        }
      }
      fprintf(ff16, "\n");

The data that this code is reading follows:

P7
1
0.005 0.007
0.02700 0.02625 0.02525 0.02425 0.02325 0.02225 0.02125 0.02025 0.01925
0.01825 0.01725 0.01625 0.01525 0.01425 0.01350 0.01100 0.00725 0.00475
0.00375 0.00275 0.00175 0.00150 0.00150 0.00150 0.00150 0.00100 0.00000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

card# 164 : Password

LIST OF SOLVENCY SURCHARGE RATES

IS SURCHARGE APPLIED TO THE BASE YEAR ( YES = 1, NO = 0 ) = 1

INITIAL PERCENTAGE = 0.005000, MAXIMUM PERCENTAGE = 0.007000

Read in line is 0.0270 0 25 525 2425 02325 .02225 0.02125 0.0202

SOLVENCY SURCHARGE RATE = 0.027000
SURCHARGE RATE IS STORED IN ELEMENT 1 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.000000
SURCHARGE RATE IS STORED IN ELEMENT 2 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 25.000000
SURCHARGE RATE IS STORED IN ELEMENT 3 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 525.000000
SURCHARGE RATE IS STORED IN ELEMENT 4 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 2425.000000
SURCHARGE RATE IS STORED IN ELEMENT 5 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 2325.000000
SURCHARGE RATE IS STORED IN ELEMENT 6 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.022250
SURCHARGE RATE IS STORED IN ELEMENT 7 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.021250
SURCHARGE RATE IS STORED IN ELEMENT 8 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.020200
SURCHARGE RATE IS STORED IN ELEMENT 9 OF THE LIST OF SURCHARGES

Read in line is 0.0270 0       25      525      2425    02325   .0225  0.02125  0.0202

SOLVENCY SURCHARGE RATE = 0.018200
SURCHARGE RATE IS STORED IN ELEMENT 10 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 5.000000
SURCHARGE RATE IS STORED IN ELEMENT 11 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 25.000000
SURCHARGE RATE IS STORED IN ELEMENT 12 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 625.000000
SURCHARGE RATE IS STORED IN ELEMENT 13 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 1525.000000
SURCHARGE RATE IS STORED IN ELEMENT 14 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 1425.000000
SURCHARGE RATE IS STORED IN ELEMENT 15 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.013500
SURCHARGE RATE IS STORED IN ELEMENT 16 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.011000
SURCHARGE RATE IS STORED IN ELEMENT 17 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.007200
SURCHARGE RATE IS STORED IN ELEMENT 18 OF THE LIST OF SURCHARGES

Read in line is  0.0182 5       25      625     1525    01425   .01350  0.01100  0.0072

SOLVENCY SURCHARGE RATE = 0.003700
SURCHARGE RATE IS STORED IN ELEMENT 19 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 5.000000
SURCHARGE RATE IS STORED IN ELEMENT 20 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 75.000000
SURCHARGE RATE IS STORED IN ELEMENT 21 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 175.000000
SURCHARGE RATE IS STORED IN ELEMENT 22 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 150.000000
SURCHARGE RATE IS STORED IN ELEMENT 23 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 150.000000
SURCHARGE RATE IS STORED IN ELEMENT 24 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.001500
SURCHARGE RATE IS STORED IN ELEMENT 25 OF THE LIST OF SURCHARGES

SOLVENCY SURCHARGE RATE = 0.001500
SURCHARGE RATE IS STORED IN ELEMENT 26 OF THE LIST OF SURCHARGES
Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

Read in line is 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000

The fgets has no problem reading the last six lines of the nine by nine matrix of data, but cannot read the first three lines of that matrix. It seems that the problem is caused by having five significatn digits to the right of the decimal point, but fgets is reading a character string, so that does make any sense. I am struggling to figure this one out, and could use some help. Please note the following:

The variable str is defined as char str[82] earlier in the code, all variables you see are defined earlier in the code or in a .h file.
I have completely verified that there are no tabs in the lines of data.
I have completely verified that the data is no more than 80 columns( other code enforces that).
I have completely verified that there is a line feed at the end of each line read by fgets.
I have tried putting one space between the fields, and two spaces. It fails with both, but works better with one space.
The P7 is read and used to check that the code is reading from the correct place in the file, but the P7 itself is not printed out.

I need to get this code working properly in this file, and then use it in another file. I appreciate any help I can get in solving this.

[rubberman]

1. You need to be sure that the str variable is big enough to handle your input. You might want to consider using getline() instead. Read the man pages. Are you using this in Windows, or Linux/Unix systems?

[JohnMcPherson]

This code is running on a Solaris Unix system. The str variable is defined as char str[82], which should hold 80 columns of data, the line feed, and the null terminator. I will research getline(), and see if it is a better option for reading the data.

[AssertNull]

Lines 39 and 40 look all right to me code-wise. You say you've checked to make sure there are not more than 80 characters in the line. Add the '\n' and the NULL terminator and that takes you to 82 characters, the size of your buffer. It might be worth checking to make sure you don't actually have "\r\n" as your line end, which could potentially screw things up if you have 80 characters in the line, plus '\r' plus '\n' plus the NULL puts you at 83 characters. I doubt that's the problem here. Just something to look at. Make sure there are no weird non-printable ASCII Control characters too. Again, doubtful that's the problem.

As to what is going on and why you are getting the weird output, as mentioned, lines 39 and 40 appear fine. However, since you are reading from and printing to a FILE* and you are inside of a loop, I suggest changing line 40 to print to the screen rather than the file. Then add a couple of getchar() statements after line 40 to slow things down so you can see them. Set str to NULL with memset. And harness the return value from fprintf. Make sure it looks right. So lines 39 to 40 end up as

   memset(str, 0, 82); // clear str to all zeroes.
   fgets(str, 82, fip);
   int c = fprintf(stdout, "Read in line is %s\n", str); /* DEBUG only, remove when solved */
   printf("%d characters read\n", c); // make sure this looks right
   getchar(); // wait for user input just to pause the loop
   getchar(); // wait for user input just to pause the loop

Now see if it prints out right. If it does, then the problem is your output file. If it doesn't, then you have a problem somewhere else, so try changing line 39 to something less than 82, like 5 or 10 or whatever. At some point it will stop working right. And consider making str bigger than 82 just to check, even though you've already checked.

If it prints fine to the screen, have it go through the loop again, see if it keeps printing right each iteration. If it prints to the screen OK, but not to the file, you can investigate that (File permissions, path, whatever).

In other words, nothing in the code sticks out (to me), so you're onto debugging it. If you want to zip the whole thing up and post it, I can compile and run it.

[AssertNull]

To the OP:

Digressing from your original question, it appears that you are trying to read 81 floating-point values into an array of doubles? I am wondering if you can simply skip reading the lines in as strings and parsing the strings into 9 floating point values and instead do an fscanf 81 times straight into the array:

int i;
for(i = 0; i < 81; i++)
{
    fscanf(fip, "%lf", &solvsurchgrt[i]);
}

[JohnMcPherson]

Thank you for the suggestions. I tried using getline(), but unfortunately, I am running on an older compiler, and it does not support getline(). Putting in the suggested code to check how many characters were read from the line, it said there were 90 characters read in, which seems very strange. It also stated that 90 characters were read in from the lines of data it was able to read correctly, so I am not sure what is going on. It may be that the 90 characters are what was returned from fprintf, not what was read by fgets. I used TextPad to look at the data with its Visible option, but the data looked correct in TextPad. I suspect the data is the problem, since I had this code working before with similar data, but I need to be sure. Is there another tool available to look at the data in detail, and make corrections if needed, besides TextPad?

[AssertNull]

Programmer's Notepad and Notepad++ are both free. If you open up a text file with non-printable or strange characters (i.e. control characters or characters in a non-English language), they'll be visible and you can delete them. If you have more than one space or a tab between numbers, you can delete them as well.

Alternatively, from a programming point of view, you could read a line into a larger than 82 character buffer and write a function that strips out any characters that are not digits, a decimal point, or a newline and return that stripped string. The ctype.h library will help there.

That said, looking at your printout above (re-pasted below with code tags), you appear to have quite a few extra spaces in there. If that was indeed your output from running the program, you definitely have extra spaces in there which take you beyond 82 characters. Does it show up in the text editor like it shows up below? This is a copy/paste from your initial post.

Read in line is  0.0182 5       25      625     1525    01425   .01350  0.01100  0.0072

[AssertNull]

Note to my last post. Notepad++ and Programmer's Notepad are both Windows programs. I don't know if they have Solaris/Unix/Linux versions. Worst case scenario, I guess you could edit on Windows and transfer to your computer.

[Freshly]

Have you tried?

fgets(str, sizeof(str), fip);

[JohnMcPherson]

Thank you for all the suggestions. It turns out the problem was the five significant digits to the rights of the decimal point. I changed the data to be one hundred times as large, which reduces the significatn digits to three right of the decimal point, and then divide it by one hundred in the code when appropriate, and multiply it by one hundred when I write back to the file. I don't know why fgets and fscanf have a problem with five significant digits to the right of the decimal point. I can close this thread as solved, but first let me know if you want to discuss the five significant digit issue further.

[AssertNull]

getline is not standard in C. It is standard in C++. So if you're using a C compiler, it's not surprising that it's not there. I'm not sure that it has anything to do with your compiler being "old". If you use C++ rather than C, you can use getline on any compiler, old or not, I believe. Regarding whether to mark the problem solved, that is up to you. You can solve your problem just fine using the standard C functions, however. Again, please make sure that you have indeed verified that the input file is 80 characters or less as you say other code has verified. See my post above regarding that. It at least looked to me like there was some spacing in there that might make it longer than you thought. Worth checking. Also note that control characters count as characters and you can't always see them in all text editors. Again, see posts above.

As far as having five significant digits after the decimal point, I really don't see why that would pose a problem. Multiplying and dividing by 100 seems unnecessary and as such seems like it's masking some other problem. Make sure you are using doubles, not floats for more accurate storage.

My advice would be to mark this particular thread solved even if it is not. If it is not, then start a new thread with updated code, updated data, and an updated question. The reason to do this is because when threads get long like this one, people who might have answers might not chime in if they have not already because they have to read through the whole thing and they figure that whoever is already helping will see it through, which is not always the case, particularly a thread like this that was started a month ago. I, for example, had forgotten about this thread and just happened upon it by chance and checked it out since it seemed vaguely familiar.