Runtime abort when using different input file

[toneranger]

Hi,

I'm wondering what could cause this type of runtime error:

My program basically reads data from different kinds of data files, either a minute data file or a 5 minute data file and then does some analysis. When I run the program on the 1 min data, no problem. BUt when I run it on the 5 min data, I get a runtime error message/assertion failure that says:

Line 163
**
Expression: ("_Myptr + _Off <= ((Myvec* this->_Getmycont()))->_Mylast &&_Myptr + _Off>= etc**

Line 163 simply states:

163     std::vector<SwingIdentified>::iterator SwingIter; 
164 for (SwingIter= NamedSwings.begin(); SwingIter!=NamedSwings.end(); SwingIter++) 
165 {   
166     outfile << NamedSwings << std::endl;
    }

Any idea why the compiler would spit this out when I run the 5 min data file and not the 1 min data file? I tried rebooting my PC and tried looking into the data files to see if there were any glaring problems with the 5 min compared to the 1 min but can't tell of any.

Thanks,
TR

[David W]

Perhaps the longer data file is too long to handle?

Try splitting it up into 1/2's and see if each 1/2, by itself, processes ok?

Is you output file looking correct for the smaller file that you said seems to run ok?

I would have thought your code would be more like this:

std::vector<SwingIdentified>::iterator SwingIter;

for (SwingIter= NamedSwings.begin(); SwingIter!=
NamedSwings.end(); SwingIter++)
outfile << *NamedSwings << std::endl;

[toneranger]

Hi David,

Thanks much for your thoughts. Well, both files are small samples, covering a few days worth of data only. I tried the indirection above but that caused a compile error, so it's not the code above.

I'm attaching the two files below

[David W]

Did you look at the two .txt files with a text editor?

There is something glaringly wrong with the 2nd longer file ... it seems quite corrupted, just a little way into it ...

Using an editor ... like notepad ... or programmers notepad ... take a look !

[toneranger]

David,

I looked at the five mintest.txt file in notepad and didn't notice anything unusual. What I see is two days worth of data, spanning 9/9/2010 to 9/10/2010. Did you see something else?

Thanks.

[David W]

The longer (2nd) file length is given as 194.93 KB

The shorter (1st and good) file length is given as 78.12 KB

Did you not see 'a problem' with (all) the (missing) contents of the (supposed to be) longer file?

Note: I am looking at/comparing the two .txt files using my iPhone.

[nullptr]

The 5 minute data is only valid for the first 10420 bytes.
Thereafter, the data is padded with the following sequence of bytes:

0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x0D 0x0A

[toneranger]

David and Nullptr,

Thanks. Nullptr, sorry for the silly question, how did you get to 'see' those bytes? And what can I do to fix the file? I basically have the data in a much larger excel file that spans 3 months. So if I only want to use a few days of it say, is copy/paste not the right method? Should I instead try deleting all unwanted data first in the excel file and then saving as a txt file?

[toneranger]

I attached a new 5 min file i tried with, but no luck. I made this one to be only a few days long with no extraneous lines in it. But still aborting at runtime, frustrating.

[David W]

Without seeing all the code you are using ... we can only guess at the problem(s), according to our own experience.

Inspect the good (working ok) 1st file ... by loading it into a text editor, like notepad, and see if there is one and only one newline char at the end of the last line of data there ... or if instead, that last line ends with EOF ? (i.e. NO newline char there.)

Then, make sure, (using a text editor), that the 2nd file ends like the first.

This will resolve the problem if the issue is (only) related to how you program reads lines of data and in particular the last line.

Another possible issue is the possibility of unrecognized char's being embedded in the 2nd data file chunk ... and then, the program may be halting with an exception thrown there ... but check out first idea above .. firstly ... as that 'data file (corruption and/or ending) problem' is suggested by the original two data .txt files you presented.

[nullptr]

Shouldn't line 163 - 167 be:

    std::vector<SwingIdentified>::iterator SwingIter; 
    for (SwingIter= NamedSwings.begin(); SwingIter!=NamedSwings.end(); SwingIter++) 
    {   
        outfile << *SwingIter << std::endl; // <==
    }

[toneranger]

Hi David,

I found this code below online and gave it a try to read the last line in each of the two files. In the file that works, "csv file.txt", the output to the below is nothing, i.e., an empty line. In the file that doesn't work, "09five.txt", the output for the last line is the last line of data:

NQ05MIR 5 12/14/2009 1610 1806.25 1808 1805.75 1807.5 2698 1806.25

So, does that mean that in the case of the "csv file.txt", the last line ends with either a /n or newline?

Flying blind here, could really use your help/code if you can share. Thanks.

int main(int argc, char const* argv[])
{
    std::ifstream read ("csv file.txt", std::ios_base::ate);
    std::string tmp;
    int length = 0;

    char c = '\0';

    if (read)
    {

        length = read.tellg();

        for (int i = length-2; i> 0; i--)

        {
            read.seekg(i);
            c = read.get();
            if (c =='r' || c == '\n')
                break;
        }

        std::getline(read, tmp);//read last line
        std::cout << tmp << std::endl; // print it

    }

    return 0;

[toneranger]

Nullptr,

I changed the NamedSwings to the dereferenced iterator as you suggested, but still the same abort at runtime. I checked out what you said earlier about this line:

0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x09 0x0D 0x0A

And it looks like it's a bunch of \n \r and \t

not sure what to do from here

[David W]

So add a 'newline' char to the end of the bad '2nd' file to make it match the ending in the 'good 1st' file and try it then.

[toneranger]

Tried it with another 5 min file which ends with a blank line and no luck, still aborts. I've attached below:

So, I noticed that the code gets through the first function in my "main", because it appends the data to a file, but it doesn't make it to my moving average function (below) because it doesn't create the "MA.txt" file. Do you see anything glaringly wrong below?

int CalcMovingAverage (int a, std::vector<PriceInfo>& priceinput)

{

std::vector<PriceInfo>::iterator itr; 
itr = priceinput.begin();
std::string first_Date;
first_Date = itr->Date;

int length = a;
int count = 0;
double sum = 0;


if (itr->Date == first_Date) 


for (;itr!= priceinput.end();) 

{
    sum += itr->Close;
    count ++;
    if (count >= length)

    {
        itr->MovingAve = (sum / (double) length);
        sum -= (itr-(count-length))->Close;

    }



    else

    {
        first_Date = itr->Date;

        }


  ++ itr;

 }


    std::ofstream outfile2;
    outfile2.clear ();
    outfile2.open ("MA.txt", ios::app);
    std::vector<PriceInfo>::iterator it2; 
    it2 = priceinput.begin();

    for (it2= priceinput.begin(); it2!=priceinput.end(); it2++) 

{   
    outfile2 << priceinput << std::endl;
}

    return 0;
}

[David W]

What does your program do?

Could you resend two files:
1st good file that process ok
2nd bad file that crashes

[toneranger]

the program reads in the data from the txt file, then it does some analysis on the data in the main function, outputting to a txt file. Then there is more analysis i.e., the moving average function (which i posted above) which also outputs to a txt file, the "MA.txt" (see code above). When I try the 5 min data, and check the 2 output files, the first one is good but the 2nd txt file, i.e., the MA.txt doesn't generate at all. Could that be the source of the abort? It does when I try the 1 min data.

I'm reattaching both files. The first one is 1 min data in the "csv file.txt". The second is 5 min data in the "five09.txt". Both files end with newlines as far as I can tell. I just tried again, and verified again that the 1 min data file is working as expected, no aborts there. But the 5 min isn't working.

[David W]

If you are writing to a data .txt file ... and then trying to read that file ... before closing and reopening it ... that also will be a problem.

You can write a short program to validate the data in your .txt files ... each line seems to have 10 regularly occurring fields ... that you can confirm if all exist ok. If data files are all fields ok ... then you have isolated the problem to be with the programs.

[toneranger]

that's why I chose small input data files, so I could manually eyeball them. The 5min data file looks fine. The issue must be the program because it outputs the first analysis output file to the directory, but it doesn't output the moving average output file.

Actually on second thought: the above can't be because the program works fine with the 1 min data file, it outputs the moving average txt file just fine.
Wow, runtime errors are fun!

[David W]

'Eyeballing' is not sufficient here ... You need to validate each line and every field in each line, with a short data validation program that matches the data field requirements of your original data processing programs. That fairly short and easy to code program can just read every line till done ... then using perhaps stringstream objects ... or a custom 'split' function ... parse every field in that line to make sure all fields there are expected (and each is correct type and in valid acceptable range) and thus that no extra or corrupted fields are present.

[David W]

Here is a link to a C++ split function that you might like to use in your data validation program. Reading the size of the list formed from each split up line ... will tell you right away how many fields (strings) were present in each line ... then you can traverse the list of strings for that line ... validating that each is 'acceptable' ... or not.

http://developers-heaven.net/forum/index.php/topic,2584.msg2930.html#msg2930

[David W]

Both of your test files ...

https://www.daniweb.com/attachments/2/d01478039506623f9c6fedd582703af8.txt

https://www.daniweb.com/attachments/2/da9dcdde1e01ed1698cce4689c5ff447.txt

... seem to be valid when (only roughly) tested like this:

// fileReadValidate.cpp //  // 2015-09-01 //

#include <iostream>
#include <fstream>
#include <string>
#include <sstream>

using namespace std;

const char* FILE_IN  = "five09_15.79 KB.txt"; // "csv_file_78.11_KB.txt"; // // "five09_15.79 KB.txt"; //

template< typename T >
bool isValid( const string& str )
{
    T tmp;
    istringstream iss( str );
    if( iss >> tmp ) return true;
    return false;
}

bool isValidIntRange( const string& str, int min, int max )
{
    istringstream iss( str );
    int tmp;
    iss >> tmp;
    if( tmp < min || tmp > max )
    {
        cout << "Date field value " << tmp << " not in range " << min << ".." << max << '\n';
        return false;
    }
    return true;
}

bool isValidDate( const string& str )
{
    if( str.size() == 10 )
    {
        if( str[2] == str[5] && str[2] == '/' )
        {
            if( !isValid< int >( str.substr(0,2) )
                || !isValid< int >( str.substr(3,2) )
                || !isValid< int >( str.substr(6,4) ) )
            {
                cout << "Invalid date field ... non integer day or month or year ...\n";
                return false;
            }
            else
            {
                if( !isValidIntRange( str.substr(0,2), 1, 12 ) )
                {
                    cout << "Invalid month field\n";
                    return false;
                }
                if( !isValidIntRange( str.substr(3,2), 1, 31 ) )
                {
                    cout << "Invalid day field\n";
                    return false;
                }
                if( !isValidIntRange( str.substr(6,4), 1900, 3000 ) )
                {
                    cout << "Invalid year field\n";
                    return false;
                }

                return true;
            }
        }
        else
        {
            cout << "Invalid date field ... missing '/' ...\n";
            return false;
        }
    }
    else
    {
        cout << "Invalid date size of " << str.size() << endl;
        return false;
    }
}


bool isValidLine( const string& line )
{
    istringstream iss( line );;
    int count = 0;
    string tmp;
    while( iss >> tmp )
    {
        ++count;
        if( count == 2 || count == 4 || count == 9 )
        {
            if( !isValid< int >( tmp ) )
            {
                cout << "Invalid field at " << count << endl;
                return false;
            }
        }

        else if( (count >= 5 && count <= 8) || count == 10 )
        {
            if( !isValid< double >( tmp ) )
            {
                cout << "Invalid field at " << count << endl;
                return false;
            }
        }

        else if( count == 3 )
        {
            if( !isValidDate( tmp ) )
            {
                cout << "Invalid field at " << count << endl;
                return false;
            }
        }
        else // count == 1 //
        {
            if( tmp != "NQ01MIR" && tmp != "NQ05MIR" )
            {
                cout << "Invalid field at " << count << endl;
                return false;
            }
        }
    }

    if( count != 10 )
    {
        cout << "Invalid field count of " << count << endl;
        return false;
    }

    return true;
}




int main()
{
    ifstream fin( FILE_IN );
    if( fin )
    {
        string line;
        int count = 0;
        while( getline( fin, line ) )
        {
            ++count;
            if( !isValidLine( line ) )
            {
                cout << "Line " << count << " was NOT valid\n";
                break;
            }
        }

        fin.close();
        cout << "Done now." << endl;
    }
    else
        cout << "There was a problem opening file " << FILE_IN  << endl;
}

So the problem, then, seems to be with your program.

[David W]

Late edit to above: better to substitute above function with this function ...

template< typename T >
bool isValid( const string& str )
{
    T tmp;
    istringstream iss( str );
    if( iss >> tmp
    && iss.eof() ) // The 2nd test ensures entire string converted. //
    {
        return true;
    }
    // else ...
    cout << "Field value " << str << " not valid\n";
    return false;
}