problem in converting txt to binary

[yashyash]

Dear experts
I have a large file with this format:

1 7252 0.1613
1 7253 0.1613
1 7254 0.0645
1 7255 0.0409
1 7256 0.0394
1 7257 0.1872
1 7258 0.1872
1 7259 0.0986
......

and I want to change this file to the binary format but it does not work for all file. it would only convert the first 100 kb of the file. here is my code.

#include <stdlib.h>

#include <stdio.h>

#include <sys/stat.h>

#include <fstream>

#include <iostream>


using namespace std;




int convertBack(char* binfile, char* asciifile);

int convertForth(char* asciifile, char* binfile);



void printUsageAndExit(){

		cout << "Usage" << endl;

		cout << "conversion h\t\t\t\t\t(print this help)" << endl;

		cout << "conversion b <binfile> <asciifile>\t\t(convert bin -> ascii)" << endl;

		cout << "conversion f <asciifile> <binfile>\t\t(convert ascii -> bin)" << endl;

		exit(0);

}



int main (int argc, char* argv[]){

	if (argc<3||argv[1][0]=='h'){

		printUsageAndExit();

	}

	else if (argv[1][0]=='f') convertForth(argv[2],argv[3]);

}


int convertForth(char* asciifile, char* binfile){

	cout << "converting forward" << asciifile << "->" << binfile <<endl<<flush;

	ifstream infile (asciifile);

	ofstream outfile (binfile);



	int index1;

	int index2;

	float simvalue;
        while( infile >> index1 >> index2 >> simvalue )
{

    outfile.write((char *)&index1, sizeof(index1));
    outfile.write((char *)&index2, sizeof(index2));
    outfile.write((char *)&simvalue, sizeof(simvalue));
}

	infile.close();

	outfile.close();

}

any suggestion is very very much appreciated. Best regards and thanks in advance.

[Ancient Dragon]

how do you know it does not convert the entire file? Did you write another program to read back all that binary data and compare it with the original file?

[yashyash]

how do you know it does not convert the entire file? Did you write another program to read back all that binary data and compare it with the original file?

Yes, I'm sure because of two reasons: 1- the size of the converted file is about 80 kbs which is very small in compare with the original file which is about 600 MB ...
2- I wrote something to convert it back and it only converted the first 1000s lines of the file.
any idea is very appreciated.

[__avd]

open file with ios::binary mode.

ofstream outfile;
outfile.open(binfile,ios::out | ios::binary);

[yashyash]

open file with ios::binary mode.

ofstream outfile;
outfile.open(binfile,ios::out | ios::binary);

the problem with this is that the process of converting does not end and the file output reaches few GB till you manually stop it. so I dont think this is a solution since I've tried it before.

[__avd]

Please write suitable code to solve your problem and if you got problem then post your problem.

[yashyash]

Please write suitable code to solve your problem and if you got problem then post your problem.

but any solution so my problem and my codes? any C monk here? ;)

[__avd]

but any solution so my problem and my codes? any C monk here? ;)

Had to open your file in binary mode?

[yashyash]

Had to open your file in binary mode?

yes but it goes to an unfinishing process and the bin file reaches about few GB of size :(

[__avd]

Never say thanks :D This code is written by you.

#include <stdlib.h>
#include <stdio.h>
#include <sys/stat.h>
#include <fstream>
#include <iostream>

using namespace std;
int convertBack(char* binfile, char* asciifile);
int convertForth(char* asciifile, char* binfile);

void printUsageAndExit(){
		cout << "Usage" << endl;
		cout << "conversion h\t\t\t\t\t(print this help)" << endl;
		cout << "conversion b <binfile> <asciifile>\t\t(convert bin -> ascii)" << endl;
		cout << "conversion f <asciifile> <binfile>\t\t(convert ascii -> bin)" << endl;
		exit(0);
}

int main (int argc, char* argv[]){
	if (argc<3||argv[1][0]=='h'){
		printUsageAndExit();
	}
	else if (argv[1][0]=='f') convertForth(argv[2],argv[3]);
}


int convertForth(char* asciifile, char* binfile){
	cout << "converting forward" << asciifile << "->" << binfile <<endl<<flush;

	ifstream infile (asciifile);
	ofstream outfile;
                outfile.open(binfile,ios::out | ios::binary);

	int index1;
	int index2;
	float simvalue;
        while( infile >> index1 >> index2 >> simvalue )
{
    outfile.write((char *)&index1, sizeof(index1));
    outfile.write((char *)&index2, sizeof(index2));
    outfile.write((char *)&simvalue, sizeof(simvalue));
}
	infile.close();
	outfile.close();
}

[yashyash]

Never say thanks :D This code is written by you.

#include <stdlib.h>
#include <stdio.h>
#include <sys/stat.h>
#include <fstream>
#include <iostream>

using namespace std;
int convertBack(char* binfile, char* asciifile);
int convertForth(char* asciifile, char* binfile);

void printUsageAndExit(){
		cout << "Usage" << endl;
		cout << "conversion h\t\t\t\t\t(print this help)" << endl;
		cout << "conversion b <binfile> <asciifile>\t\t(convert bin -> ascii)" << endl;
		cout << "conversion f <asciifile> <binfile>\t\t(convert ascii -> bin)" << endl;
		exit(0);
}

int main (int argc, char* argv[]){
	if (argc<3||argv[1][0]=='h'){
		printUsageAndExit();
	}
	else if (argv[1][0]=='f') convertForth(argv[2],argv[3]);
}


int convertForth(char* asciifile, char* binfile){
	cout << "converting forward" << asciifile << "->" << binfile <<endl<<flush;

	ifstream infile (asciifile);
	ofstream outfile;
                outfile.open(binfile,ios::out | ios::binary);

	int index1;
	int index2;
	float simvalue;
        while( infile >> index1 >> index2 >> simvalue )
{
    outfile.write((char *)&index1, sizeof(index1));
    outfile.write((char *)&index2, sizeof(index2));
    outfile.write((char *)&simvalue, sizeof(simvalue));
}
	infile.close();
	outfile.close();
}

thats the problem :) I cant figure out why my file which is 600 mg is converted not completely to bin file to a 86 kb file which is only first few words. when the file is small it works well, but for a big file it does not work. that was my main problem of creating this thread. any suggestion?

[Ancient Dragon]

The bin file might be smaller than the text file because each integer written in binary mode only requires 4 bytes of file space (sizeof(int)) or 6 bytes for a float sizeof(float), while the same integer in a text file will require one byte per digit plus spaces between numbers and line terminating characters '\n'. The only way to predict the size of the binary file is to count up the size of all the integers, floats and doubles.

It might be interesting if you would zip of the text file and post it so that we had something more concrete to work with.

[yashyash]

The bin file might be smaller than the text file because each integer written in binary mode only requires 4 bytes of file space (sizeof(int)) or 6 bytes for a float sizeof(float), while the same integer in a text file will require one byte per digit plus spaces between numbers and line terminating characters '\n'. The only way to predict the size of the binary file is to count up the size of all the integers, floats and doubles.

It might be interesting if you would zip of the text file and post it so that we had something more concrete to work with.

dear Dragon
thanks for your reply, the text file is about 700 mg and its impossible to be sent but you cold easily generate it based on the template I sent in first post by repeating it 1 milion times in a text file. its something like this:

1 7252 0.1613
1 7253 0.1613
1 7254 0.0645
1 7255 0.0409
1 7256 0.0394
1 7257 0.1872
1 7258 0.1872
1 7259 0.0986

but millions of lines. first and the second columns are integer ranged from 1 to 8000 and the third column is from -1 to 1. you simply can generate a toy version of it which is big enough. I tried many different ways but still not able to fix it. Thanks very much again.

[Ancient Dragon]

I wrote this program to generate the text file

int main()
{
    srand( (unsigned int)time(0));
    double x = 0;
    ofstream out("data.txt");
    for(int i = 0; i < 8000000; i++)
    {
        x = (double)rand() / (double)RAND_MAX;
        out << i+1 << " " << i+7000 << " " << x << "\n";
        if( i > 0 && (i % 100000) == 0)
           cout << i << "\n";
    }
}

And this one to convert it to bin file

int convertForth(char* asciifile, char* binfile);


int main (int argc, char* argv[])
{

    convertForth("data.txt", "data.bin");

}

int convertForth(char* asciifile, char* binfile){

	cout << "converting forward" << asciifile << "->" << binfile <<endl<<flush;

	ifstream infile (asciifile);

    ofstream outfile (binfile, ios::binary);


    time_t t1, t2;
	int index1;

	int index2;
    int i = 0;
	float simvalue;
    t1 = time(0);
    while( infile >> index1 >> index2 >> simvalue )
    {

        outfile.write((char *)&index1, sizeof(index1));
        outfile.write((char *)&index2, sizeof(index2));
        outfile.write((char *)&simvalue, sizeof(simvalue));
        if( i > 0 && (i % 100000) == 0)
           cout << i << "\n";
        i++;

    }

	infile.close();
	outfile.close();
    t2 = time(0);
    cout << "Execution time: " << t2 - t1 << " seconds\n";
   return 0;
}

The original text file was 200 meg and the bin file 94K. The second program took 112 seconds to convert from text to binary.

With an 800meg text file you might want to check your hard drive to see if it contains sufficient space to write out the data. Your description of the problem sounds like your hard drive might be full.

[yashyash]

I wrote this program to generate the text file

int main()
{
    srand( (unsigned int)time(0));
    double x = 0;
    ofstream out("data.txt");
    for(int i = 0; i < 8000000; i++)
    {
        x = (double)rand() / (double)RAND_MAX;
        out << i+1 << " " << i+7000 << " " << x << "\n";
        if( i > 0 && (i % 100000) == 0)
           cout << i << "\n";
    }
}

And this one to convert it to bin file

int convertForth(char* asciifile, char* binfile);


int main (int argc, char* argv[])
{

    convertForth("data.txt", "data.bin");

}

int convertForth(char* asciifile, char* binfile){

	cout << "converting forward" << asciifile << "->" << binfile <<endl<<flush;

	ifstream infile (asciifile);

    ofstream outfile (binfile, ios::binary);


    time_t t1, t2;
	int index1;

	int index2;
    int i = 0;
	float simvalue;
    t1 = time(0);
    while( infile >> index1 >> index2 >> simvalue )
    {

        outfile.write((char *)&index1, sizeof(index1));
        outfile.write((char *)&index2, sizeof(index2));
        outfile.write((char *)&simvalue, sizeof(simvalue));
        if( i > 0 && (i % 100000) == 0)
           cout << i << "\n";
        i++;

    }

	infile.close();
	outfile.close();
    t2 = time(0);
    cout << "Execution time: " << t2 - t1 << " seconds\n";
   return 0;
}

The original text file was 200 meg and the bin file 94K. The second program took 112 seconds to convert from text to binary.

With an 800meg text file you might want to check your hard drive to see if it contains sufficient space to write out the data. Your description of the problem sounds like your hard drive might be full.

Dear Dragon

Indeed there is no problem of space in my hard disk since i have about 400 GB free space. However, have you changed your binary file back to the txt file? Indeed, if you change it back you see that not all the file is converted but only the first part is converted. That's why it is only few kbs in compare to 200 meg. That's exactly the problem I'm facing. it is converted and the binary file size is less than 100 kb but its not the full conversion. again thanks for ur helps :)

[ArkM]

I have created dummy text file like OP source (~572 Mb, 40000000 lines) then converted it to binary form with VC++ 2008 for ~3.5 minutes without special buffer control. That's OK, no problems.

1. Open both files in binary mode.
2. Inspect source file with hex editor. Probably it's corrupted. Test your disk drive integrity.

What compiler are you using?

[Ancient Dragon]

>>However, have you changed your binary file back to the txt file?
Yes, and both files were exactly the same (other than one being text and the other binary).

#include <iostream>
#include <fstream>
#include <string>
using namespace std;

template <class T>
bool getnum(ifstream& in, T& num)
{
    in.read((char*)&num, sizeof(T));
    if( in.gcount() != sizeof(T))
    {
        cout << "Read error\n";
        return false;
    }
    return true;
}

int main()
{
    ifstream in1("data.txt");
    ifstream in2("data.bin", ios::binary);

	int index1;
	int index2;
    int i = 0;
	float simvalue;
    while( in1 >> index1 >> index2 >> simvalue )
    {
        int a, b;
        float c;
        if( !getnum( in2, a) || !getnum( in2, b)
            || !getnum( in2, c) )
        {
            cout << "read failed on line " << i << "\n";
            break;
        }
        ++i;
        if( a != index1 || b != index2 || c != simvalue)
        {
            cout << "Values are not the same\n";
            cout << a << " " << index1 << "\n";
            cout << b << " " << index2 << "\n";
            cout << c << " " << simvalue << "\n";
            break;
        }
        if( (i % 16000) == 0)
            cout << i << "\n";
    }
	return 0;
}