Efficient method to transpose big files

[sadsdw]

Hi friends,

I'm looking for an efficient method to transpose rows and columns from a BIG file.
I wrote this code but is very inefficient. In that, I'm closing and opening a file many times. (// #*#)

Could anyone help me please?

Thanks a lot!

ifstream testTranspose;
	testTranspose.open("data.txt");
	int nrows = 0;
	string lines;
	while(!testTranspose.eof())
	{
		nrows++;
		getline(testTranspose,lines);
	}
	
	testTranspose.close();
	ifstream testTransposeCol;
	testTransposeCol.open("data.txt");
	string colLineString = "";
	string buf2;
	getline(testTransposeCol,colLineString);
	std::stringstream ss2(colLineString);
	int ncol=0;
	
	while(ss2 >> buf2)
	ncol++;

	testTransposeCol.close();

	// INVERTING ROWS AND COLUMNS
	
	ifstream IN_transposeRowColumn;
	ofstream OUT_transposeColumnRow;
	OUT_transposeColumnRow.open("OUT_ColumnRow.txt");
	string row = "";
	string bufSs3 = "";
	vector <string> lineToInvert;
	
	for(int nr=0;nr <ncol;nr++)
	{
		IN_transposeRowColumn.open("data.txt"); // #*#
		while(!IN_transposeRowColumn.eof())
		{
			getline(IN_transposeRowColumn,row);
			std::stringstream ss3(row);
			while(ss3 >> bufSs3)
				lineToInvert.push_back(bufSs3);
			
			OUT_transposeColumnRow << lineToInvert[nr] << " ";
			lineToInvert.clear();
			row = "";
		}
		OUT_transposeColumnRow << endl;
		IN_transposeRowColumn.close(); // #*#
	}

[L7Sqr]

Instead of opening and closing the file in a loop, why not just rewind the file (this will require you to use C-style IO)
I'm not convinced that that is your only bottleneck but I have not looked very hard at your code either. Are you sure that that is the choke point? If so, how have you determined that?

[Ancient Dragon]

The loop on line 37 will cause the stream to read the last line in the file twice. What you want is like below -- not necessary to test for eof() because getline() will stop when eof() is encountered.

while( getline(IN_transposeRowColumn,row) )
{
   // other code goes here
}

line 44: what is variable nr? I don't see it declared anywhere. And if nr is an integer then that line will only output one string, not the whole vector of strings. Is that what you want?