This a bit of a lengthy problem to explain, please bear with me.
Read in a data file (written in ASCII) and convert it to a smaller format (such as binary)
I start off with a proprietary format called Heim RawData. I currently don't have access to how they pack their data, but I don't think I'm allowed to share even if I do get it. Each raw file is approximately a gig in size; their proprietary software can take a sample of that (I usually take between 1 and 10 seconds worth) and convert it into an ASCII data format. We're trying to do data analysis so the more data we can look at once the better, but by converting it into ASCII each number/letter is represented with 8 bits therefore increasing the use of memory drastically. I'm trying to automate the system so we don't have to use excel, limited to only 60000 some values or 1000000 if using 2007 for a part of our analysis.
Typically when writing to a file, C generally prints in ASCII correct? Is it possible to write in a binary mode with C? Is there a better language to do this kind of data packaging? I'm only using C because the analysis team will most likely be using C as well. What kind of formating would be best?
Even if I write as in HEX notation, doesn't it still write to the file in ASCII. I don't think that will substantially reduce my file size and I'll have to convert back to decimal for analysis.
If someone can provide me with some insight or recommendations that would be greatly appreciated.