Custom Binary Format

[Doogledude123]

So, I've written a Custom Binary Format outline but I'm not sure how to translate it to code. I was looking for a few tips on where to get started?

Here's my outline.

MAPDATA
{
    signature  // U24  0x4D, 0x41, 0x50 (MAP)
    version    // U8   0x01
    width      // U16  (MAX of 10000)
    height     // U16  (MAX of 10000)
    tileCount  // U32
    tileList   // TILE[tileCount]
}

TILE
{
    id         // STRING
}

STRING
{
    dataLength // U16
    data       // U8[dataLength]
}

[mKorbel]

I was looking for a few tips on where to get started?

• is possible define MAPDATA as enum

• depends of design I'm miss there tileCount == U32 and U32 == tileCount

[Doogledude123]

Okay, it seems that this question isn't quite elaborate enough. Here's some more info.

I need a way to save maps, and savedata. To do this, I thought about XML or JSON but after some research I figured out that a binary file exported to my own file extension would be the way to go because apparently it's much faster.

So I looked at what it takes to make one and you simply need to design the layout of what was needed, then set up the code to save the bytes to a file. I'm assuming I need to actually code the design, then code a save method which saves the bytes from my editor, or something of the sorts, in the order of which my design is laid out.

This is where I'm stuck, how do I code the values and stuff as those specific data types? and what would I use for saving the bytes to a file?

[JamesCherrill]

I question the need for your own special file format ("much faster" probably means you will save a few microSeconds), but anyway...
You can use a DataOutputStream to write java primitives, byte arrays, and Strings to a binary file as a sequence of bytes in their normal internal binary format. And DataInputStream to read them in again.

DataOutputStream out = new DataOutputStream...
out.writeByte(0x4D);
out.writeByte(0x41);
...
out.writeShort(width);
...
out.writeUFT(id);

...

[Doogledude123]

When the map sizes are huge, apparently it makes a difference.

Question now is how do I declare each variable as the size I need it to be?

PS: I am using Java 8

[JamesCherrill]

int = 32 bits
short = 16 bits
byte = 8 bits
byte[n] = 8*n bits
etc

But check out the API - writeByte, writeShort, writeInt all take an int as parameter, so it's the method, not how the vars are declared that matters

[~s.o.s~]

I thought about XML or JSON but after some research I figured out that a binary file exported to my own file extension would be the way to go because apparently it's much faster.

I agree about the text v/s binary part. But any reason you are rolling out your own format instead of using protobuffers?

Anyway, keep in mind that data written using DataOuputStream can only be read using DataInputStream i.e. it is portable across Java versions but not portable across languages/runtimes. If you want to be able to read your map data in C++ or Python let's say, you need to write out raw bytes to the file stream (FileOutputStream). As as example look at how Java's implemention of zip file format write works.

[JamesCherrill]

"data written using DataOuputStream can only be read using DataInputStream i.e. it is portable across Java versions but not portable across languages/runtimes"

Were you thinking of ObjectOutputStreams? DataOutputStream just writes the bytes you tell it to write in the order you tell it. You just need to know that integers are big-endian. There's no reason why you cannot read or write the file from another language or runtime.

(But having said that, if you feel the need for your own binary file, then Google protocol buffers would be an obvious option)

[Schol-R-LEA]

Wouldn't make sense to have the class implement Serializable rather than hard-coding the file output? I haven't used Java serialization before, so I don't know the limitations and weaknesses of it, but I would expect that using that would save a good deal of effort and make the code less brittle.

[JamesCherrill]

I think the problem here is that Doogledude is concerned that there will be a performance issue around saving and restoring his game state, and is trying to fix it in advance. Until the game is complete and tested the exact data to be stored will remain uncertain. There are also unanswered questions about whether his files will need to be compatible across Java versions, or with other languages.
In my opinion it's another case of premature optimisation. It's an irrelevant diversion away from actually getting the game complete and running. I would advise him to define an interface for saving and restoring the state, and supply the simplest implementation possible (which is, yes, to make the classes Serializable and write the whole thing to an object output stream).
As and when the requirements are complete, if and when there is a performance issue, then that may be the time to replace the initial implementation.

[Doogledude123]

James; I've done research from other tile based games written in Java and after reading what I have read I have concluded that I would rather just spend the time and learn Binary stuff with Java anyway. I appreciate the concern when it comes to premature optimization however.

SOS; I have read that using protobuffers takes a lot of fooling around with and if you change things with the class then the format breaks, rendering old versions useless.

Schol-R-LEA; Also tends to break old versions after changing something.

I'm choosing this way for multiple reasons, I could code in a simple way to read different versions of the saves and therefore keep support for old files. Also, I want to dive in and learn something new, which is a good chunk of why I'm doing this project in the first place.

[~s.o.s~]

I have read that using protobuffers takes a lot of fooling around with and if you change things with the class then the format breaks, rendering old versions useless

Not really true about the version bit. It takes the same amount of "fooling around" like any other technology but YMMV.

Also, I want to dive in and learn something new, which is a good chunk of why I'm doing this project in the first place

This is a good enough reason for implementing your own binary format for maps as I believe it's a good exercise. Bonus points for trying to make sure your map data can be read from other languages/operating systems.

[JamesCherrill]

If your intent is to invest some time to learn about binary files in Java then I'm 100% happy with your plan.
J