The Mooncraft / Lunacraft File Format

(This is an article I originally wrote for the Lunacraft Wiki on 2011 November the 12^th. The current version, Userblog:Mrob27/InternalData_Format, may be slightly different.)

- - -

NOTE: The information in this article was determined from files created by version 1.7.0. It does not necessarily apply to other versions. Also, your lunacraft files will not be backed up after you upgrade your device to iOS 5.0.1.

I have now become sufficiently interested in this genre of gaming that I'd like to be able to export my saved game data to other programs in the future.

So I've started looking at the data that the game saves when you create and/or save the moon.

Contents

Mobs and Other Non-Grid Entities

Practical Applications

---

Introduction

In general, this is done by having two different ways to look at the world. Usually the game itself is one way of looking at the world, then you use some other method (like looking in memory or looking at saved files, etc.). You make changes in the game and see how it affects the files, or you change the files and see how it affects the game.

In this case I don't want to change the world, I just want to figure out how to make a copy of it, maybe in a plain text file of some kind that would be suitable for converting into some other program's format years from now.

Prior to the introduction of iOS 5.0.1 and the new iTunes-free "backup to iCloud" feature, there was an easy way to get at the saved game files for any iPad, iPhone, or iPod Touch game. Owners of such devices can connect them to a PC or a Mac, and use the iTunes program to backup and restore the data on the device. Backups are very important for handling situations like a lost or broken iPhone.

In order to avoid filling the limited iCloud storage space with its huge map data files, lunacraft tags all its files with the "do not back up" attribute. This attribute is described here: folder marked as do not back up in iCloud. The lunacraft game files will not be backed up, even if you use iTunes (rather than iCloud) to back up your device. Until Apple makes full backups available as an option in iTunes, the backup process described here will not work for lunacraft.

The basic backup procedure is to connect the iPhone/iPad/etc. to the computer, wait for iTunes to launch and complete its "sync" process. Sometimes backup is not automatic (that's a settable option); in that case you select the device, right-click for the pop-up menu and select "Back Up". The data files from the device (but not apps, music, videos, and other content that originally was loaded onto it from your PC) get copied off the device into a directory somewhere on your computer.

Recovering data files from iOS device backups has been an important issue for many different reasons. There is usually not much of a privacy issue, because you own the device and presumably any personal data on it. (If an iOS device is stolen, it is usually possible to get Apple or your mobile telephone company to erase the data from the device remotely).

For that reason, programmers figured out where the backup files are located and how to get a listing of the files in the backup. I recently updated an article on this topic which you can read here: How to parse the manifest.mbdb file.

A New Moon

The first thing to try is creating a new moon, then immediately saving it and quitting lunacraft (without doing anything, and hopefully before mobs do anything to it). So I created a new moon. Here are some photos I took of it a little while after creating it:

Looking straight down at the square it spawned me on (north is at the top)

Notice the sort of elliptical gray blob in the center. This is a fairly flat "hill" that I spawned on when the world was created.

Looking towards the east

Here, notice a partly-submerged circular crater.

Since I had not yet put iOS 5.0.1 on my iPad, I got a full backup including all the Mooncraft files. Here are the files it created as soon as I created this new "Moon A":

-Size- -Checksum- -Pathname-------------------- 12 727432202 Documents/0_-128_-128.dat 16384 94748750 Documents/0_-128_-128.hm 183334 3972780535 Documents/0_-128_-128.mw 12 1346119545 Documents/0_-128_0.dat 16384 2818815432 Documents/0_-128_0.hm 193702 2563623835 Documents/0_-128_0.mw 12 1571299781 Documents/0_-128_128.dat 16384 3381136342 Documents/0_-128_128.hm 185686 370496922 Documents/0_-128_128.mw 12 2979332072 Documents/0_0_-128.dat 16384 726987670 Documents/0_0_-128.hm 211224 2039904936 Documents/0_0_-128.mw 84 1316255155 Documents/0_0_0.dat 16384 590261377 Documents/0_0_0.hm 199534 2494107293 Documents/0_0_0.mw 120 2133446136 Documents/0_0_128.dat 16384 1689383028 Documents/0_0_128.hm 192998 3976573523 Documents/0_0_128.mw 12 1941709056 Documents/0_128_-128.dat 16384 2087501117 Documents/0_128_-128.hm 208818 1082692611 Documents/0_128_-128.mw 12 148617843 Documents/0_128_0.dat 16384 3032337052 Documents/0_128_0.hm 195136 3117172632 Documents/0_128_0.mw 12 89111759 Documents/0_128_128.dat 16384 1318452703 Documents/0_128_128.hm 202934 2206439436 Documents/0_128_128.mw 6893 3775319515 Documents/0_game.dat

For each file I am showing a size in bytes, and a "checksum" which is used to figure out if a file's contents have changed (which will be useful later).

The first thing to notice is that these all start with "0_". If you create a Moon B, you get a bunch of files starting with "1_", and so on.

Then there are two more numbers, and in each case the number is -128, 0 or 128. I guessed right away that these are coordinates, and that the world is being stored in 128×128 block sections. For each section there is a ".dat" that is small, a ".hm" that is 16384 bytes, and a ".mw" that is about 200K.

Right away I noticed that 16384 is another power of 2, and in fact it is... 128×128!

So that made me think right away, the .hm file might be a 2-dimensional map of some kind.

I went into the terminal program, with a terminal window set to 128 characters wide, and simply dumped the backup file from Documents/0_0_0.hm onto the screen. Here are the last 50 or so lines of it:

Notice anything familiar? Near the bottom about halfway across is the same shape as the little hill in the first photo above. It is sideways, and flipped over (a mirror image) but is otherwise identical.

In the file dump, the hill is made of the characters @ and A. If you look up the ASCII code (try the Wikipedia page on ASCII, about halfway down) you'll see that @ is 64, and A is 65.

Since we already know that the "bottom of the world" is 63 blocks below ground level, it makes sense that our hill is made up of the numbers 64 and 65. The bytes in the 0-0_0_0.hm file simply tell how high the surface is, measured in blocks up from the bottom of the world!

A little bit to the east of my small hill is the circular "crater" (see the photos above). You'll probably be able to recognize the above-water part o this crater in the ASCII dump. In this case the terrain goes a couple blocks higher, and we have the letters A, B and C in the file dump. The underwater parts inside the crater are made up of characters like 8, 9, :, ;. In the ASCII code, these all correspond to numbers a bit less than 64.

A Large-Scale View

Now that it's clear that the .hm files like "0_0_0.hm" and "0_-128_128.hm" contain 128x128 parts of the map, it is then clear that the numbers are north-south and east-west coordinates. By inspecting maps from a moon that I was much more familiar with, it was easy to determine that the first number increases from east to west, and the second number increases from south to north. If the 9 map pieces in my example new moon are laid out in proper relative positions, the numbers go like this:

E A S T -128 -128 -128 N 128 0 -128 S O O R 0 0 0 U T 128 0 -128 T H H 128 128 128 128 0 -128 W E S T

So for example the northeast piece is in the file "0_-128_128.hm". I put east at the top because that's the way the data in the .hm files is arranged: the first 128 bytes in the file give the terrain height of the eastern-most row of the 128×128 square. The detailed data in the .mw files (see below) is in the same order. I decided to stick with this orientation to avoid the need to rearrange large amounts of data in my program.

Once we know that files are for 128×128 pieces of the map, it becomes easy to produce a very large-scale map of the whole world. This is simply a grid showing which 128×128 "superblocks" are actually present in the world data files:

Superblock map for moon A (East is UP): 1...-1 -1 [][][] 0 [][][] 1 [][][] World is 3 superblocks from east to west, and 3 from north to south. Total area: 9 superblocks (1.0 sq km), 147456 blocks

It becomes more interesting when the world is large. Back before I understood that the emulated world was infinite, I used one moon to attempt to discover the size of the world. I walked in the same direction continually, placing Light blocks as I went. I changed direction every now and then. I expected that I would eventually go all the way around the moon and cross my path (this never happened). Here is the superblock map of that world:

Using these examples and others I was able to establish the relationship between the number of map superblocks and the "explored area" number that the game shows you. It is a very close correspondence: 9.23 superblocks per sq km, or 0.108 sq km per 128×128 superblock. The text at the bottom of the image says:

World is 41 superblocks from east to west, and 72 from north to south. Total area: 277 superblocks (30.0 sq km), 4538368 blocks

A single block turns out to be 6.6 square meters in area, which makes the blocks 2.57 meters (8.4 feet) in each dimension, or 17 cubic meters in volume. If that's really true, then the Astronauts are giants: they are over 3 blocks tall!

The .dat Files

The smallest files created by lunacraft are the ".dat" files:

-Size- -Checksum- -Pathname-------------------- 12 2979332072 Documents/0_0_-128.dat 84 1316255155 Documents/0_0_0.dat 120 2133446136 Documents/0_0_128.dat [. . .] 6893 3775319515 Documents/0_game.dat

These files change continuously, regardless of any changes to the world.

Possibly the most interesting of these is the one called "0_game.dat". This is a binary plist file, viewable in the Apple Property List Editor (which comes with the Xcode Tools). This can be exported to a textfile, and contains a lot of empty nodes intermixed with lines like:

locationz, locationx, health, pitch, inventory, rotation, suitStatus, locationy, oxygenLevel, 155, 231, 100, -90, "inventoryEntry-2", "inventoryEntry-42", "inventoryEntry-28", "inventoryEntry-35", (... and so on) nextEventTime, skyColor, gameTime, 380.7181119918823, 177.2975, 62.08589021861553,

I spent quite a while looking at this. All I can report at present are:

The numbers are values and the rest are the names of values.

The values appear after their names and in the same order, but the two types are intermixed with each other.

There are not enough "inventoryEntry" values to account for the type of item and the quantity (there are 50 inventory slots in the game, and each needs both a type and a quantity, which requires 100 numbers but there are less than 100 numerical values in the entire 0_game.dat file).

The other .dat files record the locations of Mobs and Mini_Lights within each 128×128 section of the terrain. I was forced to figure this out when my game became unplayable due to the continuous gunshots of befriended Astronauts. I successfully recovered by getting the '.dat' files from moon A out of a previous backup of the iPad, and combining them with all of the other iPad backup files from a current backup, then using the resulting directory of files to restore the iPad. All the work I had done (including tunnels and other objects I created) was restored, but astronauts and mini-lights were back the way they were in the older saved game.

Detailed Maps

The files ending in .mw, like 0_0_-128.mw, are the largest and clearly these must be where all the 3-D map information goes.

I started by looking at the file byte-by-byte. Here is a little example starting at the beginning of the file "0_0_0.mw":

: od -A n -t u1 -v -N 128 8c676d7863301fcc8574e0b4599c8046af72fc6e 4 47 6 5 2 3 3 3 8 6 0 63 4 47 6 5 2 3 3 3 8 6 0 63 4 48 6 5 2 3 3 3 8 5 0 63 4 48 6 5 2 3 3 3 8 5 0 63 4 48 6 5 2 3 3 3 8 5 0 63 4 48 6 5 2 3 3 3 8 5 0 63 4 49 6 5 2 3 3 3 8 4 0 63 4 51 6 5 2 3 3 3 8 2 0 63 4 52 6 5 2 3 3 3 8 1 0 63 4 50 6 5 2 3 3 3 8 3 0 63 4 50 6 5 2 3 3 3

I looked at this data in ASCII, like before, and thought about it for a while. There are many ways (some less obvious than others) that a programmer might store 3-D Minecraft type data.

As described in the Renderblock article there is a bug that makes blocks transparent. I noticed that when the bug is fixed (by drilling out a block), the bug is fixed in one vertical column, but not in any as-yet-untouched vertical columns. We also know the "bottom" and "top" of the world are 63 blocks up and down from "ground level". So I had the idea that data is probably stored one column at a time, and that the columns were 128 blocks high.

I was planning to do lots of tests that involve placing blocks, backing up the iPhone, examining the 0_0_0.mw file, placing more blocks, examining 0_0_0.mw again, and so on. But I didn't want to do that and after a couple days of thought, I decided to try run-length encoding. Consider the vertical column of blocks (and space where blocks could be), 128 blocks high. There are usually lots of plain Rock at the bottom, and lots of air at the top. You could save a lot of space by just putting the number for "Rock" and "42" to represent a stack of 42 blocks of Rock, rather than storing the number for "Rock" 42 times.

Right away, the data sample above looks like a good fit: there are lots of "0 63", which looks like 63 repeats of 0 which is "nothing" in the Items data file. I looked at the first few bytes above, grouped in pairs:

4x47, 6x5, 2x3, 3x3, 8x6, 0x63, ...

The second numbers in the pairs add up to 127: 47+5+3+3+6+63=127. I also immediately recognized the layer thicknesses (5, 3 and 3) of the Gravel, Dirt and Sand found under ice (which I earlier discovered was universal, see the Geology article).

The first numbers in the pairs should be Rock, Gravel, Dirt, Sand, and Water. Looking at the "blocktype" column in items.txt you can see the numbers are off by 1. (I soon discovered that "nothing" is indeed 0, and the "top layer of dirt" gets the value 1, while normal dirt is 2.)

So based on just that, I went ahead and wrote a program that reads through the whole "0_128_-128.mw" file for a world I have been doing most of my gameplay. I already knew this was the newest .mw file, so it was probably the area of the map where I was working most recently. There should be familiar terrain in whatever image I get. I was not disappointed:

Output from my script, showing a portion of the (128,-128) block of my Moon A.

The same part of Moon A viewed in the game, with the aid of the InfiniteJumpingGlitch (some blocks were changed after the map was made)

Here the player is looking towards the northwest from a tower built near the large astronaut test chamber in the southeast

A number of my constructed features, as well as some of the natural features like hills and plants, can be recognized in my map as well as in the game screen shots.

After further investigation I figured out that every "superblock" in the world has a ".mw" file just like this, and the .mw files have all the blocks in the 128×128×128 space and nothing else.

Using this knowledge I have been able to discover lots of cool things, including the occasional appearance of an odd type of block, comprehensive statistics on the relative abundances of materials, hidden and abnormal Astronaut lairs], and lots of other things.

Mobs and Other Non-Grid Entities

Here again are the files for one 128×128 superblock:

-Size- -Checksum- -Pathname-------------------- 120 2133446136 Documents/0_0_128.dat 16384 1689383028 Documents/0_0_128.hm 192998 3976573523 Documents/0_0_128.mw

After figuring out that the ".hm" file is a map giving the elevation of each square in a 128×128 grid, and that the ".mw" file contains all of the blocks in the 128×128×128 3-D space (and nothing else), the only file left is the ".dat" file. So this file must be the place where the game stores information about Mobs and dropped Disks and things like that.

I happen to have one moon with a lot of Astronauts and two giraffes in one place:

The cocktail party (giraffes drink for free)

Based on the maps like the ones shown above, it's easy to find the room shown in the photo. It is 53 blocks below ground level (or about 10 blocks up from the bottom of the world) and it's in the superblock whose files start with "0_256_-128". The "256" is in the east-west direction and the "-128" is in the north-south direction. Within the 128×128 area, the room in the picture goes from 96 to 108 in the N-S direction and from 101 to 114 in the E-W direction. Adding these numbers together, the room's coordinates work out to -32 to -20 in the N-S direction and 357 to 370 in the E-W direction.

I've had some experience dealing with floating-point binary formats so it was fairly easy to start scanning through the "0_256_-128.dat" file looking for numbers that seem close to these coordinates. I found some pretty quickly:

12 44 34 25 25 360.290192 16 41 b0 00 00 11.000000 20 c2 74 66 66 -30.549999 24 43 27 58 6c 83.672699 ... 48 44 33 77 1c 358.930542 52 41 b0 00 00 11.000000 56 c2 2c 56 13 -21.542028 60 43 34 10 11 90.031380 ...

There are a lot more that all have a number "11.000000", right in between two numbers that are right near the other N-S and E-W coordinates of my astronaut party room. There is another number right after these, which I expected to find: an angle in degrees (which tells what direction the mob is facing).

Along with these 4 numbers (which use up 16 bytes) there is another 20 bytes per mob, which must include other things like the mob's aggressiveness and its health/damage level.

Practical Applications

Unfortunately, with iOS 5.0.1 it is no longer possible to back up your lunacraft game files. However, if this situation is ever improved, or for users who have iOS 5.0 or earlier, the information in this article has several practical applications:

Maps extracted from the saved game files can be used to find your present location, and find your way back to your home base if you are lost. The procedure would be roughly as follows:

Place a unique pattern using a unique block type (such as an X made out of Blue_Stone on top of Gravel)

Back up your iPhone/iPad/iPod Touch using iTunes

Run a program that scans the maps for the X pattern, and also scans for obvious human artifacts like blocks in places they shouldn't be, ice above ground level, etc.

Un-do the X pattern and walk in the estimated direction that you want to go. If necessary, make a new X mark and repeat.

Map data can be used to compile statistics on the relative abundances of the various Materials, and also to locate any unknown material types. I write a bit about this in another blog article: User blog:Mrob27/Block type 130.

Maps can be used to re-create a world in another program. For example, it is possible someone could take their lunacraft world data and use it to create world files in Minecraft (whose format is also known and documented, see Chunk_Format on the Minecraft forums).

(If you like this kind of thing, I also figured out the Eden World Builder file format.)

This page was written in the "embarrassingly readable" markup language RHTF, and was last updated on 2013 May 06.

s.27