Save file structure

[ciuly]

Hello,

I am trying to get a hold of the fileformats for d2s and d2i. I am mainly interested in d2i (I'm working on an automated muling application (basically an item dumper). I can dump all items with one problem: socketed items loose their fillings (and I get the fillings separately). So I am trying to find a way around this. Currently I'm working for v1.10 but all the info I have found on the net is ... apparently wrong.
for example the 1.10 format from SVR
he says bSocketed is bit number 28 (offset 27, calculated based on his table)
Trevin, in his 1.09 format also says it's at offset 27.
BUT, I did the following. found a normal shield in the game. saved it with atma as d2i. used the socket formula to add some sockets (all this to have as little changes in the item itself as possible). I compared the binary data in a hex editor and I found that the socket bit is at offset 28 (bit number 29). counting from left to right, the socketed bit is in the 4th byte (counting from 1), the 5th bit if counting bits from 1 and the 4th bit if counting them from 0 (again, left to right). so we have byte 4: 00001000 for a socketed item. but both formats mentioned expect it to be 00010000.
Same problem I have with ethereal bit. In my tests it's bit 34 (offset 33). in both formats above however, the placement is way off (not 1 bit as above, but more). I didn't test anything else since I'm not particulary interested in anything else, but I suspect most of the stuff from SVR is wrong OR for another version and not for 1.10 as advertised.
I made a small util to convert the binary d2i to xml format and that's how I noticed the issues. Other values are also wrong (my code is based on SVR format),
I tried contacting tenshi but his mailbox over hotmail is no longer available. I'll give SVR a PM with this post in case he's still around.

Anybody has soemthing I can use? I could waste a few days to figure things out but I'd rather not
thanks.

[SVR]

You probably have your bit reader backwards.

In a bit stream, the low bit can be first or last depending on how you look at it.

10000000b

This is 0x80.

If you assume left to right the first bit (the 1) will be bit 7 (the High bit).
But in practice the first bit should be bit 0.

This becomes really important with values that span byte boundries.

If you edit the item in UDIETOO you'll see the bytes are backwards (low bit left) so all bits form a continuous stream from 0-xxx, left to right.

This way, the bits of any value will be contiguous and can be interpreted right to left for that particular value.

[ciuly]

so what you say is that 2 bytes: 0x80 0x08 should be seen as a bit stream looking like: 00000001 00010000 ?
that seem to {filtered} for my miss-numbering because I'm using 10000000 00001000, meaning just the way you would normally write it on paper. this would account for both socketed and ethereal bits if I see it correctly (haven't changed the code yet).
is this something standard or just one way of seeing a bitstream? I'm asking so that if it's standard, I'll make my bit-manipulation code a library instead of keeping it project specific

thanks

[SVR]

Yes 0x80 0x08 would display as 00000001 00010000

It is standard in a bitstream.

If you read through the previous page, I go into more detail about values spanning bytes and show how they get split if you don't use low bit first convention.

If a value started at bit 4 and was 8 bits long in the above stream it's value would be 0x88.

The order keeps the two halves together in the display. The other way, one 8 would be all the way left and the other on the right (just like the hex).

[ciuly]

thanks. I hate being a pain, but maybe I'm way off again. doesn't nGems represent the number of things (gems, runes, jewels, etc) from a socketed item?
BYTE nGems 3 Number of Gems 1 unk51 1
if not, which one is it? because after name/description/comments I can't figure it out :|

[SVR]

Yes nGems is number of things in sockets.

The cond columns basically says these bits are present if unk51 <> 1.
Most of the fields with cond1 are '0' (equal).

Based on these conditions, a field is present or not. This may mean the remaining structure is entirely diferent (as in the case of ears).

[ciuly]

"these bits are present if unk51 <> 1."
drats. I was testing for =1 :wallbash:
during heavy debug and checking each field twice I found a small error in your VSD
BYTE unk60 10
it's either
BYTE unk60 8 (or less)
or
WORD unk60 10
no? BYTE is 8 bits according to VSD
initially, I used 8 instead of 10, cumulated with my unk51 bad test, that lead to those bad results.

also, regarding unk51. the VSD says its 1 cond1 1, which according to VSD, the field is included if the value of unk51 compared to 1 is 1 (=greater than).
so basically 0 (the usual value of unk51) compared to 1 results in 1. which is kind of wrong since 0 will never be greater than 1 after all VSD says compare 'condvar1' with 'condval1', not the other way around (this could be something language specific, but in all languages I worked with the result of comparision shows how the first term relates to the second).
shouidn't the VSD be
BYTE nGems 3 Number of Gems 0 unk51 0
?
moreover, if puting logic in the equation, shouldn't the test field actually be bSocketed? because the way I look at it if an item is not socketed, it won't need those bits included since obviously nGems will be 0. if it's socketed, it must contain those bits to signal how many fillings it has.
but,. of course, I haven't seen most of the items data so my logic is probably missing something

and another "small" problem regarding bit streams. you said that 0x80 (10000000) is actually read up 00000001. so the bit 1 is at offset 7.
I changed my code to read up this way, so I read up 2 characters, 16 bits from the bit stream, (the JM) which obviously will not end up JM but R something.
so obviously this bitstream reading mode does not apply to the ID.
my question is: what other ASC8/ASC7 fields are read the same way as the ID field? all?

I'm only asking out of curiosity (for now) since I managed to implement the socketed support, apparently ok. for which thanks are in order so thank you.

I now have some missing items though :| This one will be hard to track down. save file reports there are 114 items but I only managed to extract 96. So there are 18 items missing in action. (yes, I'm using a mod to increase storage space which is why I cannot use ATMA anymore and I'm doing this little util)
I manually checked the extracted items and it appears that I am missing the left+right alternate hands weapons and shields. both are socketed. on has 6 socket, all filled and the other 3 sockets, all filled. this would howveer account for only 2 out of missing 18. my merc has no items. I am also missing a ring from one of my fingers. so 3 out of 18. the rest will take some time to compare but I will do it
btw, I counted my items in game (on body, in belt, in inventory, in cube and in stash), twice. they're 113. is gold counted as 1 item? I can't think of anything else.

[Necrolis]

You might be missing your hirelings items(iirc bodyloc items are at the start of the inv section)[edit: my reading skill is shot :/ gold is stored as a stat, checked your cube?]. Anything that is compacted is read as a bitsteam, which is anything that uses stats, so the stat section and item section aswell as the quest section and waypoint section. You should never read in to an id, instead you should do pointer difference on pStartOfSection + 2 - pStartOfNextSection, so you get section length, * 8 gives total bits in stream(ofc ).

[ciuly]

my hireling (merc0 has no items. I checked in game. Actually, I had to revive him to check
the gold question came because I counted manually the items, and they add up to 113, but in the save file it reports 114 items. I counted twice and I got 113 so I might be missing something, somewhere.
in any case, I started to manually cross-check game-atma-extracted items to see what is missing from where. I can say for sure that 1 small rejuv is missing from either the belt, or the cube. not very helpfull though.
this is probably my code to blame since I am not using any "nice" way of extractng the items. I basically search to the first "JM", read up the number of items (1 word), then I start extracting everything, raw, from next JM (included) until next-next JM. I do that until I run out of JMs or I reach end of item list or there is no JM after current JM but still not end of item list. Shouldn't happen but I'm paranoic so I am checking anyway
it's not nice, but it will do the job for what I have. provided that I fix it to extract ALL items

actually, I think I'll comment the socket support so that I extract absolutelly all items separately and then see if that gets me everything. and if it does, it's my socket code that skips items, otherwise ... I'll be lost :|

brb

edit: just hit me: could it be that the save file total item count is counting my mercs default weapon?

[SVR]

"these bits are present if unk51 <> 1."
drats. I was testing for =1 :wallbash:
during heavy debug and checking each field twice I found a small error in your VSD
BYTE unk60 10
it's either
BYTE unk60 8 (or less)
or
WORD unk60 10
no? BYTE is 8 bits according to VSD
initially, I used 8 instead of 10, cumulated with my unk51 bad test, that lead to those bad results.

Correct, this table was written as things were discovered and not all changes got propagated to related stuff.

also, regarding unk51. the VSD says its 1 cond1 1, which according to VSD, the field is included if the value of unk51 compared to 1 is 1 (=greater than).
so basically 0 (the usual value of unk51) compared to 1 results in 1. which is kind of wrong since 0 will never be greater than 1 after all VSD says compare 'condvar1' with 'condval1', not the other way around (this could be something language specific, but in all languages I worked with the result of comparision shows how the first term relates to the second).
shouidn't the VSD be
BYTE nGems 3 Number of Gems 0 unk51 0
?
moreover, if puting logic in the equation, shouldn't the test field actually be bSocketed? because the way I look at it if an item is not socketed, it won't need those bits included since obviously nGems will be 0. if it's socketed, it must contain those bits to signal how many fillings it has.
but,. of course, I haven't seen most of the items data so my logic is probably missing something

Not sure. I think originally unk51 was more than 1 bit.
But these bits do not rely on bSocketed. They are still there when bSocketed is 0.

and another "small" problem regarding bit streams. you said that 0x80 (10000000) is actually read up 00000001. so the bit 1 is at offset 7.
I changed my code to read up this way, so I read up 2 characters, 16 bits from the bit stream, (the JM) which obviously will not end up JM but R something.
so obviously this bitstream reading mode does not apply to the ID.
my question is: what other ASC8/ASC7 fields are read the same way as the ID field? all?

I'm only asking out of curiosity (for now) since I managed to implement the socketed support, apparently ok. for which thanks are in order so thank you.

All the bitstream data will read correctly if you put it in the variable correctly. The bits are streamed left to right, low bit first but the value is interpreted as usual.

Code: Select all

// BitStream pseudo-example
// bitstream contains 0xA3 0xEA in hex ( 10100011 11101010 )
// bits - 11000101 01010111 (in bitstream order - low bit first)
//        01234567 89ABCDEF

BYTE bValue1; // in first 4 bits
BYTE bValue2; // in next 6 bits
BYTE bValue3; // in next 3 bits

// last 3 bits are padding and ignored/not read


          Hi             Lo 
bValue1 = [x x x x 3 2 1 0]
                         |
                          \-- ReadBit();  // bit 0 = 1
                       |
                        \-- ReadBit();    // bit 1 = 1
                     |
                      \-- ReadBit();      // bit 2 = 0
                   |
                    \-- ReadBit();        // bit 3 = 0

bValue2 = [x x 5 4 3 2 1 0]               
                         |
                          \-- ReadBit();  // bit 4 = 0
                       |
                        \-- ReadBit();    // bit 5 = 1
                     |
                      \-- ReadBit();      // bit 6 = 0
                   |
                    \-- ReadBit();        // bit 7 = 1
                 |
                  \-- ReadBit();          // bit 8 = 0
               |
                \-- ReadBit();            // bit 9 = 1

bValue3 = [x x x x x 2 1 0]
                         |
                          \-- ReadBit();  // bit A = 0
                       |
                        \-- ReadBit();    // bit B = 1
                     |
                      \-- ReadBit();      // bit C = 0


bValue1 is 0x03
bValue2 is 0x2A
bValue3 is 0x02


For any given position the byte is position/8 and bit is position mod 8

ASC7 & ASC8 are interpreted/read as individual bytes as you would read any 8 or 7 bit value.

WORD & DWORD values continue in the same pattern, shift each bit read up to the next slot.

I now have some missing items though :| This one will be hard to track down. save file reports there are 114 items but I only managed to extract 96. So there are 18 items missing in action. (yes, I'm using a mod to increase storage space which is why I cannot use ATMA anymore and I'm doing this little util)
I manually checked the extracted items and it appears that I am missing the left+right alternate hands weapons and shields. both are socketed. on has 6 socket, all filled and the other 3 sockets, all filled. this would howveer account for only 2 out of missing 18. my merc has no items. I am also missing a ring from one of my fingers. so 3 out of 18. the rest will take some time to compare but I will do it
btw, I counted my items in game (on body, in belt, in inventory, in cube and in stash), twice. they're 113. is gold counted as 1 item? I can't think of anything else.

I don't think (don't remember anyway) alt weaps are stored any different.
And gold is a player stat, not an item.

It's probably a missed case in an item throwing you off. Be sure you are accounting for properties correctly.

Check your items in UdieToo. The Bin editor will display each field so you can see what is actually in the item record.

You will need the modded txt files for it to work correctly.

EDIT:

this is probably my code to blame since I am not using any "nice" way of extractng the items. I basically search to the first "JM", read up the number of items (1 word), then I start extracting everything, raw, from next JM (included) until next-next JM. I do that until I run out of JMs or I reach end of item list or there is no JM after current JM but still not end of item list

OUCH. There is no really good way to do this because you can never reliably 'see' the next JM. It may be anywhere in the stream. And there may be 'false' JM's in the item data as well.

[ciuly]

found the problem. I was testing for end of list with JM 0 0 but I had an item with 0 0 after JM I changed that check to account for the actual number of items from save fiel and now everything is extracted out fine.

thank you all and thank you SVR for your explanations on the bit stuff. got my light up

[ciuly]

OUCH. There is no really good way to do this because you can never reliably 'see' the next JM. It may be anywhere in the stream. And there may be 'false' JM's in the item data as well.

you are right so I started a VSD-like parsing. all goes well until sType is met in the condition section. does that refer to dwType? or something else?

edit: also, in extend2.txt iDef is conditioned by iType. which field is iType referring to?

much later edit: I've read through the topic again. I found the explanations for sType/iType/etc

however, this is getting a little too big and complex for what I wanted requiring too much time to work on. having the item name would have been nice, but I think I'll go (for the filename) with something like "item GUID" and if that exists and item appears to be different, "item GUID counter".
also, I am having the intial JM 2 bytes JM check which should be sufficient to guarantee the correct offset for the item list and also a late check that the number of extracted items matches the number of items initially found. if all the checks are ok, I think it's safe to assume that I have correctly found all items.
in worse case scenario, I will re-extract some items a few times, leading to "item GUID counter" files but that's acceptable considering the alternative of not extracting the item and if actually being differrent, loosing it.

just a quick question: haven't anybody considered of creating a DLL for save file manipulation? I think it would benefit more people if we could concentrate on new functionality instead of practically reinventing the wheel just a thought.

[SVR]

Yes it can get big in a hurry. For any 'real' item extractor to work you have to use the game files (itemstatcost etc,etc) to get the correct number of bits for properties and such.
Even then there is not enough info to fully crack the properties. That's why we all use some sort of txt that discribes what we 'know about' in a vanilla setting.
Turning all that into a dll someone can easily use probably won't happen.

[Necrolis]

Yes it can get big in a hurry. For any 'real' item extractor to work you have to use the game files (itemstatcost etc,etc) to get the correct number of bits for properties and such.
Even then there is not enough info to fully crack the properties. That's why we all use some sort of txt that discribes what we 'know about' in a vanilla setting.
Turning all that into a dll someone can easily use probably won't happen.

I would say there is enough info, but the problem is that the bit values are needed, so for mods this could cause problems(apart from the modders themselves complaining...)

[ciuly]

Turning all that into a dll someone can easily use probably won't happen.

I'm just curious, don't take it any other way is that because of protecting ones work, or because it's too complicated? or something else?
I'm asking because I'm mainly a contributor to the community when it comes to my free time, so I often release a lot of code and when I don't want the code out because I worked too hard for it, I release an util or something
so I am pretty open minded of getting something like this out as a dll, so if there are not complication issues in question I'm not seeing, it'll probably make me want to do it, just so that it's done

what's your opinion on the subject?

[SVR]

I would say there is enough info, but the problem is that the bit values are needed, so for mods this could cause problems(apart from the modders themselves complaining...)

Not enough from just the txt's. The hard coded prop funcs make full decoding (without artificially adding these differences) imposible.

I ended up hard coding each propfunc ID in a case in Udie2. If a modder changes these (through CE) it will break.
As far as mods in general, Udie2 has to have the txt's included to work at all on mods, so all modders need to do is not include them

EDIT:

I'm just curious, don't take it any other way is that because of protecting ones work, or because it's too complicated? or something else?
I'm asking because I'm mainly a contributor to the community when it comes to my free time, so I often release a lot of code and when I don't want the code out because I worked too hard for it, I release an util or something
so I am pretty open minded of getting something like this out as a dll, so if there are not complication issues in question I'm not seeing, it'll probably make me want to do it, just so that it's done

what's your opinion on the subject?

Making the dll easy to use and still work correctly is the problem.
Even so, you should keep in mind what others may do with your work.
You should absolutey require txt's if it is intended to work with mods.

[ciuly]

Making the dll easy to use and still work correctly is the problem.
Even so, you should keep in mind what others may do with your work.
You should absolutey require txt's if it is intended to work with mods.

well, somebody has to start somewhere, right?
I'll see what I can come up with in the following days/weeks

[Necrolis]

All the mods that i have seen that have prop func changes that affect encoding have changed the save format and iirc cube ops doesn't change encoding[the only excepting is if some idiot changes the stuff column, but this will break encoded stats]. I think the real need for something like this isn't really enough to warrent it anymore(though in ahead by all means)

[linuxcat]

An update on my php character decoder.

It now decodes runewords, ears, and personalized items. The stats on items in sockets are displayed. The item decoder is much improved - specifically the magic properties.

It is written in PHP so you can view all of the source.

I tried to provide a link to an example character but the spam blocker prevents me from posting links. The download link is the same as it was in my previous post in this thread.

If you open the link given earlier and remove /charinfo.zip from the URI you will find a link to the character decoder page and an example decoded character.

[ChaosMarc]

hi linuxcat

i've tested your chac decoder and i'm sold on it

i found one little flaw:
http://chaosmarc.bplaced.net/chaos-marc/corpse.png

don't know what this should be
my char doesnt have a corpse actually and i've never killed a lvl14 necro

[linuxcat]

I don't know. I do have 1 guess. Is your hireling dead? I have a level 46 Necromancer ear on my level 82 Barbarian as my last item (before the hireling). I did not resurrect my hireling at some point.

[ChaosMarc]

hmm my hireling is alive. but it's possible that i didn't resurect my old hireling before i bought a new one

[Gamall]

Hello,

SVR's homepage http://home.roadrunner.com/~svr/ seems to have gone out-of-date today: Not Found The requested URL /~svr/ was not found on this server.. It's one of my bookmarks and it was still working yesterday.

Is there a mirror somewhere? There were valuable resources on that page (Udie, comprehensive info on d2s file format...)

edit: it's back. *relief* The problem with old games is that you get a little jumpy about the availability of good resources...



A strange thing I have just run into is that item types are noted here http://home.roadrunner.com/~svr/formats/extend.htm as ASC8, that's to say null(\0)-terminated strings, while in fact it seems to be space (0x20)-terminated. I don't recall reading this anywhere.

[Nefarius]

Thats because they are not strings but tokens, the 0x20 isn't the end, but just another value.

Code: Select all

#define REVERSE16( w ) ( ( (w & 0xFF00) >> 8 ) | ( (w & 0x00FF) << 8 ) )
#define REVERSE32( dw ) ( ( (dw & 0xFF000000) >> 24 ) | ( (dw & 0x00FF0000) >> 8 ) | ( (dw & 0x0000FF00) << 8 ) | ( (dw & 0x000000FF) << 24 ) )
#define CODE32( sz ) REVERSE32( ( ( const unsigned long )( sz ) ) )
#define CODE16( sz ) REVERSE16( ( ( const unsigned short)( sz ) ) )

Code: Select all

pItemsTxt->Code != CODE32('leg ')

Code: Select all

20011DF5  |.  81B9 80000000>CMP DWORD PTR DS:[ECX+80],2067656C
20011DFF  |.  74 03         JE SHORT METALSTO.20011E04

etc

[Gamall]

Maybe it's the lack of sleep lately (and probably also my lack of knowledge of game mechanics; so far I have only looked at the d2s fileformat on its own), but I really don't get the connection between the code in your answer and my observation...

By your definition of CODE32, which just reverses the order of the 4 bytes, we have the mirror image

Code: Select all

CODE32('leg ') == ' gel'

Great, but what's the point?

I assume pItemsTxt makes reference to a specific structure defined somewhere?

The way I see it, I get the 3-char string (eg. "leg") [or the 32bit (All item codes in {armor,weapons,misc}.txt are exactly 3-char long.) value (eg. 'leg '), whose fourth byte must always be 0x20 and is discarded] out of my item's bitstream , and look it up in weapons.txt

Code: Select all

(weapons.txt:90) Wirt's Leg	club	leg   ...

and thus I get everything I need to know about the base item type. So seeing the field as a ' '-terminated string seems to always work, at least for my purposes.


And sadly, I don't grok assembly at all.