[Tool] D3TexConv

[Necrolis]

Made a small tool for ripping out the dds textures from the d3 tex files, you can dl it here (version 0.9b) with source (no fancy readme, and I can't link to the wiki entry as its a gray area on the ToS).

So for anyone wanting the items from D3, now is you chance

[mirecek]

Well.. thanks

I have extracted the texture mpq to lots of .tex files, but (using the 0.8 ) unable to get anythnig out of it.

does it need any parameter, or am I doing something wrong by moving the file over the converter (exe)..

http://img585.imageshack.us/img585/5508/errj.png


where does it dumps the files ?

[Necrolis]

Yes, its drag and drop, and you can do more than one file at a time.

you get a DDS file and a tab delimited txt file in "C:\Downloads\Textures\" named the same as the input text file.

you can rad DDS files using the DDS photoshop plugin from nvidia or infranview

[mirecek]

that is my problem. the number of files inside the folder where the exe is ... remains the same. no new files created

[Necrolis]

that is my problem. the number of files inside the folder where the exe is ... remains the same. no new files created

its not created in the fodler where the exe is, its created in the folder where the inputted text files are

[mirecek]

I put that exe INTO the textures folder.

*news: it created the dds files to documents&settings/user

[Necrolis]

I put that exe INTO the textures folder.

*news: it created the dds files to documents&settings/user

odd, it might be the settings under XP then.

just as a side note, there is a jpg dump here (not by me) to save people some time

[Sixen]

Necrolis, nice work! Now get crackin' on the models, . I've had the tex's for a bit, just no models to put them on, .

[Necrolis]

Necrolis, nice work! Now get crackin' on the models, . I've had the tex's for a bit, just no models to put them on, .

models where cracked a few days ago (same time I cracked the tex encoding), see this and this. there is actually a modding wiki for D3, if you know how to look you'll find it

[Sixen]

Necrolis, nice work! Now get crackin' on the models, . I've had the tex's for a bit, just no models to put them on, .

models where cracked a few days ago (same time I cracked the tex encoding), see this and this. there is actually a modding wiki for D3, if you know how to look you'll find it

Yeah, after replying here I was checking out the wiki and stumbled upon both of those topics. I didn't realize how far Falo and BoyC got since the last time I checked that thread, .

If I understand correctly, that script should parse the .app files and dump it into some kind of readable format, how do we then get it into a format that a modeling program can see? Or am I incorrect and it just dumps to a model-type format already? Sorry for the potentially dumb questions, .

[Necrolis]

From what i see, it dumps out obj files, which any common modeling program can edit

[Lurix]

Thanks for the link, it's a nice pack.
What's interesting for me, that the imagination of Staff and Spear seem to be changed a lot since age of Diablo II, I noticed that even when I was making pack of Torchlight item pack.
Same in Diablo III some of the items don't actually look like the way we're used to see them in previous versions. But it's still Blizzards decision so..

[Sixen]

From what i see, it dumps out obj files, which any common modeling program can edit

That seems to be the case! I'll check it out more when I get some time, .

[Muaziz]

Just came across this tool and really appreciate you releasing the source. Although I am not terribly interested in how images are encoded, I was interested in whether the \Texture\*.tex files in the Diablo 3 MPQs contained unique ids used by the game data to reference these images. Obviously they must at some point, but the binding could be done at runtime making it difficult to link items with their images (contained in these *.tex files).

So I was curious if anyone knew how the game (in either D2 or D3) binds the items to the appropriate images.

Necrolis, I also had a few comments about the source code. It's been working fine, but I was confused about a few things.

1) You have a comment that the "struct TexHeader" is 0x378 bytes. I think that it is actually 0x368 bytes since this structure does not include the MPQ 0x10 header. This doesn't affect any of the code as this is just a comment.
2) This same structure contains the following:
TexMipLevel pMipLevels[31]; //+28
uint32_t dwLoadType; //+124
The "TexMipLevel" structure itself is 8 bytes. So the size of this should be 31 x 8 = 248 = 0xF8. When you add this to the starting offset (0x28), the next entry in the structure really should be at offset: 0x28 + 0xF8 = 0x120. Also I wasn't sure if it really was a 31 (rather than 32) element array.
3) Finally, the last element of "struct TexHeader" is:
uint32_t dwUnk5[10];
I believe that there are only 5 or 6 DWords left (depending on whether the above array is 31 or 32). In any case, this doesn't affect the rest of the code since this "unknown" structure isn't being used anywhere.

I know these are all minor things, but thought I would point them out just in case.

As a final note, I've been actively working on reverse-engineering all of the D3 game file formats (from the beta client). While I have made great progress, I am looking for folks that want to collaborate on such efforts. This site seemed as good a place as any to look for folks with the skills to work on this. Please PM me here if you are interested.

[Necrolis]

1) You have a comment that the "struct TexHeader" is 0x378 bytes. I think that it is actually 0x368 bytes since this structure does not include the MPQ 0x10 header. This doesn't affect any of the code as this is just a comment.
2) This same structure contains the following:
TexMipLevel pMipLevels[31]; //+28
uint32_t dwLoadType; //+124
The "TexMipLevel" structure itself is 8 bytes. So the size of this should be 31 x 8 = 248 = 0xF8. When you add this to the starting offset (0x28), the next entry in the structure really should be at offset: 0x28 + 0xF8 = 0x120. Also I wasn't sure if it really was a 31 (rather than 32) element array.
3) Finally, the last element of "struct TexHeader" is:
uint32_t dwUnk5[10];
I believe that there are only 5 or 6 DWords left (depending on whether the above array is 31 or 32). In any case, this doesn't affect the rest of the code since this "unknown" structure isn't being used anywhere.

that source is about 2 versions behind, so somethings have been removed and fixed. a lot of the padding was put in post mapping of the fields needed to decode, so some of it may be off, but I remember checking it, and it was correct (I'll double check tonight, just haven't had time to finish what I need due to uni). also the MipLevels has only 31 entries, it can't have more entries. as for the final size, that was something from the MPQ decompressor (it ignores the SNO header).

[chippydip]

So I was curious if anyone knew how the game (in either D2 or D3) binds the items to the appropriate images.

I've been wondering about this as well. I did a few full scans of the data files and didn't see any references to the file IDs of the icon texture files. The only column in the items table that seems to point to appearance-related data is a reference to an Actor file for each item (which seems to have references to additional Actor files for each class/sex combination) but I have yet to find any info that would point toward the textures that are used for the inventory graphics.

I finally gave up and decided to just parse the filenames of all the icons to make a best guess at which item(s) they are for, but I'd really love to figure out how the game makes that connection.

As a final note, I've been actively working on reverse-engineering all of the D3 game file formats (from the beta client). While I have made great progress, I am looking for folks that want to collaborate on such efforts. This site seemed as good a place as any to look for folks with the skills to work on this. Please PM me here if you are interested.

I've been working on this as well (I'm building an item database site) and would be happy to share with other people that have been digging into the files.

[Muaziz]

I've been wondering about this as well. I did a few full scans of the data files and didn't see any references to the file IDs of the icon texture files. The only column in the items table that seems to point to appearance-related data is a reference to an Actor file for each item (which seems to have references to additional Actor files for each class/sex combination) but I have yet to find any info that would point toward the textures that are used for the inventory graphics.

I came across the same thing. I wrote a binary grep program that allows me to search all of the MPQ extracted files to try and correlate various files together. Sadly, it appears that some of these bindings occur at runtime or use offsets (rather than hard ids). Hopefully, I'll get access to the beta client and some point and will be able to track down some of this information.

I finally gave up and decided to just parse the filenames of all the icons to make a best guess at which item(s) they are for, but I'd really love to figure out how the game makes that connection.

Yeah, the manual process works, but its tedious and every time there's a patch, you may have to redo a ton of work. Not the best approach, but might be the only way until we can learn more about the internal formats.

I've been working on this as well (I'm building an item database site) and would be happy to share with other people that have been digging into the files.

I've been looking for a forum where people are getting into the nitty gritty of the MPQ files and the code. Being new to the D3 scene (and not having done any hacking work in D2 since 2001), I wasn't sure which sites were the most active on this front.

For now, I have posted a couple of things over OwnedCore. Here are the two links to the two posts I have made thus far:
* MPQ game files and *.gam file info
http://www.ownedcore.com/forums/diablo- ... ost2132350
* A more detailed (and prettier) version of this information is also available here:
http://www.d3inferno.com/MPQ.html
* MPQ \GameBalance\AffixList.gam
http://www.ownedcore.com/forums/diablo- ... t-gam.html
* A more detailed (and prettier) version of this information is also available here:
http://www.d3inferno.com/AffixList.gam.html

Mods: If you'd rather I not post external links, please delete the links and I can re-post this information in these forums instead.

[chippydip]

* MPQ game files and *.gam file info
http://www.d3inferno.com/MPQ.html

0x00C - I think this can be more correctly described as a record type code. In one of the patches the size of the item structure changed and this field was updated, but I think the location of the offset/length fields remained the same (not 100% sure).
0x010 - Seems to be another char[256] string (but the value is always "0000.gbi" so it doesn't seem to have any real significance)
0x214 - Always zero
0x218 - Seems to be an array of 49? 8-byte structs: 0x0: offset, 0x4: length, 0x8: 0, 0xC: 0 (the file type determined which item in the array is actually used)

* MPQ \GameBalance\AffixList.gam
http://www.d3inferno.com/AffixList.gam.html

I think they changed the format slightly since 7318. This is what I have:

0x000 - vtbl_ptr? (my guess is that these structures are copied directly into memory and then this offset is set to the proper class's vtbl pointer)
0x108 - alvl
0x110/0x114 - I suspect one or both of these effect how the buy price of the item changes with this affix
0x118 - required lvl
0x124 - rare name prefixes (file id)
0x128 - rare name suffixes (file id)
0x12C - affix group name (hash) (nothing with the same value here can spawn on the same item)
0x130 - affix group name 2 (hash) (this is more for reference I think)
0x134 - resource type (hash) (Mana, Fury, Discipline, Arcanum, etc)
0x138 - ? (hash) (only set for Legendary mana affixes)
0x168 - flags?
0x16C - more flags? (these 2 fields might be an 8-byte bit-field)

The id values used all over the *.gam files are actually hashes of string values (typically the name column of some table). They are computed like this:

Code: Select all

        public static uint D3Hash(this string name)
        {
            var hash = 0u;
            foreach (var c in name.ToLowerInvariant())
                hash = hash * 0x21 + c;
            return hash;
        }

The data that those "ModCodes" point to seems to be simple virtual machine instructions for generating the value for each mod. I suspect that its defined as an array of something like this:

Code: Select all

union vmInstruction {
    byte opCode;
    float value;
}

Opcodes are only 1 byte, but the union is actually 4 bytes. It seems that when they allocate space for the instruction array they don't actually zero it out before using it, so when an opcode appears, the next 3 bytes can be any random junk value. This doesn't cause any problems, bit it makes the patch files slightly bigger than they need to be.

Here are the opcodes I've found (assuming a simple stack-based VM):

0x00 - return (returns the value at the top of the stack)
0x01 - call (the next opcode determines which function to call)
0x03 - call_range(A, B) (pops 2 numbers and pushes a random value between A and A+B)
0x04 - call_rand(A, B) (pops 2 numbers and pushes a random value between A and B)
0x06 - push (the next instruction is the float value to push)
0x0B - add (pop 2 values and push the sum)
0x0C - sub (pop 2 values and push the difference)
0x0D - mul (pop 2 values and push the product)
0x0E - div (pop 2 values and push the quotient)

(There could be others, but these successfully parse everything in Affixes and Items)

[Muaziz]

0x010 - Seems to be another char[256] string (but the value is always "0000.gbi" so it doesn't seem to have any real significance)

Yeah, that makes sense and is consistent with what I have seen.

0x00C - I think this can be more correctly described as a record type code. In one of the patches the size of the item structure changed and this field was updated, but I think the location of the offset/length fields remained the same (not 100% sure).
0x218 - Seems to be an array of 49? 8-byte structs: 0x0: offset, 0x4: length, 0x8: 0, 0xC: 0 (the file type determined which item in the array is actually used)

It would be an array 0x50 (100 dec) which makes sense. And your thought about the file type being the index into this array is exactly what I thought at first. And it works for some files, but not others. Here is a list of file names, 0x0C value, and (absolute) file offset of the data start offset:

Code: Select all

ItemTypes.gam	1	228
Items_Armor.gam	2	238
Items_Legendary.gam	2	238
Items_Legendary_Other.gam	2	238
Items_Legendary_Weapons.gam	2	238
Items_Other.gam	2	238
Items_Quests_Beta.gam	2	238
Items_Weapons.gam	2	238
ExperienceTable.gam	3	248
Hirelings.gam	4	348
MonsterLevels.gam	5	268
Characters.gam	7	288
AffixList.gam	8	278
MovementStyles.gam	A	298
LabelsGlobal.gam	B	2A8
LootDistribution.gam	C	2B8
RareNames.gam	10	2C8
MonsterAffixes.gam	12	2D8
MonsterNames.gam	13	2E8
SocketedEffects.gam	15	2F8
ItemEnhancements.gam	17	308
ItemDropTable.gam	19	318
ItemLevelModifiers.gam	1A	328
QualityClasses.gam	1B	338
RecipesBlacksmith.gam	20	398
RecipesJeweler.gam	20	398
RecipesMystic.gam	20	398
SetItemBonuses.gam	21	358
EliteModifiers.gam	22	368
ItemTiers.gam	23	378
PowerFormulaTables.gam	24	388

As you can see, the 0x0C values of "4" is a real outlier. Also, the values of "7" and "8" are reversed. Finally, the value "20" maps to 0x398 even though there are still four more values afterward.

In other words, it doesn't quite work although it seemed intuitive enough. That's why I thought that the 0x0C value might be an index into some other structure which in turn would yield the location where the data offset is located. Seems more convoluted, but not sure what else works.

I think they changed the format slightly since 7318. This is what I have:

I thought I had updated everything for version 7318, but will double check.

I am still working through all your comments, and will make the necessary adjustments in my post.

Thanks again for the help and I will let you know once everything is updated.

I will play around with the opCode suggestion you mentioned to see if that yields anything interesting. Looking at "Prec 1", here is the data:

Code: Select all

"Prec 01" 06 1A 77 FF 00 00 80 3F 06 2B BB FF 00 00 00 41 01 37 E7 FF 4 3A F3 FF 00 3F FA FF

So here's what I get:
1) 06 is a Push (the other 3 bytes are ignored)
2) 0x3F800000 is the value being pushed. (A = 1.00)
3) 06 is a Push (the other 3 bytes are ignored)
4) 0x41000000 is the value being pushed. (B= 8.00)
5) 01 is a Call (the other 3 bytes are ignored)
6) 04 is a call_rand(A, B) (the other 3 bytes are ignored) --> a random number between 1.00 and 8.00
7) 00 is a Return (the other 3 bytes are ignored)

Using the IEEE 754 notation for floats seems to make sense with the limited data I looked at.

It seems like the 3 bytes after the opCode are always ignore btw. I don't think they are unioned with a float. When a Push occurs, it is the next DWord that is pushed.

(There could be others, but these successfully parse everything in Affixes and Items)

What do you mean by "successfully parse"? Did you already calculate the values for all this variable data? I was going to write a script to do this tonight based on your extremely helpful input.

I will be sure to acknowledge your contribution once I update my posts. Let me know if you want me to link to any source material that you may have already posted.

PS: How did you figure this out? While it does make sense from a pure programming perspective, it seems far from obvious.

[chippydip]

And your thought about the file type being the index into this array is exactly what I thought at first. And it works for some files, but not others.

I guess I wasn't clear. I think this field is some sort of sequence number that's generated during the build process for each class (record type) this is being serialized into *.gam files. When the layout of these classes is changed, a new number in the sequence is used to represent the new format. My guess would be that there's an internal table mapping these type codes to information about the class that it should be loaded as (part of this information would be the offset in the 0x218 table, as well as the record size and the value to inject for the vtbl pointer field at the start of each record.)

In any event, I didn't spend too much time on it once I figured out it was an array. My parsing code just scans the array for a non-zero entry and uses that.

I thought I had updated everything for version 7318, but will double check.

Version 7338 was the most recent version until today (7447) but I guess none of the *.gam files have changes since 7318. I think some of the columns shifted offsets between the base version and 7318 and it looked like some of your offsets hadn't made the shift.

Thanks again for the help and I will let you know once everything is updated.

No problem!

What do you mean by "successfully parse"? Did you already calculate the values for all this variable data?

Yeah. My parsing program right now reads in all the *.gam files (and *.stl and a few other things) and dumps it into a database. To start it just guesses the record size (using either the distance between those string fields or by using minimal-average-standard-deviation metric for files without string keys). It then assumes that columns are all 4-byte aligned and looks at all the values in each one and guesses a datatype (int, uint, float) based on the range of the values and a few heuristics.

As I figure out what the columns in a given file do I give the columns meaningful names and tweak the data type if needed (or create custom handlers for stuff like the formula references). So far I've been focusing on tables as I need them for the item database site I'm working on (I've figured out most of ItemType and Items_* and AffixList). Other tables I've looked at and have a general idea of what's what, but haven't focused on really defining their formats yet.

So, back to the original question: I have a custom handler for those formula references that builds a string representation of the formula as well as the min and max values it can have and its worked on all the formulas in AffixList and the Items_* files. I certainly don't expect any problems with the formulas I haven't gotten to yet, but its possible there are additional opcodes that don't appear in the files I've parsed thus far.

Here's an example of the current output ("hash" is the hashed value of the affix name):

Code: Select all

RecNo       hash stat  param desc                                             min                 max 
----- ---------- ---- ------ -------------------------------- ------------------- ------------------- 
    6 2044719016  159      0 (rnd(18, 19) / 100)                0.180000007152557   0.189999997615814 
    7 3051218193  159      0 (rnd(20, 21) / 100)                0.200000002980232   0.209999993443489 
    8 1905952666  159      0 (rnd(22, 23) / 100)                0.219999998807907   0.230000004172325 
    9 3051218206  159      0 (rnd(24, 25) / 100)                0.239999994635582                0.25 
   10 2044719029  159      0 (rnd(26, 27) / 100)                0.259999990463257   0.270000010728836 
   11 3051218622  159      0 (rnd(28, 29) / 100)                0.280000001192093    0.28999999165535 
   12 1905966823  159      0 (rnd(30, 31) / 100)                0.300000011920929   0.310000002384186 
  114 1548086705  278 (null) ((250 + (rnd(0, 3) * 25)) / 100)                 2.5                3.25 
  115 3842221114  278 (null) ((350 + (rnd(0, 3) * 25)) / 100)                 3.5                4.25 
  116 2239245283  278 (null) ((450 + (rnd(0, 3) * 25)) / 100)                 4.5                5.25 
  117 3842221127  278 (null) ((550 + (rnd(0, 6) * 25)) / 100)                 5.5                   7 
 1035  982068194   88 (null) (rnd(21, 26) / 60)                 0.349999994039536   0.433333337306976 
 1036  982068195   88 (null) (rnd(27, 41) / 60)                 0.449999988079071   0.683333337306976 
 1037  982068196   88 (null) (rnd(42, 69) / 60)                 0.699999988079071    1.14999997615814 
 1038  982068197   88 (null) (rnd(70, 110) / 60)                 1.16666662693024    1.83333337306976 
 1039  982068198   88 (null) (rnd(111, 137) / 60)                1.85000002384186    2.28333330154419 
 1040  982068199   88 (null) (rnd(138, 162) / 60)                2.29999995231628    2.70000004768372 
 1041  982068200   88 (null) (rnd(163, 198) / 60)                2.71666669845581    3.29999995231628 
 1042  982068201   88 (null) (rnd(199, 225) / 60)                3.31666660308838                3.75 
 1043  982068202   88 (null) (rnd(226, 263) / 60)                3.76666665077209    4.38333320617676 
 1071 2076777148  178 (null) 0.01                             0.00999999977648258 0.00999999977648258 
 1072 2076777149  178 (null) 0.02                              0.0199999995529652  0.0199999995529652 
 1073 2076777150  178 (null) 0.03                              0.0299999993294477  0.0299999993294477 
 1074 2076777151  178 (null) 0.04                              0.0399999991059303  0.0399999991059303 
 1075 2076777152  178 (null) 0.05                              0.0500000007450581  0.0500000007450581 
 1076 2076777153  178 (null) 0.06                              0.0599999986588955  0.0599999986588955 
 1077 2076777154  178 (null) 0.07                              0.0700000002980232  0.0700000002980232 
 1078 2076777155  178 (null) 0.08                              0.0799999982118607  0.0799999982118607 
 1079 2076777156  178 (null) 0.09                              0.0900000035762787  0.0900000035762787 

I will be sure to acknowledge your contribution once I update my posts. Let me know if you want me to link to any source material that you may have already posted.

Thanks. I don't have any of this posted anywhere (just documented in the comments of my parsing program and in the database output).

PS: How did you figure this out? While it does make sense from a pure programming perspective, it seems far from obvious.

I was working on the Items_Armor file first and noticed that shield were the only items that referenced extra data (3 mods each). I also noticed that there wasn't any info about block chance/amount anywhere, so I figured that data had to represent block chance and min/max block value. When I looked at the data I saw those floats embedded in it pretty quickly (low bytes zeroed and high byte around 0x3F was a dead giveaway). I also realized that the low bytes of the non-floats were the same for all the shield mods. I spent quite a while trying to figure out a pattern to the rest of those bytes, but nothing seemed to fit. After a bit more staring and thinking about how I would serialize the sort of formula that I knew this must be, a light-bulb went off and I realized that those extra bytes were probably just garbage. After a few quick checks, that certainly seemed to be the case.

[Muaziz]

I guess I wasn't clear. I think this field is some sort of sequence number that's generated during the build process for each class (record type) this is being serialized into *.gam files. When the layout of these classes is changed, a new number in the sequence is used to represent the new format. My guess would be that there's an internal table mapping these type codes to information about the class that it should be loaded as (part of this information would be the offset in the 0x218 table, as well as the record size and the value to inject for the vtbl pointer field at the start of each record.)

Yeah, that was exactly my thinking although I did not word it well at all. I've updated it to (hopefully) convey this idea better.

Yeah. My parsing program right now reads in all the *.gam files (and *.stl and a few other things) and dumps it into a database. To start it just guesses the record size (using either the distance between those string fields or by using minimal-average-standard-deviation metric for files without string keys). It then assumes that columns are all 4-byte aligned and looks at all the values in each one and guesses a datatype (int, uint, float) based on the range of the values and a few heuristics.

Wow, that seems very complex (although elegant and more flexible than my approach). I just opened the files in a hex editor and the record sizes were trivial to determine (albeit, I had to do it on a file-by-file basis).

I've also been dumping all this information to a database with version info so I can look at diffs between patches. One of my goals is to be able to generate web pages with most of the important changes contained in a given patch file.

I have a custom handler for those formula references that builds a string representation of the formula as well as the min and max values it can have and its worked on all the formulas in AffixList and the Items_* files. I certainly don't expect any problems with the formulas I haven't gotten to yet, but its possible there are additional opcodes that don't appear in the files I've parsed thus far.

That will be my next step. And after that, I hope to get a web page up with all the affixes, their mod codes, and the range values for each mod. I'll want affixes to be linked to the item types so you can see all available affixes for a given item type. I'll need to add item type hierarchy support first though (which should be easy enough).

I was working on the Items_Armor file first and noticed that shield were the only items that referenced extra data (3 mods each). I also noticed that there wasn't any info about block chance/amount anywhere, so I figured that data had to represent block chance and min/max block value.

That much I had figured out a while ago...

When I looked at the data I saw those floats embedded in it pretty quickly (low bytes zeroed and high byte around 0x3F was a dead giveaway).

Add that is one of the crucial clues that I completely missed. I so rarely deal with floats that I didn't even recognized 0x3F800000 as the "1.00" floating-point value. Pretty impressive, I know...

I also realized that the low bytes of the non-floats were the same for all the shield mods. I spent quite a while trying to figure out a pattern to the rest of those bytes, but nothing seemed to fit. After a bit more staring and thinking about how I would serialize the sort of formula that I knew this must be, a light-bulb went off and I realized that those extra bytes were probably just garbage. After a few quick checks, that certainly seemed to be the case.

I had correctly assumed that the last 3 bytes of the (non-data) DWords were junk, but without knowing that the data values meant, I had no way to figure out the op-code approach.

Very well done!

I have now updated the main MPQ page (minor changes):
http://www.d3inferno.com/MPQ.html

and made significant updates to the AffixList.game page:
http://www.d3inferno.com/AffixList.gam.html

Thanks again for the insight. And if there's anything I can help you out on, don't be afraid to ask.

[chippydip]

Wow, that seems very complex (although elegant and more flexible than my approach). I just opened the files in a hex editor and the record sizes were trivial to determine (albeit, I had to do it on a file-by-file basis).

Yup. That's how my version 1.0 worked as well ;-) Initially I was just looking at the files one at a time and, once I had the record sizes figured out, I just dumped each column as an unsigned int. From there I just improved things bit by bit: First comparing all values in a column and dropping any column that was all zeros and then any column that was a constant value for all rows to help focus in on what mattered. Then after spending way too much effort plugging the numbers into a calculator to figure out if they made more sense as signed or floating point numbers, I wrote the first pass at column type guessing.

Around that point, I figured out how the hash values and file ids worked, so I tweaked things a bit to try and load in all the files so I could build a separate table with just the hash values to make it easy to figure out what table a given hash column was referencing (and then eventually a table with every file id in it). I auto-detected the record sizes for files with names in them, but just hard-coded the sizes for the ones without.

Then I realized that I had just been working off the base versions of the files and when I tried to switch to the newer versions stuff broke because some records sizes had changed. My code was getting a little convoluted anyway, so I decided to rewrite it at this point. I already knew all the record sizes, but figured it would be better if I could detect them on the fly to make dealing with future updates easier. I also improved the column type guessing code to the point where its pretty much getting things right 100% of the time now. All told, its made it much easier to figure out what's going on when mostly all I have to do is give the columns names once I figure out what those values stand for.

I've also been dumping all this information to a database with version info so I can look at diffs between patches. One of my goals is to be able to generate web pages with most of the important changes contained in a given patch file.

Yeah, I haven't been keeping track of version changes directly, but I have my extracted mpq files setup so that I can easily generate my database against whatever version of the game I want so at some point I would like to circle back and work out some diffs for an "undocumented patch changes" kind of list. Right now, though, I'm more focused on getting my weighted item stat searching code finished (think wowhead.com) so I can launch my item database site with something above and beyond what Blizzard has up on their official page. ;-)

[Muaziz]

Sorry for completely derailing this thread, but let me just finish off by saying that the process to 100% accurately bind the Items to the 2D images (located in the Textures files) has been posted on my site (D3Inferno). Check out the following two looks which should provide the full explanation:
http://www.d3inferno.com/Actor.acr.html
http://www.d3inferno.com/Textures.tex.html

[Muaziz]

FYI, in Patch 8610 (#13), the pHeader->dwSNOType can now have a value of 0x2D.

To get D3TexConv to work, you can make the following simple change to the source code in main.cpp:

enum eD3SNOTypes
{
SNO_MAGIC = 0xDEADBEEF,
SNO_TEXTURE = 0x2C,
SNO_SURFACE = 0x2B,
SNO_NEW_PATCH_8610 = 0x2D,
};

if(pHeader->dwMagic == SNO_MAGIC &&
(pHeader->dwSNOType == SNO_SURFACE || pHeader->dwSNOType == SNO_TEXTURE || pHeader->dwSNOType == SNO_NEW_PATCH_8610))
DecodeTexture((TexHeader*)(pBuffer + sizeof(SNOHeader)),szOutName);
else
(...)

[Necrolis]

FYI, in Patch 8610 (#13), the pHeader->dwSNOType can now have a value of 0x2D.

I've been meaning to actually update and finish this tool just haven't had any time. can you give me the names of any files with this new type?