Macintosh (again!), bin conversion: YES!!! But...

[Souricette]

Hello again all.

I had a new idea concerning the Macintosh txt2bin conversion: comparing the "official" (embedded) bin files by the one generated by the patched Macintosh version (the ones with bad endianness), and trying to extract the fields which have to be converted by comparing these files.

I extracted all the official bin files, I generated the bad ones, and I developed a little application which tried to automatically parse the 2 folders and generates corresponding "ConvertRules" files which tell at what offsets are shorts and longs in a single bin table entry.

To my great surprise, ALL FILES except one have successfully generate a ConvertRules file - which means that we now have a way to generate bin files on Macintosh! (The converter has to be developed using the ConvertRules files - but that's not a problem!)
The way would be to ask the user to use the Patched application to generate badly-endianned bin files, and then to run the converter on these files to change the endianness at the corresponding offsets, based on the ConvertRules files.

There is ONE file which failed to follow my "rules": uniqueprefix.bin.
What is strange is that there is a bin row, corresponding to a UniquePrefix named "Black" (and only that one!), which is filled with 0 using Mac -direct -txt, while it is filled with 1 on the original bin file. Does it ring a bell in your head? I wonder why that single row for that single file has such behaviour...

Besides that, the good news is that we *may* have a very fast way to convert txt files on Mac too - and that it's quasi all-versions-proof (ie for 1.10, I'll just have to rebuild the ConvertRules files, and we will have a 1.10 Converter!)

Unsolved problems are the columns always set to 0 in original files, but which may be changed by mods - my ConvertRules file can't do a valid guess... For example, in the NPC.bin file, "questflag C" column is always empty, and is stored as a long at offset 84 (what I assumed by extrapoling "manually") - but since it's always 0, my ConvertRules builder has no way to guess if at offset 84, it's a long, or 2 shorts, or 4 bytes...
Same goes for columns filled with values which all are endian-compatible (for example, if in a column there are only 0x00010100 values, I can't guess what is representing that column)

But it's a big first step to a valid solution!

Souricette

[NewbieModder]

Earlier, before this was posted, I had an idea to eliminate the need for this and have D2 generate correct BIN's from the start.

You said earlier in a different thread that the solution to this problem is a C++ check and swap, correct?

So perhaps you could write/attach D2external (the C++ custom DLL) on a windows comp with the new DLL intercepting the bin generation routine and adding in that check. Then see if you could use the DLL to generate proper bins on a mac.

[Myhrginoc]

Souricette is working on a Mac, so DLLs and the whole D2External thing don't apply. The Mac OS doesn't have the same module structure as Windows does, and DLLs are purely Windows features.

Now that you have the actual converter code identified, you can find out what the bin format is by looking at the stack build as you enter the function. In the Intel version, that function gets called with five arguments:

Arg5 - limit of record length in bytes
Arg4 - location that receives the number of records (after definition it is a stack location)
Arg3 - location on the stack for the first field name (bottom of table structure)
Arg2 - location of the table name
Arg1 - always zero, as far as I have seen

Follow Arg3 and you should see a construct something like

Code: Select all

0012E260   6FDC580C  .XÜo  ASCII "Name"
0012E264   00000001  ...
0012E268   0000003F  ?...
0012E26C   00000000  ....
0012E270   00000000  ....
0012E274   6FDCB3D4  Ô³Üo  ASCII "Token"
0012E278   00000001  ...
0012E27C   00000002  ...
0012E280   000000C0  À...
0012E284   00000000  ....

which you will notice has five elements for each field in the original txt file. These elements are:

Field Name (location of predefined string)
Data Type
Data Length (if a string type it is nonzero)
Record Offset
Linking Address

The data types are, as I know them:

0 - null data, used by the End Record marker only - this is not the same as a Term field
1 - String (needs length field)
2 - dword
3 - word
4 - byte (signed)
6 - byte (unsigned)
9 – three and four letter codes
A – lookup code - four letter code as source
B – lookup code - four letter as pass-through (compiles to a byte)
D – lookup code - four letter as reference (compiles to a byte)
10 – lookup code - string as source (still researching)
11 – lookup code - string as reference (still researching)

and presumably there is code to compile the missing data types deeper in the function (for Intel people, it is Fog.10207).

So to find out what form a particular field should have, look at its stack construct. Then, if the field in the txt file is blank, you will have either a zero or a -1 value depending on the data type.

[Souricette]

I agree that *THE* solution would be to have all the stack tables for each of the txt files (70 for 1.09, it seems). I had stopped looking for them before - maybe I should not have. My "problem" is that the tool I am using to trace application execution is a MacOS 9 tool (Macsbug, for the ones knowing it), and my home computer does not like Macsbug - it crashes in video drivers when I boot using MacOS 9 and Macsbug... And I'm not used enough to gdb to start debugging in MacOS X.

Myhrginoc, since I'm not sure that the technical content of that specific point is interesting a lot of people there (and since it seems you are the one who digged the most in that part of the code), I'll contact you privately so that you can assist me in re-following that path.

Concerning the "uniqueprefix.bin" file mentioned in original message, I know now why that file had such behaviour: in the Patch file, there's only a uniqueprefix.txt file, not the corresponding bin file - so bin files are generated based on that "orphan" txt file, and will be different from the used one.

Thanks all,
Souricette

[Myhrginoc]

*This topic has been moved from Tools to Code Editing*

I'll just move this thread to Code Editing, where it will be appropriate to get down and dirty. Until we have a finished tool, there won't be much interest in the non-coder world anyway. And in case anybody is following along, I'll reprint your PM and my response.

I'll restart debugging Diablo II application to decode the bin tables - I don't have my notes here at work, so I can't go very deep now.

I remembered I had found the code you mentionned previously - the one doing OPEN-READ-CLOSE of a file. But I remember I had found no code doing massive stack filling. I even did not find any string containing "TLCData" (as mentioned in your example) in the executable...

Well, I'll investigate more at home - I have to work now!

I assume you are using as debugger, not just a disassembler. At the first point where you build the file string (e.g. "data\global\excel\playerclass.txt") you should be near the front of the function. Go forward until you find the first instruction. I don't know if Blizzard aligned the functions on 16-byte boundaries, but if they did you should see the Mac equivalent of a bunch of NOP instructions. Put a breakpoint on that first instruction. Now every time you call the function, you should stop right there---which will be every time you load a txt file that could be a bin file. Even if you aren't loading txt files you should hit that breakpoint, unless the compiler completely reorganized how the function is used. (It sounds like you figured out how to force it anyway.)

When you stop, look at your stack viewer. You should have a return address at the stack pointer, then the next five dwords should be your parameters. The middle one in the Intel version is the stack address of the table construct I referred to. Scroll through the stack and you should start seeing field names and the field structure I posted about.

[Paul Siramy]

[quote=Myhrginoc";p="101431"]And in case anybody is following along, I'll reprint your PM and my response.[/quote]
I am. I won't say I can folow all the technical parts in assembly, but it let me know if I need to keep thinking of create my own txt-->bin (not the reverse) convertor for mac or not. Luckily I won't need, seems it's appear to be more or less implemented in the Mac too.

[Souricette]

[quote=Paul Siramy";p="101497"]I am. I won't say I can folow all the technical parts in assembly, but it let me know if I need to keep thinking of create my own txt-->bin (not the reverse) convertor for mac or not. Luckily I won't need, seems it's appear to be more or less implemented in the Mac too.[/quote]

Yes, lucky guy... My quest is over, and I earned a few skill points and plenty of experience points to spend on auto-satisfaction skills...

Using wise Myhrginoc's indications, I extracted the different tables used by the application to convert txt files to bin files, and from these tables I extracted the offset of fields which are bad-endian-generated.

So, we now have a -direct -txt equivalent on Macintosh!!!

I generated a 1.0a4 version of my "Diablo II MPQ Tool" application (http://www.coucou.net/mpqtool/index.html), which enables the whole process.
In short, once you have your .txt files, you have to insert them in the patch file, then to use a patched Diablo II application to generate bin files in the DATA\GLOBAL\EXCEL folder, then run a converter on that folder to convert endianness of the bin files, and finally to put the bin files in the patch file. All these steps are explained and detailed in my application!

You know what?
I'm happy!

See you,
Souricette

[Alkalund]

*This topic has been moved from Code Editing to Mac Modding*