(October 26th, 2015, 06:31)MrBiscuits Wrote: I have to say these changes sound absolutely amazing, thanks for putting the massive amount of work in.
I have to ask how you are able to understand the code and mod the changes, could you give a very brief outline of how you do it? Also I am wondering whether you would consider making a repository for your code so others can see each change and where it occurs in the code.
Basically it's 6 steps
1. Disassemble the code with a disassembler (IDA is recommended)
2. Figure out what the referenced data structures contain (This one is hard but you only need to do it once per data structure)
3. You need to be able to read and understand 8086 assembly code and write new code for some of the more advanced tasks
4. You need to find where the code you want to change is within the exe and what is the problem with it in case of fixing a bug.
5. If the change needed isn't something as simple as replacing "add 3" with "subtract 2" you need to find useless parts of the code that do nothing and remove them, then rearrange code so that the empty space is where new code is to be inserted, all of this using just the hex editor. Including recalculating all offsets for all jumps pointing to the moved code.
6. After step 5 is done, you use an assembler to compile the code you want to insert and add it into the space you made.
Then you run the game and see if it works, and use the disassembler to make sure all offsets and code was properly inserted. Miscalculating the offset even by 1 makes the code unstable and do totally random things.
It usually takes many times of doing step 5 and 6 until the result works as intended, as it's quite easy to overlook some detail.
I'm not going to make a repository because it would increase the amount of work many many times.
Instead, after the mod is complete (or at least reaches 1.0), I will make a differential file of this and 1.40n and split up the changes into individual files addressing various fixes, then gather the general purpose ones and apply them to 1.40n and release it as a "Raid 1.50" bugfix release. The individual files will also be available for public.
Unfortunately a lot of the AI logic won't be compatible because some changes depend on others.
For example the change of Lionheart uses the space gained through the removal of Black Channels. The new AI logic for battle spell choices is inserted to where the choice for Word of Recall was, as that spell is now gone. A lot of AI changes are related to spells having new effect, and it would be a huge work trying to separate all of these from changes involving the unchanged spells. However, where it's obvious which change the code belongs to, will all be separated, eventually. I want to make sure everything works perfectly before I start doing that, there is no point separating individual changes when there still is a chance they need to be further altered.
Also, although I name the identified variables and such to make the disassembled code readable, those names are only saved for the original wizards.exe. The changed exe will not have the names inserted because it is a different, just similar exe file that has to be disassembler from zero every time I open it after a change. I don't think the disassembler has the function to export names from one exe to another, changed copy of the same file, although it would be nice if it could do that.
For reference, the two hardest things to do so far were
-Making the AI use the "grant gold for killing enemies" event which was completely unused. This required the making of over 160 bytes of space, and writing new code of that size (roughly 50 lines). Doesn't sound that much but the truth is 160 bytes are a lot when a single instruction is 2-6 bytes and you have to remove existing code to make the space. Fortunately compiled code such as wizards.exe often contains redundant calculations, but not always and not everywhere.
-Making Spell Ward counter combat spells of the same realm. This needed about 80 new bytes in a much tighter area than the previous, and it had to use data I wasn't yet familiar with, the "combat copy of city enchantments" one. I found that out the hard way, after seeing the normal city data is swapped out during combat from the memory.
Compared to this, changing alliance to ignore your units near towns was a single byte change, had to make the jump related to it go to the "end of procedure" instead of the "give the warning" and that's it. Of course, figuring out all the diplomacy variables to the point that I was able to recognize this code is giving the warning took weeks of reading the code and identifying parts and effects.
Below is the code I inserted for the gift gold event, and it had to be separated to 2 parts because that's how I was able to get enough free space.
cmp byte ptr [bp+0Ch],8
jnz skipall ; Action type=killed units in combat
cmp di,0
jnz skipall ; Performed by the player
mov cx,1
allwizardsloop:
cmp cx,si
jz nextwizard ; Skip the wizard being hurt
mov ax,cx
mov dx, 4C8h
imul dx
add ax, si
mov bx, ax
cmp byte ptr [bx-5FDEh], 3 ; 'üE
jnz nextwizard ; Skip if they aren't at war with the victim
mov ax,[bp-2] ;Use base diplamatic value not charismatic one
neg ax ; as a positive value
sar ax,1 ; Effect/8
cmp ax,1
jl nextwizard ; No effect if it's too weak
mov bx,cx
cmp byte ptr[bx-5FCCh],0 ; Skip if wizard already wanting to talk about something else
jnz nextwizard
mov byte ptr[bx-5FCCh],6 ; Event type 6 thanks for attack
mov bx,cx
add [bx-5FE4h],ax ; add the points
shl bx,1
mov [bx-5FD8h],20 ; counts as though it was a +20 effect to avoid overwriting with trivial things
mov [bx-5FC6h],si ; set victim as referred wizard
push ax
mov ax,cx
mov dx, 4C8h
imul dx
mov bx, ax
pop ax
add [bx-5FE4h],ax ; add the points to other side too
cmp byte prt [bx-5FDEh],2
jnz nextwizard ; done if no alliance, otherwise
mov dx, [bx-5DE0h] ; wizard's gold
sar dx,1 ; halve it, not giving more than half owned
cmp dx,0
jz nextwizard ; can't give gold if not having any
call alliance
alliance:
push bx
mov bl,64h ; Multiply effect strength by 100
mul bl
pop bx
cmp ax,dx
jle ok
mov ax,dx ; max gold given is what we have/2
ok:
sub [bx-5DE0h],ax ; deduce spent gold
mov bx,cx
mov byte ptr[bx-5FCCh],2 ; Event type 2 pay gold instead of just saying thanks
shl bx, 1
mov [bx-5EFAh], ax ; Offer this much gold*100
add [-5DE0h],ax ; player gain the gold
mov [bx-5FA2h],si ; set victim as referred wizard
retn
As you can see, the code tells nothing about where it goes or how the space is made for inserting, so it's not really useful for the public.
Thanks for your swift and comprehensive reply. It's more complicated than I originally thought!
I downloaded IDA and got it to dissemble the Wizard.exe, but I'm struggling to navigate to even start looking at the code. I have a tab called "Hex view A" that just contains lines like these and none of the sort of machine code that you wrote above:
seg038:0CC0 82 00 74 13 83 3E AA 82 00 7E 0C 0E E8 10 FF C7 é.tâ>¬é.~ Þ Ã
How for example would I look at the machine code for where the program "starts" ? And then how would you follow the code from that point?
I don't understand why you need to create space before adding new code. What would happen if you added code without freeing up the space first? Is there no way of adding things without removing other things first and just change the reference to point at the new space?
Point 5 in your list looks the most difficult to me, I hadn't quite realised how restricted you were when making changes. I wonder if some custom tools could be written to make this stage much easier and less painstaking. Or somehow find an alternative solution to stage 5 in your list.
Sorry for all the question, I've done a fair bit of coding but never anything quite like this before. I'm going from "maybe I could help out" to "how on earth is that even possible". If this is cluttering up your mod thread too much maybe we could move this discussion to a new topic instead.
Also I am curious, do you think it would be possible to rewrite the code from scratch using the disassembled code? Not that I think it would be a good idea and it is obviously a huge amount of work, it's just something that I'm wondering about. It might be easier in the long run than messing around with hex code for each tiny change.
And I'm glad that you are going to split up the changes and add them as a bugfix release. I had visions of you arriving in a whirlwind, make this amazing mod and then disappearing without anyone knowing how you managed to do any of it.
Hex view : This just shows you the bytes in the file. So if you want to find the machine code for some instructions, you can look here.
Ida view : This is the disassembled code. If it's not open by default then try View-subviews-disassembly.
Names : A list of stuff you have named after you identified them. To name a procedure or data reference, right click it and select rename.
Functions : A list of the functions/procedures in the file.
Strings : these are just the string constants found in the exe. Not all of them are actually here, but most are. This view is useful if you know the text the game displays and want to find where it gets displayed. For example you want to allow building when there are already 9 units in town then you search for that message.
There are more views but they aren't very important for now.
Quote:I don't understand why you need to create space before adding new code. What would happen if you added code without freeing up the space first?
You can't. If you add even a single extra byte, everything in the file after that will be at a different address than where the loader looks for it and the game would not even load, as well as all jumps between the two parts being off by one byte.
Quote:Is there no way of adding things without removing other things first and just change the reference to point at the new space?
You can jump to a new space, yes, but you need to MAKE the new space first. There aren't miraclous unused large spaces in the file everywhere for us, sadly. And, it has to be in the same segment, otherwise the process is ultra-difficult. Also if the jump you modify is a byte value then it has to be within +-127 bytes, otherwise you need to make +1 byte space for a larger jump instruction.
Quote: I hadn't quite realised how restricted you were when making changes. I wonder if some custom tools could be written to make this stage much easier and less painstaking. Or somehow find an alternative solution to stage 5 in your list.
Not really. You literally have to look for "adds 2 to the value, then throws the result away and overwrite it with 5." style of coding where you can remove the "adds 2" part without changing the overall effect. In other words, use your brain, so it cannot be done automatically, only humans are able to do this part.
In easy cases one such place can yield 9-15 bytes of space and you can find 6-8 of them near together.
In hard cases, you have to actually re-code stuff to make it use less space through code optimalization.
For example to add the "Turns = x" display in combat, I had to recode it from
set position X=45
set position Y=60
set text "mana"
call text display
set position X=55
set position Y=60
convert amount of mana to text
call text display
set position X=45
set position Y=60
set text "skill"
call text display
set position X=55
set position Y=60
convert amount of skill to text
call text display
into this
set a=55, b=60, c="mana", amount = amount of mana
call "displaytext"
set b=70, c="skill", amount = amount of skill
call "displaytext"
set b=80, c="turns", amount = amount of turns
call "displaytext"
displaytext:
set position X = a, position Y=b
set text "c"
call text display
set position X= a+20
convert amount to text
call text display
as you can see "displaytext" is the original second half which displayed the "skill" line, except it uses the passed parameters instead of the original contants. The first half to display the "Mana" line is removed, and calls the second half 3 times, with 3 sets of parameters to display 3 lines that are no longer limited. To further restrict the options, "Turns" had to exist as a text constant available and ready to use in the data segment, fortunately it did, as it is displayed elsewhere. You can see it uses the text belonging to another game part because "Mana:" and "Skill:" has a ":" but "Turn" does not, it was only available without it. Otherwise I would have also needed free room in the data segment to add new text (I guess there are some unused texts like "as swiftness spell" which kryub made not to appear").
In this particular case there was no space that could be made, and as far calls are unmovable, I had to re-use the same one to be ran once more than originally to display more lines of text. Took about 4 hours to get it work.
Assuming you have a large block of free space somewhere (I actually do because I replaced a huge AI routine somewhere so the original is not called anymore) you could in theory get away with only making 3 bytes of space, and using a near call to it IF you are within the same segment. There are about 160 segments so guess what the chances are for that? Yeah... my nice huge empty space will not be useful, it's in a segment where I don't need to add stuff at least at the moment.
By the way, if you are looking for a variable or a procedure in particular, it might be faster to ask than to try find it amongst the 3500 procedures the game has yourself. Even I only identified about one third of them so far.
(October 26th, 2015, 08:26)Seravy Wrote: You'll need these views :
Hex view : This just shows you the bytes in the file. So if you want to find the machine code for some instructions, you can look here.
Ida view : This is the disassembled code. If it's not open by default then try View-subviews-disassembly.
Names : A list of stuff you have named after you identified them. To name a procedure or data reference, right click it and select rename.
Functions : A list of the functions/procedures in the file.
Strings : these are just the string constants found in the exe. Not all of them are actually here, but most are. This view is useful if you know the text the game displays and want to find where it gets displayed. For example you want to allow building when there are already 9 units in town then you search for that message.
The IDA view is weird. If you click on a name you get some code like you have below, but if you scroll using the scrollbar then it turns into a graph and you can never get back to the view you had before.
I had a function tab, but it has disappeared for some reason.
Quote:You can't. If you add even a single extra byte, everything in the file after that will be at a different address than where the loader looks for it and the game would not even load, as well as all jumps between the two parts being off by one byte.
Dumb question, but could you update the loader to point in the right place after you change the files?
Quote:You can jump to a new space, yes, but you need to MAKE the new space first. There aren't miraclous unused large spaces in the file everywhere for us, sadly. And, it has to be in the same segment, otherwise the process is ultra-difficult. Also if the jump you modify is a byte value then it has to be within +-127 bytes, otherwise you need to make +1 byte space for a larger jump instruction.
When you say segment, do you mean in the same address in memory? How large is a segment?
Quote:Not really. You literally have to look for "adds 2 to the value, then throws the result away and overwrite it with 5." style of coding where you can remove the "adds 2" part without changing the overall effect. In other words, use your brain, so it cannot be done automatically, only humans are able to do this part.
In easy cases one such place can yield 9-15 bytes of space and you can find 6-8 of them near together.
In hard cases, you have to actually re-code stuff to make it use less space through code optimalization.
For example to add the "Turns = x" display in combat, I had to recode it from
Very interesting. How do you calculate the number of bytes that you've saved?
Quote:as you can see "displaytext" is the original second half which displayed the "skill" line, except it uses the passed parameters instead of the original contants. The first half to display the "Mana" line is removed, and calls the second half 3 times, with 3 sets of parameters to display 3 lines that are no longer limited. To further restrict the options, "Turns" had to exist as a text constant available and ready to use in the data segment, fortunately it did, as it is displayed elsewhere. You can see it uses the text belonging to another game part because "Mana:" and "Skill:" has a ":" but "Turn" does not, it was only available without it. Otherwise I would have also needed free room in the data segment to add new text (I guess there are some unused texts like "as swiftness spell" which kryub made not to appear").
In this particular case there was no space that could be made, and as far calls are unmovable, I had to re-use the same one to be ran once more than originally to display more lines of text. Took about 4 hours to get it work.
So you literally have to search the code to find somewhere to save a couple of bytes somehow and then code up to that exact amount of bytes the changes you want to make. Amazing.
Quote:Assuming you have a large block of free space somewhere (I actually do because I replaced a huge AI routine somewhere so the original is not called anymore) you could in theory get away with only making 3 bytes of space, and using a near call to it IF you are within the same segment. There are about 160 segments so guess what the chances are for that? Yeah... my nice huge empty space will not be useful, it's in a segment where I don't need to add stuff at least at the moment.
Did you mention far calls? Would they be able to access space in other sections?
Quote:By the way, if you are looking for a variable or a procedure in particular, it might be faster to ask than to try find it amongst the 3500 procedures the game has yourself. Even I only identified about one third of them so far.
I was interested in looking at the part that deals with how many spells you can pick from common, uncommon and rare. Mainly though I was wanting to know roughly how modding the game works at all and what limitations there are for future changes.
How do you go about finding a piece of code in game if it isn't found already?
Quote:Dumb question, but could you update the loader to point in the right place after you change the files?
Well I guess you could, in theory, update the header to point to the correct places, but you would still need to manually adjust all jumps and calls inside that segment as well as those from the outside pointing to it. I think that would be like, 10x more amount of work as well as being that much harder and with more chance to mess up too.
Quote:When you say segment, do you mean in the same address in memory? How large is a segment?
Okay, I think you need to first find a book or online information about how the 8086 processors address memory.
A segment is a max of 64k bytes in terms of memory addressing, but actual code segments and data segments are as long as needed for the code to fit. The only restriction is max 64kbytes, and have to be divisible by 16.
Quote:Very interesting. How do you calculate the number of bytes that you've saved?
Well, you locate the code you want to remove in the hex editor, overwrite it with zeroes or 90s whichever you prefer, and it's that many bytes, you see on the screen, as simple as that.
Quote:Did you mention far calls? Would they be able to access space in other sections?
Yes, but don't do it. It's extremely complicated. I've only ever used it once, and to a very severe bug that couldn't have been fixed in any other way, in particular the bug that autosave was called halfway through the end of turn not when it was actually finished.
Quote:I was interested in looking at the part that deals with how many spells you can pick from common, uncommon and rare. Mainly though I was wanting to know roughly how modding the game works at all and what limitations there are for future changes.
I've only found that part for the AI yet.
Quote:How do you go about finding a piece of code in game if it isn't found already?
You search for
-references of a related variable or data structure, in this case the table where the number of picks are stored could be a good plan. Spells in current wizard data would be another. (wizard profiles are DS:6900h in magic.exe, format is identical to what you see in tweaker.)
-references of a text constant related to it, in this case probably "Common" "Uncommon" "Rare" "Very Rare" can help
-reference of an offset related to what you need, for example Sawmill is 34h in the city data (it isn't just inserting a random number) then you search for the value "34h" in the file. There will be a few hundred results, but one of them will be what you need. Of course this only really works for bigger numbers, stuff like 0, 1 or pretty much anything below 20 are far too common.
Word values are pretty easy to find, like 6475h would only get a few results, if any.
-You can also look at a code part you already found which is related, and go through procedures it calls. Like if you know "End_of_Turn" then you can find anything that's done at end of turn, but you do have to look through hundreds of calls. In other cases the relation is closer and it's easier to find for example "casting a combat spell" and "AI selecting a spell to cast".
(October 26th, 2015, 14:54)Seravy Wrote: "Please use a newer version of IDA to open this database"
Wait what? I downloaded the only free one from the official site.
Do I need to pirate a pro version or something to open these files?
Erm... I'm using IDA Pro Advanced 5.5.0.925t (32-bit).
Sorry if that means you can't open it.
Well, a certain torrent site can always help I guess. I really hope the new version will be able to open the old files I have, though.
Thanks for uploading them, I'll do something about this "problem" of mine
...finally, opened. Wow, you even added the diplomacy data I found already!
Holy, so much information! This might actually help me figure out how the AI is moving overland units.
hi. ive recently started modding mom after stumbling across this page. i used the tweaker at first and eventually sat down and started disassembling. i never used a disassembler before, but serenas database proved to be very helpful.
i did a couple of small things first and then happened to find the function that displays the item window (if you right click an item). since i have a number of items with more than 5 powers, i thought it to be a good idea to have this function display more than 5 powers.
i first wanted to increase the size of the window but that seemed very complicated or impossible. i found out how the strings for the item powers and the descriptions (the part after the comma) are made and so removed the descriptions instead to make room for more powers. i reused the stack adress for the descriptions to take the powers instead, because the descriptions are longer. this part worked fine.
next up i tried to split up the window into to columns of 5 powers each, in case there were more than 5. that didnt work, whenever the item had more than 5 powers, the strings and icons would appear at weird locations and occasionally jumble around. so i tried to jump back and forth between the 2 columns working from the top down. this didnt work either.
in fact, what happens is that the strings change their (utterly wrong) location when the mouse is moved / pressed. the icons dont appear at all.
i dont have an idea why this would happen, i have no experience writing assembly language but from the looks of it, the code is fine. ida can read it and creates a diagram from it that looks like it should.
the memory on the stack i use for my vars should be free as it is in the area that would normally be used by the strings for the item powers. i dont knowingly use any registers other than the ones that are used originally in that location. my best guess at this point is that the game translates my code differently than ida and somehow reads in mouse pointer data.
ive uploaded the databases for the 2 functions on my github. the far calls are based on my wizards.exe and are different from the ones used in serenas db. im unsure tho which version that is based on or which one i am using. in the strings function i only changed a few bytes however, and in the display function the changed code block contains only 2 far calls . (4 originally, 2 are removed)
the get strings function goes from offset 5e5cf to 5eb0a. named sub_69d8f in serenas db
the display function goes from offset 617fc to 619c4. named sub_6cccc in serenas db
