Extracting the Civizard (MoM PS1 port) Tileset graphics & other stuff

[Blake00]

Hey guys,

I thought I had a thread here like I do at Slitherine but apparently not.. so here is a copy of most of my original 01/03/2020 post from there:

For those unaware of it, Civizard - Majutsu no Keifu is the Japanese playstation conversion/remake of Master of Magic. Its essentially the same game but with totally new graphics. It's pretty awesome!! Being able to extract those assets and use them in a mod for either MoM, CoM2 or MoM-IME would be pretty awesome!

Sadly there's no English version of the game but if it's of interest to anyone I've done a preliminary English translation on GameFAQs of its main interface which was not easy as I don't speak Japanese haha! I plan to play a bigger a game of it in the future and hopefully translate more. I even bought a physical copy of it and had it imported from Japan! (posted photos in the MoM FB group I run HERE )







As you can see in the above media the tiles appear to be stretched across a large world floor texture (kinda reminds me of oldschool Final Fantasy and SNES Mode 7 3D tricks) on an angle which makes screenshotting them impractical. That means the graphics need to be extracted out of the game and as far as I know no fan has done that sadly. If that was successfully done then presumably someone like Seravy or Serena would know how to extract and replace graphics assets in original MoM. 

.

[Blake00]

Originally posted at the Slitherine forums on 04/07/2020:

Forgot to say a while back that I spoke with @I like Serena on the MoM Discord and he confirmed that his MoM Tweaker tool can import and export classic PC MoM's assets

(Note: The post link below won't work unless you've joined the MoM Fans Discord using THIS LINK )

https://discord.com/channels/61272579399...7579184175
I like Serena 15/03/2020
Hey @BlakesSanctum,
Sure. All MoM graphics are available.
Tweaker can view, export, and replace most of them (LBX Editor).
And Tweaker's map view obviously has all the overland and combat tiles.

So theoretically his tool could be used to import the Playstation versions graphics into the PC version (although there will likely be some colour pallet conversions required beforehand). However the problem still remains of extracting the graphics out of playstation version of course haha.

.

[Blake00]

Originally posted at the Slitherine forums on 04/01/2022:

Someone was posting about this thread which reminded me that I haven't posted updates on our progress since the last post here.

(Note: The post links below won't work unless you've joined the MoM Fans Discord using THIS LINK )

Firstly one of our users Marzepain was able to extract graphics from the Civizard CD using a popular tool however sadly the extraction did not include tileset graphics.
https://discord.com/channels/61272579399...3635093524
"Marzepain:
If you need the playstation gfx of MoM, there is a tool for that. The playstation uses a standardised format while most other consoles like the snes don't. Unfortunately not all was retrievable. Somehow the map tiles are not there. Maybe because it's 3d. The opening sequence of Civizard is 218Mb,  the rest is just a few Mb. At least that's what I found.
cvzd.7z
934.20 KB"

Secondly I believe I was able to locate the graphics folder in which the tile graphics are stored in (probably the 3 WORLD files), however sadly I failed to find a way to extract them out. Implode also had a look but didn't have any luck either as we talked about the possibility of doing a Civizard graphics mod for his MoM-IME remake (which I should mention has recently finally hit version 1.0!) since that wouldn't have the colour palette limitation of the original engine I mentioned in my previous post.
https://discord.com/channels/61272579399...3617148005
"BlakesSanctum:
here's a 7zipped copy of all the files in that folder should anyone else want to play with them..
GRPH.7z
1,014.00 KB"
NOTE: I've attached the file to this post for those that don't wish to visit the discord

So we've made some progress but we still sadly don't have successfully extracted Civizard terrain that could be used in mods for classic MoM, MoM-IME or CoM2Win.

.

[Blake00]

Figured it's time for another update as there's been some more developments! A fellow named Starwisp visited the Discord and shared my interest in trying to extract the Civizard tileset to use in mods and had more sucess than anyone before but I haven't heard from him for a few months so I'm not sure if he got any further. Apparently a friend of his on a game hacking forum was writing a decompressor tool for the tileset files, not sure how far he got.

(Note: The post links below won't work unless you've joined the MoM Fans Discord using THIS LINK )

TILESET CONVOS:

Starwisp — 05/01/2022
Played around with the Civizard files for a bit and tested a couple programs on them. It is possible to extract some stuff but not a whole tileset: only wizard portraits, spell symbols, hero pics, some animation phases, some text, the logo etc. etc. .But I am not an expert on this stuff. I suspect it is either stored in a different way than the rest of the images or not in these files at all. edit: Found someone experienced in psx sprite ripping in another discord who is giving me a crashcourse in doing this. let's see - Civizard has proven to be a hard nut to crack so far...

BlakesSanctum — 05/01/2022
oh wow.. that's awesome! Let me know if you succeed! ?

Starwisp — 06/01/2022
I am now able to dump everything that is loaded into memory while someone plays the game. the world tileset seems to be compressed which is a problem. Edit: A bit later, I found someone even more knowledgable and he is now  writing a decompressor for the files.
here are some example tilesets - from the above described method.

   

SECOND IMAGE MISSING

Starwisp — 06/01/2022
We might get a full tileset of the world if we got someone who can read japanese to play this with a controller for a couple of games with different factions, colours and mages. You just need to configure epsxe or retroarch in a certain way and it'll dump all the loaded textures from vram. But it'll take someone to play this for a while. You need to make the game load every texture in the set once.
I'd prefer if we could just get the datafiles decoded instead.

BlakesSanctum — 06/01/2022
oh wow! this is great progress mate! and yeah.. it would be better if we can just extract the terrain files from wherever they reside rather than have someone play the game for ages as there's 2 big worlds filled with all sorts of stuff that may not appear near them lol
looks like something went wrong with the city building palette in that extraction too
but oh man that's great that you've found some gurus!!

Starwisp — 07/01/2022
as was explained to me, most of these tile sheets are present in a multitude of palettes. in every variation only some tiles are correct, the rest has the wrong colour. weird and unintuitive to me. but it seems to have to do with hardware limitations of the psx hardware.
the prerecorded soundtrack of the psx version is infinitely better than what we got on the dos side.
it just has a bit too much reverb. sounds like the recording took place in a church.
Is there a way to play without fog of war?

BlakesSanctum — 07/01/2022
hahaha yeah it does sound like that doesn't it ?
not that i know of but i bet the game does have cheats.. i wonder if they're written anywhere.. prob not on a english site lol

https://discord.com/channels/61272579399...8915637338

Starwisp — 07/01/2022
Hi everyone. I am trying to reverse engineer Civizard tileset as well. Does Civizard have the option of playing without the fog of war?

BlakesSanctum — 07/01/2022
Ahh good I'm glad you're here Starwisp as I was going to post here about your recent efforts to extract Civizard assets as well. You've made some good progress!

Marzepain — 07/01/2022
In MoMl there is Nature Awareness  https://masterofmagic.fandom.com/wiki/Nature_Awareness that does lift the fog of war. It's very rare Nature with a casting cost of 800. Quite difficult to cast early game.

BlakesSanctum — 08/01/2022
did some searches for cheats but as expected.. no joy.. but you can bet that the game has cheats

Starwisp — 09/01/2022
Thanks. I found some gameshark codes for unlimited gold and mana. but that's about it.
no direct linking possible with that specific game site, unfortunately, you'd have to select it in this list.

Starwisp — 10/01/2022
I was able to use these cheats on the game and got used playing with google translate. Turns out they did not implement buying buildings. They display the price, you can sell buildings but not rush the building process with gold. leveling up a city with unlimited gold fast and building good units early in order to get to "nature awareness" fast seems out of the question. I hope I am missing something here.

BlakesSanctum — 10/01/2022
wow.. I didn't notice that.. that seems really weird that they'd remove that buy/rush production button.. I'll have to look into that next time I play.

Starwisp — 11/01/2022
ok, found it by watching a letsplay. fortunately it came up in the first 2 minutes. The cheats I was using messed with the game and I had negative gold instead of unlimited gold.

BlakesSanctum — 11/01/2022
ahh thats a relief

Starwisp — 12/01/2022
just chatted with one of the admins of the biggest hex cheat site and he corrected the cheat for Civizard https://gamehacking.org/game/93741 these and so far only these are correct. we found out you can only raise gold and mana to a certain value until it turns negative.
the ones in epsxe for example are wrong. but I figure, now that the guys above have them corrected, the codes will spread in time.
I do hope, though that we'll never need to play MoM with a controller anymore once we have the tile set and someone created a mod or incorporates this in their project (maybe as an alternate tile set)
Civizard is really an exercise in patience....

JimBalcomb — 12/01/2022
Civizard is next on my list, so if you are doing recon work on the graphics/assets, do please keep me in mind.

Starwisp — 12/01/2022
that's what I've been doing for the last couple evenings now.

BlakesSanctum — 12/01/2022
well done mate.. you've got some great connections in the console community. It's a pity we can't lure anyone in those communities to patch translate the thing too haha. So many Japanese console games have been translated but Civizard never got that love. I posted threads about Civizard across heaps of console communities a few years back and no one had even heard of it haha.
Anyway yeah both you guys should keep in touch as Starwisp has made some good progress with extracting some assets out of Civizard and Jim has been doing lots of amazing work breaking down the PC version assets. Can definitely be some crossover there.

Starwisp — 17/01/2022
for sure. and as BlakesSanctum wrote, it might also lead to other things we can't eben think of yet
by the way, do you know if the original DOS tileset has been documented somewhere? the PSX Civizard tileset is driving me nuts at the moment.

JimBalcomb — 17/01/2022
I seem to recall seeing a pile of work by Drake178 on fandom wiki, or maybe on the Realms Beyond forum.

Starwisp — 17/01/2022
ah. okay. thx.

JimBalcomb — 17/01/2022
What exactly are you looking for there? Like where they are in the LBX's, the structure/IDs, or the bitmaps themselves?

Starwisp — 17/01/2022
well, the terrain tiles (world0, world1) have a different format than the rest. So far I know the there are 4 sets of tiles in each with most probably 10 palettes
I'd like to know if this is similar to the DOS version and if the tiles  have the same order
it'd be interesting to see how the DOS tileset for the terrain tiles is structured
I'd like to find out if/ or which of them are unused
do know that this is my first project of this sort and that I got sort of a crash course on this topic by some sprite rippers
they helped me a lot researching this
but the guys at ASMIK did a lot of non standard stuff on the PSX (at least as far as I can understand presently)
(Asmik was the studio that did the port)
anyway... it might be interesting to see how the spriteset looked like on DOS

JimBalcomb — 17/01/2022
Yeah, I'm not sure about those parts. I did a bit of digging a few years ago, but I seem to not recall the particulars.

Starwisp — 17/01/2022
no problem. I will dig a bit more. ? steady wins the race

JimBalcomb — 17/01/2022
That it does. I think the ol' aphorism of 'How do you eat an elephant?' really works well for these projects. ?

Starwisp — 17/01/2022
edit: WIP stuff...  I am with a lot of help very much on the right track... many tiles from the terrain files are pretty much unusable at the moment, though because of this or that.

Sadly that was his last message about this.. I'm trying to get in contact with him again now to see if any more progress has been made. For those interested there was also a interesting convo about accessing Civizards music files below...

MUSIC:
https://discord.com/channels/61272579399...3105810545

Starwisp — 09/01/2022
hi everyone. just an idea: one also could just use the music from the Civizard CD. Sounds like they sampled the midi file and put it on the cd.
maybe that would be an option?

Implode — 09/01/2022
Hmm that's interesting?  Wonder how we'd extract that off...

Starwisp — 09/01/2022
open it in jpsxdec
then you can see some wavetracks and some XA tracks
the wavetracks play normally and the XA tracks don't
the XAs seem to contain the most iconic tracks
XA seems to be a PSX specific format
I was just playing around with Civizard when I stumbled over this thread

Anskiy — 09/01/2022
Interestingly, I think CoM 2 actually used the PS1 tracks. And they're all stored as uncompressed .wavs [Blake: I think he's wrong on that one as the Civizard tracks sound different to regular MoM and would have been very noticable in CoM2]

Starwisp — 09/01/2022
funny that you mention reverb... the ingame audio of Civizard also has way too much reverb
otherwise it sounds terrific
ah, cool. so we can skip that ?
thanks for chiming in. ? This would have taken a while

Starwisp — 09/01/2022
just checked... jpsxdec also plays the XAs ? sounds great
The iconic  MoM theme has some jitter and crackling, though... I bet the CoM2 tracks were edited.
hmmm... just comparing the two... not sure it's the same. either they used some effects on them or just reran the midi files through a midi module... the Civizard tunes are way more in  your face...

[drake178]

Thanks for re-posting this here, and sorry I wasn't there for it! Unfortunately, I don't know the first thing about Playstation programming (yet), so this is going to take me a while to digest. In the meantime though, since it may be useful to others, I very much can share the missing information about how the tiles are handled in the DOS game.

Tile Types

Probably the first thing to know about overland tiles in MoM is that they are not organized by terrain types. That is, it's not a case of, say, "this tile is a forest, this one is an ocean, and they use this and this graphic". Instead, it's the other way around: the graphic (or sprite, if you will) index is the only recorded attribute of the tiles, and there's a whole group of functions that determine, from this value, the type of terrain that the tile represents. I wrote a C header file at some point in the past to convert the indices to something more "edible", you can find it here. There's also a simpler table for the basic types in Eljay's SaveGam2.html file (archived here).

It may be worth noting that terrain transitions (ocean to land, desert to non-desert, and tundra to non-tundra) always happen inside the tiles (as opposed to being rendered separately around the edges), such that the more generic "basic land" is always on the outer side(s) of the graphic. This results in 160 tiles each just for the previous transitions alone (having two adjacent sides also adds the corner inbetween, which at least cuts the amount of possible combinations down from 256). In fact, the main reason that rivers can't be created anywhere by the map generator is actually that not all 160 land-ocean transition graphics have equivalents with river mouths on some or all of their eligible edges. Hills and Mountains also connect into ranges - again transitioning into "basic land" on the outer edges of the graphic -, but in this case connectivity is in the cardinal directions only, yielding a mere 16 combinations each. Finally, rivers also work in a similar fashion, except there is naturally no "single tile river". Swamps, forests, and nodes are not connective tiles, while grasslands connect naturally to everything else. Of course, some common terrain types also have multiple graphics to make the maps look less repetitive.

Tile Maps

MoM uses several maps to describe the game world. Of these, the only one relevant here is the actual tile map. This is essentially a three-dimensional 2-by-40-by-60 array of 16-bit word values (since there are more than 256 tile types) stored sequentially, row-by-row in memory. In the save game files, the tile map starts at offset $2698. For simplicity and efficiency, the Myrror map uses the same tile indices as the Arcanus map (in fact, while many of the tile handling functions can convert/reduce the Myrran graphic indices, some would actually fail miserably if they were passed to them as parameters).

Map Window Drawing

The function that draws the overland tiles is at $D266B in WIZARDS.EXE. It essentially loops through the 12 by 10 visible area (or alternatively, a 5 by 5 in the city view), left to right, top to bottom, and copies the corresponding tile graphics directly into the current draw frame of the video memory. There are a few more additional bits here, such as adjusting the tile types for Myrror, not redrawing non-animated tiles if the map is only being refreshed instead of fully redrawn, and drawing only a black rectangle if the tile is not scouted by the player - but the most important aspect of this procedure is arguably the way that the graphics are processed.

TERRAIN.LBX

At a glance, this file appears to be a regular LBX file. It has a valid header and 3 entries. However, the first entry is not what it appears, and isn't read by the game through the normal LBX handling routines. Instead, the entire file is loaded into EMS as is, and the first entry (index #0), which holds the actual, decoded, ready-to-copy tile images, is referenced and read directly. The second entry (index #1) of the file holds the references for each tile in an array of 2-byte structures. The first byte is the index of the logical EMS page on which the graphic starts, while the second is the index of the individual graphic within that page. A set sign bit in the logical page index means that the tile is animated, resulting in its graphic index being automatically incremented by the current animation frame index (the sign bit is then cleared, so in practice, the page index is only a 7-bit value).

EMS pages are 16,384 bytes long each, and are mapped 4 at a time, sequentially into the 64K EMS page frame. The graphics themselves are uncompressed 20x18 bitmaps with an 8-bit depth (referencing the colors of the pre-loaded palette). They do have a very basic header to allow using the existing graphic overlay functions, which brings their size to 360+16=376 bytes. However, for efficiency, this is padded to 384 bytes, or $180. This is the multiplier for the second byte referencing the graphic's index on the EMS logical page. The tile images won't display in the Tweaker or a basic lbx viewer that is not retrieving them as custom graphics, because they don't contain the full header normally needed to describe an image (Implode's lbx extract tool has such code and can therefore retrieve the graphics).

In the case of a regular lbx-stored graphic, the header contains values such as the offset of the actual image data, the length of the image data (indirectly), whether the image uses "shading" (color replacement based on underlying pixels already drawn), whether it needs to be overlaid as multiple frames, and whether it requires custom palette colors (and what those are). These are all omitted from the tile graphics, instead being inferred by the drawing function. The header that the tile images do have contains only the image size (which is needed by the VGA overlay function to control the copy loop), and the decoded image identifier (either a leftover from previous iterations, or needed by the design tools to display the image, as the low-level function ultimately called does not use this for branching, that would happen at a higher level).

It may also be worth noting that the beginning of the first entry in TERRAIN.LBX is zero-padded such that the first image is aligned to a $180 boundary in the file itself (specifically to $300), as required by the graphic indexing. This can be relevant if someone wanted to add or remove entries in this file, as doing so may result in a shift in the location of the first entry, which must be compensated for.

Implications for Civizard

There are a few things I've noted right off the bat here. The top image has at least 7 different forest tiles (and I think there's even more in the video), which makes it very unlikely that the tile ordering/enumeration would be the same in Civizard as it is in the original. However, I would expect the same design principle of the tiles being defined by their graphic index, as that lends itself to the most efficient display, which is important for keeping the frame rate up. At the same time, the combined size of the WORLD0 and WORLD1 files in the attached archive (which I assume to be Arcanus and Myrror tiles respectively) is still less than half of the size of TERRAIN.LBX. The only way I can imagine that is if they are themselves compressed archives - especially if the graphics have a higher resolution and/or tile size than the originals.

From what I've read, the PS1 has stricter, or at least different memory constraints than a 4MB 386, so separating the tile sets for Arcanus and Myrror makes perfect sense - only one is necessary at any given time. I would guess that the entire set is loaded into- and kept in video memory, and from there copied onto some kind of "world plane canvas" tile by tile, which is then stretched onto a primitive. However, I'm not entirely sure that the same principle is used for the catchment view of the city screen. Is that also a 3D image in the frame? I can't tell. But if it isn't, then if all else fails, that could possibly be used to reconstruct the tile sets using screenshots alone. Not that I would want to do such a thing, seeing as it would likely take forever =P

[Blake00]

Thank you so much for this information Drake. If only you and Starwisp had somehow got chatting 6 months ago it would have been awesome haha. Oh well the good news is my fears of Starwisp being gone were wrong, he's still around and the project continues so hopefully you guys can have a chat and exchange some ideas sometime.

Would just be so great for the community if we end up with extracted Civizard assets to use in MoM modding projects. Hell I might even able to use them in some Civ mods too lol!

Indeed it looks like the world tiles are compressed in various clever ways inside those World0&1 files which has forced Starwisp and co to work on an extraction tool to try and get them out. Best you speak with him about specifics as they've definitely made good progress but maybe you can help with some of the road blocks they've hit.

As that certainly seems like the better option than the other idea of trying to extract assets appearing on screen while playing as it would be rather difficult to capture everything even with the cheats mentioned in my previous post, which we can also thank Starwisp and his mates for discovering as well. As you point out there's just so many variations to ensure less repetition. And that's a good question about the city radius view being a simpler 2D representation or still a 3D surface like the world view... it's hard to tell!

[starwisp]

Hi everyone. I'm happy that there is so much interest in this. It would amazing if the work that has been done can benefit some other project. Just a little headsup: I am in the process of putting everything we have done so far into one easy to digest package. It will take a couple days, but I am pretty far already. Starwisp

[Blake00]

Awesome! Thank you so much!

[Blake00]

Quick update for anyone interested. JimBalcomb who works on the MoM disassembly project has posted on the MoM discord that he's decided to start working on the Civizard graphics extraction issue too (so that they can be used for an addon to his MoM rebuild which would be cool) and has contacted Starwisp and they're talking shop. With big brains like all yours on it hopefully you guys can smash those final hurdles and the MoM community will finally have those super hard to get Civizard assets to do cool stuff with!

[starwisp]

This is what we (someone not related to this community and me) have been up to.
The Python script at the moment has bugs and is being worked on right now. We think we know what's wrong with it. It can already decompress most tiles but I would advise against using it until it is reworked.
Very WIP. Hopefully this is still an interesting read.

https://github.com/starwisp/Civizard-tileset-archeology