Data extraction thread

[Guest]

I'll try to better understand how engines use .ks files.

 

In older games .ks often weren't encrypted. So I could open them with a siple text editor.

But it seems this doesn't happen with more recent games.

 

I don't have great knowledge about the different types of encoding the engines use yet.

 

Until now this thread has only been a tool to extract stuff out of older game files.

I'll try making it useful for more recent games in the future.

 

@Lolisora

If checkmate and moemoe-kyun agree, would it bother you to tell me the extraction process so I can add it to this thread ?

[メルP]

Hey guys, I need help figuring out what to do with S25 and OGV files, Google didn't help either. Hoping to get the script data out and start translating.

 

(Solved thanks to Checkmate and moemoe-kyun from AnimeShare)

ogv sounds like theora videos. So whatever you're looking for is probably in S25

[Blake]

I'll try to better understand how engines use .ks files.

 

In older games .ks often weren't encrypted. So I could open them with a siple text editor.

But it seems this doesn't happen with more recent games.

 

I don't have great knowledge about the different types of encoding the engines use yet.

 

Until now this thread has only been a tool to extract stuff out of older game files.

I'll try making it useful for more recent games in the future.

[ImperialRex]

Hi, I need some help, recently I downloaded White Album -Tsuzurareru Fuyu no Omoide- for PC

And I try to use  the tools in this thread, I extracted the data from the pck file with "exwatfopck"

I found .bin files that contain voices, sounds, images but the text scripts. Then I look for those scripts, I found file: "scripts.sdat" (not  .dat). I opened with 010 editor and I found something, first of all the top show me some meaningless katakanas, then around the line 1150h I noticed some words, and beggining i didn't understand, but then I realize that says: P00110.MES.BIN.

P00120.MES.BIN...... and continue until the line 5220h. Then finalice with the words: SCRIPT.FLAG.... pack

Then in the line 7500h I found the text of the game 「由綺は今年さ、学園祭来られそう？」

So I want a tool to extract the text from the sdat file, i look in google but I only found tools to nds and ps3 but those tools don't work. I leave you the sdat file, so you can find a tool or build a new one. Scrpt

 

[The Major]

Hi, I need some help, recently I downloaded White Album -Tsuzurareru Fuyu no Omoide- for PC

And I try to use  the tools in this thread, I extracted the data from the pck file with "exwatfopck"

I found .bin files that contain voices, sounds, images but the text scripts. Then I look for those scripts, I found file: "scripts.sdat" (not  .dat). I opened with 010 editor and I found something, first of all the top show me some meaningless katakanas, then around the line 1150h I noticed some words, and beggining i didn't understand, but then I realize that says: P00110.MES.BIN.

P00120.MES.BIN...... and continue until the line 5220h. Then finalice with the words: SCRIPT.FLAG.... pack

Then in the line 7500h I found the text of the game 「由綺は今年さ、学園祭来られそう？」

So I want a tool to extract the text from the sdat file, i look in google but I only found tools to nds and ps3 but those tools don't work. I leave you the sdat file, so you can find a tool or build a new one. Scrpt

[ImperialRex]

Well, I'm not sure if it helps, but you can also use exwatfopck to extract those BIN files you were talking about (1,107 of them) from Script.sdat. These seem to have Japanese text in them (what I assume are lines from the VN), so they may be files you want. I'm not really sure.

It works, thanks. And the text inside is really the text in the VN. 「由綺は今年さ、学園祭来られそう？」 means something like: Is Yuki coming to the school festival? (sorry I'm not native english)

Thanks again.

[raphael512]

hey dude regarding the zlib1.dll error in the exfavbin.. you just need to download the dll file and put it in the same folder with exfavbin

[Guest]

o_0 ... It works ...

Thank you very much ! =D

[Kelebek1]

So how does the encryption flag work in kirikiri engine? Is it even the encryption flag? In Imouto to Sono Yuujin it's set to 0x80000000 for most files except the protected archive and images, and all the files it's set for are encrypted. But in 迷える２人とセカイのすべて by Lass, all files except the protected archive are set to 0x80000000, but *none* of them are encrypted. If it's not the encrypted flag, then what is? or why would it be set if the files aren't encrypted? Do other games set the encryption flag even with no encryption on the files? How to detect and ignore that?

[Guest]

So how does the encryption flag work in kirikiri engine? Is it even the encryption flag? In Imouto to Sono Yuujin it's set to 0x80000000 for most files except the protected archive and images, and all the files it's set for are encrypted. But in 迷える２人とセカイのすべて by Lass, all files except the protected archive are set to 0x80000000, but *none* of them are encrypted. If it's not the encrypted flag, then what is? or why would it be set if the files aren't encrypted? Do other games set the encryption flag even with no encryption on the files? How to detect and ignore that?

[Kelebek1]

Is that relevant in this thread ?

[binaryfail]

The flag doesn't seem to have any visible effect on the use of the archive. I would just completely ignore that flag and rely on the adler32 hash to verify that a file has been correctly read from the archive.

[Kelebek1]

Ah yeah, using the adlr is a good idea, thanks. The sections of each file are written in different orders too. In Imouto to Sono Yuujin it goes info > segm > adlr, whereas in 迷える２人とセカイのすべて it goes time (not even included with the protected file, but the rest) > adlr > segm > info.

 

Do you know if these can be left out/written in any order? It'd be a real pain if the sections need to be written in a specific order, changing for each game. Order based on version maybe?

[Guest]

Yes, because it's to do with extracting .xp3 files for games that use the kirikiri engine (which is a LOAD of them)? There's even some tools in the first post about .xp3 as well as some posts about .xp3 files in this thread. It's also the format that uses the .ks files which you'll find people talking about (including you) even just a few posts above, about the very encryption I'm talking about... So I'm not sure how it wouldn't be relevant to this thread...

[binaryfail]

Chunks can be read and written in whatever order you prefer.

For repacking, you can omit the 'time' chunk... the other three (adlr, segm, info) are necessary for a working archive.

[Kelebek1]

Alright, cheers. I do remember that I couldn't put unencrypted files back into the archive for Imouto to Sono Yuujin, even when I set that flag to 0. I don't know why they even have that flag if games just completely ignore it. It really forces xp3 to be set per game which is a big pain. >:

[doppel222]

I want to extract .arc files and get the scripts but nothing seems to be working.

 

Update: I used extractdata, crass, yuno.. everything failed.

Upadate 2: The novel uses Buriko engine (BGI), I heard many people are having problems about this. I think I'm just going to give up on this. I don't even know how to use the Burikotools in TLWiki. *sigh*

[Kelebek1]

What game is it for?

[doppel222]

For Ao no Kanata no Four Rhythm (Demo)

[Yotek]

First off, thanks for the amazing thread. It has been a great help.

 

First time actually TLing a VN. Thought I would pick a short easy one that looked interesting. Decided to pick up "CoO Geass" -Inyoku no Terrorist- (https://vndb.org/v5434) VERY VERY NOT safe for work.

 

Had trouble extracting the xp3s with crass and kikiri2. Both would give me random garbage.

 

The good news is I was able to extract with xp3viewer.

 

The reason for my post is now that I have this... How am I going to make the patch.xp3 since I am having no luck with the normal tools? Any suggestions would be appreciated.

[Kelebek1]

First off, thanks for the amazing thread. It has been a great help.

 

First time actually TLing a VN. Thought I would pick a short easy one that looked interesting. Decided to pick up "CoO Geass" -Inyoku no Terrorist- (https://vndb.org/v5434) VERY VERY NOT safe for work.

 

Had trouble extracting the xp3s with crass and kikiri2. Both would give me random garbage.

 

The good news is I was able to extract with xp3viewer.

 

The reason for my post is now that I have this... How am I going to make the patch.xp3 since I am having no luck with the normal tools? Any suggestions would be appreciated.

 

Could you upload it somewhere? I can't find the game available absolutely anywhere, and there's no trial.

[Guest]

First off, thanks for the amazing thread. It has been a great help.

 

The reason for my post is now that I have this... How am I going to make the patch.xp3 since I am having no luck with the normal tools? Any suggestions would be appreciated.

 

Thanks, I'm happy it could help,

 

Did you try AnimED ? It should be able to repack into xp3.

[Yotek]

Here is a link to the game. I will be testing out AnimED shortly to see if I can get any results. Thank you both.

 

https://mega.co.nz/#!Qt4ljBTA!PxhFBNdDCgZtkwl6fx4NEIUGXTzntnSUfkaLtAFOCA8

[Kelebek1]

I tested out another one of their games and got the extracting working, but the script (.ks) and the code files (.tjs) aren't in a readable format, and they should be. I've verified that the file hashes match, so the files are exactly what they should be, but I can't find any encoding where the files are readable at all. They're not sjis, utf8, utf-16 in either endian, EUC. I'm pretty stumped. Maybe this game is different though.

 

EDIT: Ok I've managed to extract and decrypt, but there is NO way I have any idea how to re-crypt these at all. That's way out of my league. So, it would have to be an exe edit unfortunately (and I don't think I could even do that).

 

Did AnimED work for you?

[ILOVEPIE]

If anyone's feeling adventurous I could use some help with the .enc format of High School DxD for 3ds.