Data extraction thread

[Haoren]

Does anyone know how to repack .fpk files properly? I tried using AE as described in this thread, however the size balloons to x10 and the VN does not run. Placing the original data.fpk file allows the game to run properly. As for the engine, I think the game is using System-C, since there is a config file with that name.

[Anonym271]

Yes, I once wrote a tool that could repack SystemC FPK files and I also implemented a compression algorithm for it. Can you tell me what game you are working on? Because I believe there are at least two versions of the file format and my program only works with one of them.

However, you could also just try it out: https://github.com/Anonym271/betterfpk. You can find an EXE in the releases section or of course clone the repository and modify it to your desires.

It's a command line tool, use it like this:

Extraction:
betterfpk.exe -e <my-archive.fpk> [output-directory]
  
Repacking:
betterfpk.exe -p <my-folder> [output-file.fpk]

It's not really fast and the compression results also aren't that great, but it should do the job I guess.

[Haoren]

  On 2/28/2022 at 10:40 PM, Anonym271 said:

 

Extraction:
betterfpk.exe -e <my-archive.fpk> [output-directory]
  
Repacking:
betterfpk.exe -p <my-folder> [output-file.fpk]

 

Expand  

I'm getting an 'The input path does not exist error'.

[Anonym271]

  On 3/1/2022 at 10:57 AM, Haoren said:

I'm getting an 'The input path does not exist error'.

Expand  

Hm, this really should only be thrown if you entered an invalid path. What command exactly did you run? And just in case: of course you have to leave out the brackets, so an example command would be:

betterfpk.exe -p cg cg.fpk

 

[Anonym271]

@Haoren So... did it work? If not, feel free to tell me the name of the game (or give me a sample package) and I will try to adjust the program to that version.

Edited March 3, 2022 by Anonym271

[Vincent Dawn]

I'm having some trouble trying to find character CGs for Gachinko! Bitch Club. I'm using Garbro to extract the data and I can get CGs for events, backgrounds, opening stuff (like the logo, text, etc.) and the section that records event CGs, but there are no CGs for the characters. It seems like someone was able find them since there is a character CG set on e-hentai, but I can't seem to find them myself. Has anyone else had this problem or has found a solution?

[zerostage2]

So I tried to extract some of Atelier Kaguya's game with asmodean tool and some of the sprite that get split have XY coordinate in its name, is there any program to easily merge the two image? I've been using paint.net but it takes a while

[Anonym271]

  On 4/26/2022 at 10:04 AM, zerostage2 said:

So I tried to extract some of Atelier Kaguya's game with asmodean tool and some of the sprite that get split have XY coordinate in its name, is there any program to easily merge the two image? I've been using paint.net but it takes a while

Expand  

Can you provide some samples? I don't think there is an automated tool for this but if the file names already contain coordinates it should be relatively easy to create a simple script that does the job.

[loam]

I'm releasing a NekoNyanDatTool update (previously known as AokanaDatTool, but renamed because several games now use this engine).

It can now (un)pack Parquet (Mobile/Unity) and Senren Banka Switch/Unity files. Both games use the "aokana_switch" format; I've added aliases for those for ease of use, "parquet_mobile" and "senren_switch" respectively.

The reason it couldn't before was because the tool tried to decode the file names as ASCII, but Parquet Mobile and Senren Banka Switch have UTF-8 paths (due to japanese filenames). This has been fixed.

Binary: https://www.mediafire.com/file/0cw55ccjp2c77cn/NekoNyanDatTool.exe/file

Source: https://www.mediafire.com/file/3cdhkgijn56t4tn/NekoNyanDatTool_26052022.7z/file

Man, the bs5 is so clean compared to whatever mess this psb shit is, fucking M2 Cheeseware... I guess I'll be busy trying to port scenes to the Switch. Also I wonder why they had to bundle in e-mote for the relatively basic subset Senren uses...

Edited May 26, 2022 by loam

[scarlet]

I try to extract file xp3 but always like this. So I try to extract other file xp3 but still error. Can someone help me please?https://drive.google.com/file/d/1pd-yOimLH-iNL9kaN6K1RmanIPuahtpw/view?usp=sharing

[Kurobuns]

Hi. I've been trying to extract the assets from the PS2 version of Lamune, and I could use some help. This version of the game has exclusive content (new routes with new heroines, CG, and I think also new music) that never made it back to PC, and I really want to extract and preserve that.

 

I made a thread on the Zenhax forums previously, and I've been able to find out that the engine is by HuneX, and I've been able to extract data from the main DATA.BIN file, which includes raw videos and the rest of the data packaged in .dat files. Sadly, I haven't had luck extracting anything from those .dat files. 

 

It seems zoom909 worked on this project before, but sadly I don't think any tools were shared for extracting these assets, if he ever found a way.

 

I tried the recent PCSX2 graphics dumping tools but sadly it wasn't able to dump the main assets properly. It just outputs images in tiny chunks, and in a way that I don't know if it can be properly put back together.

 

Here's a google drive folder with the original DATA.BIN file and the extracted .dat files and movie files, in case someone wants to try poking at it. Any help would be super appreciated.

[ThugzillaDev]

  On 3/29/2019 at 1:31 AM, pmrowla said:

I've been working on a tool for extracting and patching LiveMaker/LiveNovel games and figured I should post it here. It started out as a fork of tinfoil's irl which I guess was kickstarted and then abandoned a while back?

Current status is:

• LiveMaker assets can be extracted from .exe or .dat files
• LiveMaker's binary .lsb format for their "chart" scripting can be dumped as text or xml
• LiveNovel scripts can be extracted from LiveMaker's binary .lsb format
• (Translated) LiveNovel scripts can be compiled and inserted back into a .lsb file
• Modified .lsb's can be patched back into a livemaker game and run with (cp932 encoded) english text, but currently the application will lock up when you try to exit.

My current code is here: https://github.com/pmrowla/pylivemaker/tree/wip (you have to use the wip branch link since I'm not pushing anything into master until I figure out the crash on exit thing)

The code itself is mostly fully commented but I haven't published any of the actual docs yet, so there's a github issue w/a summary of what works right now and what doesn't: https://github.com/pmrowla/pylivemaker/issues/1

 

Honestly idk if there's even any livemaker games worth patching but I was bored and felt like reversing something so I picked an obscure engine that afaik hasn't been dug into very much before. I think GARbro supports viewing .gal CG's from LiveMaker archives, but that doesn't cover the text extraction and patching side of things.

Expand  

Hello! I wanted to extract the visual novel "Will Melt": 

It uses Livemaker. I downloaded your github package, but I'm having trouble using it for the game's .exe or .dll. Is there a specific file in the pylivemaker I'm supposed to use? I'm very new to this so any help is appreciated

[pmrowla]

  On 9/15/2022 at 11:23 PM, ThugzillaDev said:

Hello! I wanted to extract the visual novel "Will Melt": 

It uses Livemaker. I downloaded your github package, but I'm having trouble using it for the game's .exe or .dll. Is there a specific file in the pylivemaker I'm supposed to use? I'm very new to this so any help is appreciated

Expand  

answered in PMs, but for anyone else w/this question, you have to install Python 3 first, and then install pylivemaker from pip (you don't need to download the github repo unless you want to make changes to the pylm code).

https://pylivemaker.readthedocs.io/en/latest/installation.html

[SwitchStation3]

  On 5/26/2022 at 4:27 PM, loam said:

I'm releasing a NekoNyanDatTool update (previously known as AokanaDatTool, but renamed because several games now use this engine).

It can now (un)pack Parquet (Mobile/Unity) and Senren Banka Switch/Unity files. Both games use the "aokana_switch" format; I've added aliases for those for ease of use, "parquet_mobile" and "senren_switch" respectively.

The reason it couldn't before was because the tool tried to decode the file names as ASCII, but Parquet Mobile and Senren Banka Switch have UTF-8 paths (due to japanese filenames). This has been fixed.

Binary: https://www.mediafire.com/file/0cw55ccjp2c77cn/NekoNyanDatTool.exe/file

Source: https://www.mediafire.com/file/3cdhkgijn56t4tn/NekoNyanDatTool_26052022.7z/file

Man, the bs5 is so clean compared to whatever mess this psb shit is, fucking M2 Cheeseware... I guess I'll be busy trying to port scenes to the Switch. Also I wonder why they had to bundle in e-mote for the relatively basic subset Senren uses...

Expand  

Hey i'm currently doing the Senran Banka Switch version as well, had any luck on it since? Thank You very much for updating it to make it work with Switch version's .dat files!

 

P.S. why can't they just fucking use .xp3 like the PC version? tryin be different 'n shit, makes things more compilated then it should or unless these companies are fully aware of the modding scene...

 

Edit: your tool doesn't work at all, how the hell do i use it dude?!

Edited November 20, 2022 by SwitchStation3

[Ohnoalex]

Is the link for ExtractData correct?? Because it just sends me to a site about running and the specific page has a 404 error;;

[jurnamo]

Hello, I am trying to extract the sprites and music from "Touhou Imperishable Night for Busy Person" by KasugaSoft. It has a data.xp3 file and a ksgdec.tpm. I have tried using Crass and it either extracts a warning text or just outputs "Crass Runtime Error.Access Violence.All Done.", I also tried using the command line, but that also gets stuck after extracting the warning text. I have also tried GARBro, arc-unpacker, xp3viewer, and Extractdata. The times I have managed to "extract" the files, everything is encrypted and will not open. 

Any help with this would be greatly appreciated!

Edited February 8, 2023 by jurnamo

[jurnamo]

  On 12/14/2022 at 8:12 AM, Ohnoalex said:

Is the link for ExtractData correct?? Because it just sends me to a site about running and the specific page has a 404 error;;

Expand  

https://app.box.com/s/cm8ypc3o5r

This link worked for me. 

[shonglet]

Has anyone successfully extracted all of the dialogue/scripts from Sakura no Uta? I mean, obviously someone has, since there are patches. I can't seem to get it to work myself, though. I've done lots of googling and tried Garbro, BGIKit, arc unpacker, VNTranslationTools. Hell, I even tried stepping through VNTranslationTools with a debugger and it seemed that the disassembler was failing.

Does anyone know the steps to get the untranslated scripts extracted or does someone have them on hand? If it entices you, I plan to use my limited GPT4 queries to slowly experiment with using it to translate the VN (I know this won't be anywhere near perfect given all the wordplay, but I'm just intrigued to try it).

Edited March 31, 2023 by shonglet

[loam]

  On 5/26/2022 at 4:27 PM, loam said:

I'm releasing a NekoNyanDatTool update (previously known as AokanaDatTool, but renamed because several games now use this engine).

It can now (un)pack Parquet (Mobile/Unity) and Senren Banka Switch/Unity files. Both games use the "aokana_switch" format; I've added aliases for those for ease of use, "parquet_mobile" and "senren_switch" respectively.

The reason it couldn't before was because the tool tried to decode the file names as ASCII, but Parquet Mobile and Senren Banka Switch have UTF-8 paths (due to japanese filenames). This has been fixed.

Binary: https://www.mediafire.com/file/0cw55ccjp2c77cn/NekoNyanDatTool.exe/file

Source: https://www.mediafire.com/file/3cdhkgijn56t4tn/NekoNyanDatTool_26052022.7z/file

Man, the bs5 is so clean compared to whatever mess this psb shit is, fucking M2 Cheeseware... I guess I'll be busy trying to port scenes to the Switch. Also I wonder why they had to bundle in e-mote for the relatively basic subset Senren uses...

Expand  

Hi there. I just want to give a heads up regarding support of the Steam release of Clover Day's Plus:

It's the same engine as previous NekoNyan ports.
The encryption mechanism is the same as the one in Aokana/Senren Banka Switch, Parquet Android.
You'll be able to unpack the .dat files by passing "portable" to the format argument of the tool, for example:

NekoNyanDatTool.exe --input "C:\Install Folder\Clover Days Plus_Data\StreamingAssets\adult.dat" --output outputFolder --unpack --format portable

(Beginning of rant)
Once again the assets are needlessly encrypted, killing any compression opportunities for the download size and making Steam waste its time trying to compress what's there to the benefit of absolutely no one and detriment of everyone. But at least it's the same stupid system so you guys can use keep using the same tool.

Also using il2cpp for the very needed maximum performance on a glorified image viewer with overlay text, where all of the code that had a chance of putting a load on the CPU (namely Emote) would have already been on native libraries anyways, with the usual Unity bloat that will make even the biggest CPU caches scream and run in fear, which no native compiler will ever fix. I hope next time they exclude the Augmented Reality, Web Cam, and Android modules from the Windows build of a Visual Novel (and this is assuming that Unity provides that capability, you'd assume considering how widely used the engine is... I'd be slightly surprised and disappointed if that's not a thing EDIT: okay it is a feature: https://docs.unity3d.com/Manual/upm-ui-disable.html).

I pray that they don't change the "encryption" format, not because it'll be hard, it'll just be even more bothersome because of native code shenanigans, which is just a waste of everyone's time.

Edited August 19, 2023 by loam

[shiny5478]

nevermind

Edited October 5, 2023 by shiny5478
solved issue

[yellowrock]

  On 2/28/2022 at 10:40 PM, Anonym271 said:

Yes, I once wrote a tool that could repack SystemC FPK files and I also implemented a compression algorithm for it. Can you tell me what game you are working on? Because I believe there are at least two versions of the file format and my program only works with one of them.

However, you could also just try it out: https://github.com/Anonym271/betterfpk. You can find an EXE in the releases section or of course clone the repository and modify it to your desires.

It's a command line tool, use it like this:

Extraction:
betterfpk.exe -e <my-archive.fpk> [output-directory]
  
Repacking:
betterfpk.exe -p <my-folder> [output-file.fpk]

It's not really fast and the compression results also aren't that great, but it should do the job I guess.

Expand  

Hi, I tried using this tool and it unpacked fine but if I try repacking it without changes it does not produce a file with the same size as the original, and the game does not run with the repacked file.

This is the game if you want to take a look: https://vndb.org/v6623#main

[Anonym271]

  On 2/2/2024 at 6:21 PM, yellowrock said:

Hi, I tried using this tool and it unpacked fine but if I try repacking it without changes it does not produce a file with the same size as the original, and the game does not run with the repacked file.

This is the game if you want to take a look: https://vndb.org/v6623#main

Expand  

I fixed some bugs about two weeks ago but forgot to publish them as a new release. I just updated it, could you check out the new version and see if it works?

The file size won't be the same as the original one since my compression algorithm is not as good as the developers'. I just tested the new version and repacked some CGs and it worked for me. If you still have issues, tell me and I will look deeper into the problem with your specific game.

[yellowrock]

  On 2/2/2024 at 7:03 PM, Anonym271 said:

I fixed some bugs about two weeks ago but forgot to publish them as a new release. I just updated it, could you check out the new version and see if it works?

The file size won't be the same as the original one since my compression algorithm is not as good as the developers'. I just tested the new version and repacked some CGs and it worked for me. If you still have issues, tell me and I will look deeper into the problem with your specific game.

Expand  

It works now, thanks for the quick fix. 

[Anonym271]

  On 2/2/2024 at 7:40 PM, yellowrock said:

It works now, thanks for the quick fix. 

Expand  

I am glad to hear that, happy modding!

[Valdiz]

Could someone help me, I'm trying to edit the text of the english release of STARLESS.

Extracting the data is simple enough, but altering the text is proving difficult, textractor++ messes with the coding/arrangement so I hopped over to arc_conv and the subprogram text_conv to convert the .mjo files into .txt files, but when I try to revert them back to .mjo files they end up completely empty.

I followed the steps listed here, but they just don't seem to work...

https://forums.fuwanovel.net/topic/4887-data-extraction-thread/page/17/#comment-346655

https://github.com/3licie/majiro-tools/blob/master/txt2mjo_howto.txt

For example this """""text_conv majiro staed001b_ero_scene_ed03.mjo staed001b_ero_scene_ed03.mjo < staed001b_ero_scene_ed03.txt"""""" is in the cmd prompt, with the files and text_conv in the same folder, but it just creates an .mjo file with nothing in it.

 

Edit: Also if there's a way to ADD new lines to an .mjo file I'd greatly appreciate being told how.

Edited June 1, 2024 by Valdiz
forgot something