Looks like the way I was doing it was wrong from the start. I found a link to a good tutorial in this post, but now there is a new problem, I can't find a way to replicate the "No Data" layers in g00 files:
I thought using transparent dummy images would work, although the layers are all in the right sorting order, looks like without "No Data" layers the game automatically adds empty space to the top and to the left, pushing all textures to the right. Are there any commands in G00PackMax for changing a layer's attribute to No Data?
EDIT: After some fiddling with an hex editor, it turned out that changing the attributes was easier than expected. First it's to note that the input PSD file was just a normal PNG created from merging the unpacked texture files with ImageMagick and exporting the PNG to PSD with GIMP.
The sorting order doesn't matter as long as the images are keeping the numerical name, it can be a 1x100 or 2x50 collage, as long as it's specified in G00packMax's command line (the txt file), it makes no difference.
The g00 output file and the original both have most of the informations about layers stored right in the header, the very first bytes. The new g00 will look a bit different but it's easy to understand where last byte of the header ends in both files. Overwriting the new header with the one from the original g00 file will fix the attributes making it exactly like the original.
Yet... the problem ingame doesn't change, there are other bytes that need editing somewhere else in the file, the more likely way to fix this is to just find the transparent images' bytes and replace them with 00s but I haven't tried yet, I'm still trying to figure out where the image data actually starts. Since the output file has a very different structure compared to the original g00 file, a normal comparison of the bytes won't be enough.