yes, I think most of the loading time spent on get file data from disk.
ps: I'm a programmer.

I think this was mentioned earlier, but a Game-friendly compression would be cool too.
It would create compressed files when the game started and any time after that
It would just run a check to see if the file had been modified since the last time the
game was run, and if it was, it would create a new version of the compressed file.

It would load slow the first time, but every time after that it would move quickly for non-modders.

TLB, considering YOU have inside access to Data (or at least more sway than we do) maybe this would be a good item on the to-do list?
I understand that creating a new file architecture is no simple task, but it may help since the game wouldn't
have to read/interpret user-friendly code, and your RAM would be spared because of the smaller files.

Heck, if .ini code could be loaded as fast as a .lua file, then there would be no problem.

the res files will eventually loaded in the RAM,
pack method is only a "pre process" method for reducing loading time,
not change what happens next.

another experience of me, I wrote a program in 4M,
and use upx to compress it to 2M, reduced size and improved loading time...

I know this "packing trick" from a famous game programming book(forgot which one)

the time spend on parse ini file can be reduced by introduce "inic" file,
same as compile "xxx.py" into "xxx.pyc" in python.

In case this wasn't clear, the game isn't storing the INI files themselves, as far as I know. The game searches through the INI files, determines the values of all those variables, and stores the numbers/strings themselves. When it comes time to create an actor, it loads all the data from a preset full of data from the INI files. That said, putting it all into a custom file format would eliminate the need for human-friendly code parsing, and would increase load times a bit. On the other hand, I think this is contrary to what Data wants the game to do - the reason he made it human-readable code is because he wanted it to be open to modding, which is something that a binary file couldn't do.
Now, what you're suggesting is essentially storying the raw data in a cache file of sorts. While that is entirely possible, the question really comes down to whether it's worth it. Unless you've got a ridiculous number of mods installed, the load time isn't that considerable. As well, having a cache like that would greatly bloat the game's directory size. It would be one more thing to code, one more thing to work bugs out of, and at this point in the game's development, one more section of the engine to rewrite. Could it decrease load time? Yes. Is it going to happen? Probably not.

Halida, I'm having trouble understanding what you're trying to say here. No matter where in the process you compress the data, it isn't going to make it load faster. It's simply moving the loading time to another place in the program. At some point, the data needs to be decompressed, and when that happens, you've got load time. In a game like this, the only place that could occur at is either when the program first starts, or on demand. On demand would only slow the game when you first use a piece of data. So, yes, you've eliminated some of the loading time, but instead you're just peppering the loading time across the game. Considering we're talking about an engine that most computers have trouble running without the added stress, throwing decompression operations and storing that data while in the middle of running physics simulations is probably not going to end well.
Also, please use the edit button instead of posting twice in a row.

CC using streamed gzips would be a terrible idea, however there is some merit to the point that a modern CPU can do things a lot faster in RAM than on-disk, ie loading the compressed data from disk and manipulating (in this case, decompressing) it in RAM. It'd be interesting to do tests on this sort of thing with CC-sized files. I'm expecting that straight-from-disk reading would be faster in the long term considering the size of the files and the fact that they're decompressed entirely for the most part (however, RLE bitmaps load... O_o it's a conspiracy), but there would be a point where it would be more efficient to have the shorter disk -> RAM period and an additional decompression stage if you've got the speed of RAM and a grunty CPU. It could mean CC was even more limited to modern hardware though..

This is speculation though. In reality, with an ~55% ratio of compression (gotten using windows built-in .zip compression on zombies.rte and inspecting filesizes, this would be smaller for .rars with best compression and larger wordlists I suppose) you still need a 45% speed increase in RAM, but you can probably getsomewhere near that using zlib or similar.
It's theoretically feasible to try and squeeze that extra juice out of the hardware, and considering you're going to be putting just as much data in RAM in the long term anyway there's no reason not to try it, but I suppose we'd have to know how Dan puts .rtes into memory to be able to do anything relating to a benchmark... Maybe we could just try iterating all the data in a folder and dumping it to RAM sequentially (not linking anything) and then doing the same with a compressed file but unzipping it in-RAM. I might do something like this later, to see the difference in times on my age-old lappy. I might not, I've got study to do. Considering how good a procastinator I am, I might do it though.

EDIT forgot to mention th 4 posts that happened between me writing this and posting it <_<

The main problem I see isn't the decompression, that's a factor, especially on older hard ware but it's more the fact that you've still got to iterate all the ♥♥♥♥ .ini once you're done with loading it into ram :\ I think Dan might be doing that as he goes (considering how it pauses on some lines longer than others), which would make all of this redundant.

yes, programmer face trade offs. what to optimize depend on whether or not worth it.


maybe I'm not make myself clear enough,
here is a example:
a mod name xxx.rte,
loading lots of small files take lots of time, like 10s(without parse the files)

zip rte folder into a xxx.rte.zip file,
when game start, load this xxx.rte.zip file, take 1s(1/10 the size of rte folder)
unzip it, take 1s(because gamers have strong CPU, and this unzip process take place in RAM)
so total time is 2s.

edit:
I use 7zip to compress the rtes, takes 17s in my PC, ziped form 62m to 37m,
maybe because there are lots of ogg file..

You really do need to use realistic values here halida. 40% is just about the best you're going to get for a big mod in terms of compression. Also, I'm not sure how much faster the decompression will be. I'm not disagreeing that it might be faster, just that you need to use realistic numbers or not use numbers at all.

if mod contains only bmp, ini, lua, compress rate can reach 1/10(and even more), like : Undead.rte

That's a pretty big assumption. My computer has terrible RAM and such, and I play lots of video games. Saying that all gamers have strong CPU is wrong, and making the game even laggier for people without would be a terrible decision.

What I'm hearing is that if you decrease the number of files to load, then the disk seek time will be less and speed up loading.
That's perfectly doable without zipping, just put all of your .ini in one file. As well, you can shorten your code by doing as many CopyOfs as is useful to reduce the amount of text in a file. Sounds and sprites can't be put into one file each with offsets to separate, but that would be the next step.

thanks to your avatar, I read that entire thing in the professor's voice. Hats off to you.

Halida, you officially have no idea what you're talking about. Loading raw data will always, always, always be faster than decompressing it then loading it. This is basic computer science. You're trying to add 2 and 2 and get 1. It just doesn't work like that.

EDITL: actually, I lied. If you had a computer with a insanely slow hard disc bus speed or a really big, really fragmented disc and a massive cpu and an insane amount of ram, and a really well written decompressor, then yes, that might be true. But modern computers have a fast hard disc bus and if you watch, it's the ini files that are loading slowly.

Here's some simple proof. Time the amount of time it takes CC to load [Time A]. Now time the amount of time it takes to duplicate your CC folder and divide that by 2 [Time B]. Now subtract Time B from Time A [Time C].

Time B is the amount of time it takes the game to actually load the files from disk. Time C is the amount of time CC spends interpreting those files. Time C is much larger than time A. This is because most hard disks can copy about half a gig per second, meaning it takes about 1/3 of a second to load CC from disk, and 4 seconds [on my machine] to interpret the files.

A better suggestion would be for data to optimize the ini loader. It's bizarrely slow.

I think the .ini parsing is so slow cause the game is linking everything into classes at that point rather than later, but I'm not sure. It sure as hell doesn't take that long to simply iterate a file and put all the stuff it references into RAM.

Yeah, Data's doing something weird with .ini parsing, but it's fairly well known that literally everything .ini wise is all said and done by the time that loading screen finishes. Which, considering the filesize of some mods, is actually pretty good (though quite the RAM bloat).

I said it first, other people have said it since:
Compressing mods is not going to optimize the game in any way, and even if Data did it, the most likely CC related time effect is going to be the next build taking longer.

No thanks.

I get what you're saying now. However, this is a tradeoff that will inevitably leave some people out. Presently, the game takes longer to load for people with slower hard drives. With this method, the game will take longer to load for people with slow CPUs. You can't guarantee that everybody has a fast CPU. The fact is that you're just pushing the problem to a different part of the computer, and considering the amount of time it would take to implement a system like this into an engine that's already well-established, it really isn't worth it. It would be adding one more thing for Data to work into the engine, when really it isn't necessarily improving speed, but rather changing who experiences slow speeds.