The r1q2 code prioritized pak files over all other files, e.g. as soon
as a pak file was requested no more file were added to the download
queue until it finished downloading. That way one could be sure that
assets included in the pak file weren't downloaded in parallel as single
files.
This is a better, bugfixed and more robust implementation of the same
logic. With this back in place we can switch back to parallel downloads
which gives a nice speedup on Windows. Maybe, just maybe some day
Microsoft will fix Windows crappy I/O...
Working with getter and setters was a good idea as long as we had one or
two quirks. Now we're at three with maybe more to come so it's easier to
use a struct to communicate quirks between the precacher and the HTTP
download code.
I'm not 100% sure if this is okay for q2pro, but at least in my simple
tests r1q2, q2pro and now yq2 generate the same URL. Nevertheless it's
somewhat inconssistent to search generic files at /moddir/... and the
filelist at /moddir.filelist
This closes issue #370.
The r1q2 URL generator was, like everything in this game, buggy. It took
cl.gamedir into account when generating the URLs, but overlooked that it
is only set when 'gamedir != BASEDIR'. So baseq2 assets ended up with
/maps/foo.bsp and mod assets with /mod/maps/foo.bsp. q2pro fixed that
to always include the gamedir...
Work around this by refactoring the HTTP -> UDP fallback logic to be
more generic: Count the number of iterations and depending on the
iteration set the gamedir to be used by the URL generator or force
UDP downloads.
Cleaning the download queue as soon as a file finished downloading leads
in combination with only one parallel download and multiple precacher
runs to an ugly problem: The generic filelist is requested several time
which can lead to cycles. Hack around this by rembering if we already
requested it and reset set reminder as soon as the precacher finished.
Yes, I'm feeling dirty.
If the server sends us an URL with trailing slash we're generating URIs
like http://example.com//maps/foo.bsp. While double // are perfectly
valid they might me rejected by some servers. So let's play save.
While at it replace the crappy Z_TagMalloc() with the standard malloc().
Z_TagMalloc() ist just a wrapper arround malloc(), there's no functional
change.
This looks easy, but is rather hacky... Downloading is implemented
through the precacher. The server sends an asset list, while loading
the map another one is generated. CL_RequestNextDownload() goes
through this list, in the order models / maps -> sounds -> images,
calls CL_CheckOrDownloadFile() for each file. CL_CheckOrDownloadFile()
checks if the file is already there, return true if it is and false
if not. If the return code is false CL_RequestNextDownload() itself
returns, it's called again by CL_ParseDownload() as soon as the just
queued file finished downloading. This way all missing files are
downloaded one after the other, when CL_RequestNextDownload() finally
reaches it's end (all files are there) it send 'begin' to the server,
thus putting the client into the game.
HTTP downloads are parallel, so CL_RequestNextDownload() cannot track
which files are there and which are missing. The work around for that is
to queue the file but have CL_CheckOrDownloadFile() return true. So
CL_RequestNextDownload() thinks the file is already there, continues
with the next one, until all missing files are queued. After that it
polls CL_PendingHTTPDownloads() and sends the 'begin' as soon as all
HTTP downloads are finished.
If a HTTP download fails we cannot just queue it as UDP download,
because the precacher things that the file is already there. And we
can't tell the precacher that it's not because the precacher tracks
files only by the number of downloaded files per asste type and not
their name. Just decreasing the number of downloaded files isn't
possible since the precacher may have progressed to the next asset
type.
So: On the HTTP side it's tracked if there was an error or not. After
CL_RequestNextDownload() has queued all files and waited for all HTTP
downloads to finish it checks the HTTP error status. If there was an
error the precacher state is reset and CL_RequestNextDownload() recurses
into itself to take another run. All files that couldn't be downloaded
are queued again, this time as UDP downloads.
If we're passing file handles to libcurl to write the data into, the
game may crash under Windows due to incompatible C runtimes between cURL
and quake2. This is even mentioned in the official cURL doku.
Since the moment I took a very first look at the download code I wasn't
a friend of parallel downloads. There're several reasons for that:
- Parallel downloading needs some ugly hacks. For example downloading a
pak file has a high chance to make asset downloads running in parallel
unnecessary.
- Parallel downloads are hard to debug.
- There's just no need for them. I've tested several connection, 1
GBit/s LAN, 50 MBit/s DSL, 6 MBit/s DSL, and there wasn't a
significant difference between 1, 4 or even 16 parallel downloads.
I'm leaving the parallel download code in place. I someone really wants
parallel downloads he can bump the MAX_HTTP_HANDLES define.
Until now download queues entries were created for each file, but only
removed in the unlikely event of changed download server. This leaked
about 4200 byte per file. Fix this by:
- Create a new function CL_RemoveFromQueue() that removes a queue entry
and call it everytime when a download either finished, failed or
aborted.
- Retire the 'download done' state, just the queue entries associated
with a download handle to NULL to communicate that the handle is
unused.
- Cleanup the full queue at shutdown.
Loading libcurl at runtime instead of linking it at compile time makes
things a lot easier and more reliable on Windows. On other platform
libcurl can be installed as optional dependency instead as an hard one.
