Cudos to Stephen Saunders for this build and to reeFridge for finding the issue.
This fixes a number of multiplayer issues. Create Private Match with clients now works reliably. The missing functionality of the online game browser and leaderboards hasn't been fixed.
Changelog:
* Fixes regression Multiplayer: Accessing memory after it has been freed #846 caused by an earlier, but broken memory leak fix on my part. The fix is almost identical to PR Free idLobby memory inside destructor #847
* Fixes a critical issue with mis-reading network snapshot data on the client side, which caused no end of problems with mis-rendering, slowdowns, unstable connections, etc. The client was ignoring the entity network-synced flag which tells it whether it should read entity class-specific data. It was trying to read it all the time and sometimes coming up with null data which caused tons of problems with rendering and physics calculations (e.g. operations on NaN numbers). Simple fix was to respect the entity's network-synced flag on the client side but it makes all the difference.
* Fixes an incorrect assert on multiplayer VoiceChat shutdown
* Allows r_useScissor and r_useParallelAdd cvars to be changed in multiplayer mode for use in bake* operations on multiplayer maps
* Fixed a couple of uninitialized variables that showed up in valgrind when in multiplayer mode
* Added Amstrad CPC 6128 Retro rendering mode
## .plan - January 03, 2024
This is a preview build of the new Retro 8-bit/16-bit/PSX rendering modes.
The new rendering modes can be set in the menu options but it's controlled mainly r_renderMode.
The values are 0 = Doom, 1 = Commodore 64, 2 = Commodore 64 Highres, 3 = Sega Genesis, 4 = Sega Genesis Highres, 5 = Sony PSX
The Commodore 64 mode regulates all colors down to the original 16 color palette.
The Sega mode mimics 9 bit color HW which means 3 bit per color channel resulting in a total of 512 colors.
The PSX mode only turns off linear filtering for the textures and applies a screen space dithering effect.
All retro rendering modes try to mimic the 320x240 resolution but it is extended to 16:9 so it is 480 x 270.
Highres modes only apply a higher resolution dithering on the pixelated output.
The PSX mode has no additional artifacts yet like wiggling vertices or textures.
There are also 2 new CRT filters that are drawn on top of everything else (even the console) for more arcade vibes.
Changelog:
* Fixed scissor clipping issues of regular surfaces like light flares #651
* Duplicating lights with Ctrl+D works now
* Merged script interpreter improvements from Dhewm3, especially that fixes https://github.com/dhewm/dhewm3/issues/303
* Doubled MAX_GLOBALS for the Runners 2.6 mod
* Crash fix between level switching and loading of new textures for D3HDP and other mods
* Fixed many small memory leaks (thanks to Steve Saunders)
* Reduced console spam and got rid of the depth-stencil is read-only warnings
* Added image_pixelLook to disable texture filtering on most textures regardless of the render mode
* Changed devtools.cfg so you can easily switch between the new render modes with F7 and F8
This is just a bugfix build that addresses several crashes and minor changes.
## Quick Installation Guide for Beginners
1. Make a new DoomBFG folder
2. Copy base/ from your Steam Doom 3 BFG folder into DoomBFG
3. Download the RBDOOM-3-BFG 1.3.0 full package from the [RBDOOM-3-BFG ModDB page](https://www.moddb.com/mods/rbdoom-3-bfg) and extract it over DoomBFG.
4. Do the same with the 1.5.1 patch.
If you already have a version of RBDOOM-3-BFG then you can just unpack this patch over it.
OpenGL is dead and won't be further developed. OpenGL was holding back the development of this source port for quite some time and all the HLSL -> GLSL translation code with different outputs for Windows, Mesa and Vulkan GLSL was a time eater.
I was excited when Vulkan was announced but it wasn't fun at all to work with because it is way too verbose and difficult to work with even after 17 years of OpenGL experience. It was even annoying to just render anything offscreen like shadow maps. Right now, Vulkan offers many extensions to make those tasks simpler but if you are using modern Vulkan then it is already so different that people demand to make a new Vulkan 2 API.
Vulkan became immediatly the extension hell OpenGL was and all I wanted was a nice and simple API like OpenGL ES 3 but with proper error feedback and more speed.
I was looking for several years for a good abstraction layer of modern graphics APIs like DX12 and Vulkan. I considered The-Forge, Dilligent Engine, Tiny Renderers (Google) or BGFX for reducing the maintencance costs of the RBDOOM-3-BFG renderer.
Then I stumbled over an excellent article by Alexey Panteleev about [Writing Portable Rendering Code with NVRHI](https://developer.nvidia.com/blog/writing-portable-rendering-code-with-nvrhi/).
NVRHI (NVIDIA Rendering Hardware Interface) is a library that implements a common abstraction layer over multiple graphics APIs (GAPIs): Direct3D 11, Direct3D 12, and Vulkan 1.2. It works on Windows (x64 and ARM64) and Linux (x64 and ARM64) and macOS through MoltenVK. It is not tied to Nvidia GPUs but also works with AMD and Intel drivers.
I suggested it in Discord and that we should go into that direction if we move the renderer to an abstraction library. Stephen Pridham took the initiative and wrote the initial port to NVRHI. Porting a game like Doom 3 BFG to another render API is not trivial.
It also changed my plans for 2022 and ruined many weekends by fixing renderer related bugs.
However I want to continue to work on this project in my spare time for the next 10 years so the effort was worth it. In the end it turned out that NVRHI is as easy to use as OpenGL with the Donut samples provided.
There is definitely some lack of documentation but now you have a former AAA game rendered with NVRHI as an example.
RBDOOM-3-BFG has a similar renderer complexity to older Call of Duty titles or the Source engine.
Stephen Pridham did the major porting effort and was a big help when addressing certain difficult issues.
Stephen Saunders helped out with many Vulkan related rendering bugs. He also got it working with MoltenVk on macOS and improved support for AMD Vulkan users by integrating AMD's Vulkan Memory Allocator (VMA). He also contributed many other smaller bugfixes and improvements. Big thanks for both guys for all the help!
Also thanks to Alexey Panteleev at Nvidia who helped us through Discord and by pointing us into the right directions.
NVRHI will be also the gate opener for cross platform hardware accelerated Raytracing in the future so it is overall a big win for this source port.
There are also some immediate advantages for RBDOOM-3-BFG besides the smaller renderer backend.
This is actually one of the most demananding scenes in Doom 3 with many shadow casting point lights. DX12 is now almost twice as fast as OpenGL. Vulkan is still lacking behind but DX12 is the primary API for Windows users which are the vast majority.
MSAA/SMAA have been replaced with a Temporal Anti Aliasing solution by Nvidia. This not only fixes geometric aliasing at the edges but also shader based aliasing within the geometries like extreme specular highlights caused by the new PBR shaders.
The SSAO implementation has been updated with the latest code from Nvidia's Donut samples which outperforms the old implementation and also adds better stability on different view angles.
The old SSAO in 1.4.0 also lacked filtering and looked very noisy. The TAA would help but now it's really fixed and better.
The importer is not designed to work with all glTF 2 models you can find on the internet but it serves as a solid import facility from Blender. It can read ASCII .gltf and binary .glb models. It is recommend to use .glb for faster load times.
Models imported as .glb will be cached as .bglb models in the generated folder like all other model formats.
Cool about the current implementation is that animations are converted straight to the .bMD5Anim binary format of the Doom 3 BFG edition. This makes it a very tight integration into the existing architecture.
The basic idea about the glTF 2 `loader` is to make it easier getting models into the engine without any third party plugins in Blender. Another big disadvantage of .md5mesh exporters is that the format simply lacks support for normals.
So all normalmaps baked in Blender or Substance Painter using the Mikktspace algorithm would look wrong.
glTF 2 fixes this problem and it not only has support for normals but also tangents and those have to be calculated the Mikkelsen way so that's a perfect fit.
I also want to go back and keep things simple so I don't want to import and configure a model like in Unity or Unreal. I just want to load models and use them and the maximum should be to specify which frame plays a foot sound or which bone needs to be used to attach an item to it.
However it might be the case that someone exports a model from Blender and in a team environment someone else needs an option to load it differently. E.g. the front in Blender is -Y and forward in Doom is +X. So we need to rotate a model by 90 degrees around Z axis. All these things did the Maya .mb -> .md5mesh converter plugin for Doom 3 and we allow similar options in model declarations.
There is too much to explain here and this needs to be covered in a tutorial series along with a new Blender addon for editing the entities. But this is quite cool. You can now make maps without TrenchBroom or a Radiant based editor. Theoretically it is possible to use any 3D modelling package as a level editor now.
Like in modern Quake sourceports users can now just throw their favorite soundtrack into base/music/ like base/music/aqm/*.ogg or into a modfolder/music/ path.
The game will play automatically it in the background as a looping track until the end of the map.
Each map will pick a different track but mappers can also define a "music" track in the worldspawn entity.
You can also fine tune the volume of each track if you write a sound shader for your files.
There is an example in mod_alternativequakemusic/sound/_rbdoom_aqm_tracks.sndshd for the Alternative Quake Music sound track.
## Optick Profiler
The super lightweight C++ profiler Optick has been integrated. You need to recompile the engine with the CMake option -DOPTICK=ON but then you can run the profiler along with RBDOOM-3-BFG and make detailed captures about what is going on.
Stephen Saunders contributed support for borderlesss fullscreen mode which can also be set by commandline with r_fullscreen -2. It's enabled automatically if you switch from windowed mode to fullscreen by using Alt + Enter. You can’t tell the difference visually. However, stick to borderless mode over fullscreen mode if you use multiple monitors.
You can access your displays and launch various apps while your game or work stays on the main display.
This has the advantage that you can run RBDOOM-3-BFG simultanously along with TrenchBroomBFG, Blender, Krita or any other gamedev weapon of choice and it makes it easier to to switch between the applications.
But, there is a downside to borderless mode, depending on your device.
Most operating systems like Windows, for example, will continue dedicating resources to all background processes.
This means you might get less performance and experience input lag.
If you need to squeeze some extra FPS, go back to real fullscreen mode to see if it makes a difference.
The custom TrenchBroom build has been renamed to TrenchBroomBFG. This avoids conflicts with the original program like config files that are saved in the user directory. It also ignores all the other unsupported game profiles.
The new version has been updated to TrenchBroom 2023.1. It also supports the glTF 2 formats through the Assimp library but in this version the material loader has been tweaked to work directly with Doom 3 .mtr materials.
The icon size option also affects the entity inspector now which helps with high resolutions.
The engine command exportFGD was changed so it generates more variations of FGD files:
If you switch to one of those with the additional autogenerated model entities then you can drag n drop meshes into your map without the models zoomap.
## Summarized Changelog
[RENDERING]
* Renderer uses DX12 & Vulkan instead of OpenGL
* Shaders are not compiled at runtime anymore. They are compiled in advance by CMake using the DXC shader compiler and distributed in binary form under base/renderprogs2/dxil/*.bin
* All shaders have been rewritten to proper HLSL
* MSAA/SMAA have been replaced with a temporal anti aliasing (TAA) solution by Nvidia. This not only fixes geometric aliasing but also shader based aliasing like extreme specular highlights by the PBR shaders.
* Tone mapping has been replaced using new shaders by Nvidia
* Removed stencil shadow volumes with additional fixes by icecoldduke
* Integrate the AMD's VMA allocator for images and buffer objects
* Shadow mapping uses a fat shadowmap atlas instead of switching between shadowmap buffers and the HDR render target for each light. This improves the speed of the offscreen shadowmap generation.
* Blend shadowmaps smoothly in for small distant lights
* Skip shadowmaps for small distant lights
* Added menu option to choose between DX12 and Vulkan
* Detect displays that only support 144, 165 and 240 Hz
* Show picked GPU with com_showFPS > 1
* Went back to 15 bits for VERTCACHE_FRAME_MASK to avoid weird model distortions in a single frame every 68 seconds
* Reverted com_smp back to CVAR_BOOL
* Removed obsolete cvar r_useHDR
* Removed obsolete cvar r_useSeamlessCubeMap
* Removed unused cvar r_useHalfLambert
* Show shadow atlas stats with com_showFPS 3
* Enable DX12 / Vulkan debug layers with r_useValidationLayers 2
[DMAP]
* Dmap: always write a .cm file, especially when overwriting from a mod dir
* Support the Valve 220 texture projection in MapPolygonMesh::ConvertFromBrush()
This version improves support for mapping with TrenchBroom. Until now you needed to extract and copy the vanilla Doom 3 models and textures over to the base/ folder to see the content in the TrenchBroom entity browser and texture viewer.
Owning the original game next to the BFG edition is not necessary anymore.
This version comes with a couple of new RBDOOM-3-BFG console commands that lets you export particular parts of the .resources files to the base/_tb/ folder.
You need to call exportImagesToTrenchBroom once and you are good to go to start mapping with the TrenchBroom level editor.
TrenchBroom comes with several more Doom 3 specific changes. After loading a map TrenchBroom generates unique entity names and also fixes missing or bad "model" keys for brush based entitites.
Also creating new entities like light will automatically receive names like light_2.
This patch also contains a couple of func_group related bugfixes. func_group works now with brush based entities, point entities and just regular brushes.
* Added RBDoom console command convertMapToValve220 `<map>`. You can also type `exec convert_maps_to_valve220.cfg` to convert all Doom 3 .map files into the TrenchBroom friendly format. Converted maps are saved with the _valve220.map suffix.
* Added RBDoom console command exportModelsToTrenchBroom which saves all .base|.blwo|.bmd5mesh models to _tb/*.obj proxy files. This commands also generates helper entities for TrenchBroom so all mapobject/models are also available in the Entity Inspector using the DOOM-3-models.fgd.
* Added RBDoom console command makeZooMapForModels which makes a Source engine style zoo map with mapobject/models like .blwo, .base et cetera and saves it to maps/zoomaps/zoo_models.map. This helps mappers to get a good overview of the trememdous amount of custom models available in Doom 3 BFG by sorting them into categories and arranging them in 3D. It also filters models so that only modular models are picked that can be reused in new maps.
The main goal of this 1.3.0 release is enabling modders the ability to make new content using up to date Material & Lighting standards. Adding PBR is a requirement to make the new content look the same in RBDOOM-3-BFG as in Blender 2.9x with Cycles or Eevee and Substance Designer. PBR became the standard material authoring since 2014. Many texture packs for Doom 3 like the Wulfen & Monoxead packs were made before and are heavily outdated. With this release modders can work with modern tools and expect that their content looks as expected.
Implementing Physically Based Rendering (PBR) in Doom 3 is a challenge and comes with a few compromises because the Doom 3 content was designed to work with the hardware constraints in 2004 and that even meant to run on a Geforce 3.
The light rigs aren't made for PBR but it is possible to achieve good PBR lighting even with the old content by tweaking the light formulars with a few good magic constants. However I also want to support the modding scene to allow them to create brand new PBR materials made with Substance Designer/Painter or other modern tools so multiple rendering paths have been implemented.
PBR allows artists to create textures that are based on real world measured color values and they look more or less the same in any renderer that follows the PBR guidelines and formulars.
Doom 3 BFG is a big game. Doom 3, Resurrection of Evil and Lost Missions sum up to 47 big single player levels with an average of ~60 - 110 BSP portal areas or let's call them rooms / floors. Each room can have up to 50 shadow casting lights and most of them are point lights.
I needed a good automatic solution that fixes the pitch black areas without destroying the original look and feel of the game.
I also needed to add environment probes for each room so PBR materials can actually reflect the environment.
So RBDOOM-3-BFG comes with 2 systems to achieve this and both are automatic approaches so everything can be achieved in a reasonable amount of time.
The first system are environment probes which are placed into the center of the rooms. They can also be manually tweaked by adding env_probe entities in the maps. They use L4 spherical harmonics for diffuse reflections and GGX convolved mip maps for specular reflections.
The second system refines this by using a light grid for each room which provides a sort of a localized/improved version of the surrounding light for each corner of the room.
RBDOOM-3-BFG 1.3.0 brings back the Quake 3 light grid but this time the grid points feature spherical harmonics encoded as octahedrons and it can be evaluated per pixel. This means it can be used on any geometry and serves as an irradiance volume.
Unlike Quake 3 this isn't radiosity which is limited to diffuse only reflections. The diffuse reflectivity is built using all kinds of incoming light: diffuse, specular and emissive (sky, light emitting GUIs, VFX).
This will generate a ***.lightgrid*** file next to your .map file and it will also store a light grid atlas for each BSP area under ***env/maps/<path/to/your/map/>***
Quake 3 had one light grid that streched over the entire map and distributed lighting every 64 x 64 x 128 units by default.
If the maps were too big then q3map2 made the default grid size broader like 80 x 80 x 144, 96 x 96 x 160 and so on until the maximum number of light grid points was reached.
The Quake 3 approach wouldn't work with Doom 3 because the maps are too big and it would result in up to 800k probes for some maps or the grid density would very coarse.
This way we can maintain a good grid density and avoid wasting storage because of light grid points that are in empty space.
> But what is an Irradiance Volume or Light Grid exactly?
It tells each model or lit pixel how the indirect diffuse lighting is coming from any direction. The Probulator page by Yuri O'Donnell has some good examples:
Left: The captured view using a panorama layout. Right: The Diffuse lighting using Level 4 Spherical Harmonics aka Irradiance.
Now think of this for each of the grid points in RBDOOM-3-BFG. If a model is placed between those probes the lighting will be interpolated by the nearest 8 grid points similar like in Quake 3.
Quake 3 only stored the dominant light direction, the average light color of that direction and an ambient color term for each grid point.
In RBDOOM-3-BFG you basically can have the diffuse lighting information for **any** world space direction instead.
Environment probes supplement the light grids. While light grids provide diffuse lighting information the signal isn't good enough to provide plausible specular light reflections. This is where environment probes are needed.
If a level designer doesn't put any env_probe entities into a map then they are automatically distributed through the map using the BSP area bounds and placed in the center of them.
Environment probes can be computed after loading the map and by typing:
```
bakeEnvironmentProbes
```
This will generate EXR files in ***env/maps/<path/to/your/map/>***.
The environment probes use an octahedron encoding and the specular mipmaps are convolved using the Split Sum Approximation by Brian Karris [Epic 2013]. Convolving the environment probes is done on the CPU using the id Tech 5 multi threading code so it will consume all your available cores.
For artists this basically means if you increase the roughness in your material then you increase the mip map level of the environment probe it samples light from and it gets blurier.
If you haven't downloaded the additional baked light data from the [RBDOOM-3-BFG ModDB Page](https://www.moddb.com/mods/rbdoom-3-bfg) and just run RBDOOM-3-BFG.exe with the required DLLs (or you built it yourself) it will use an internal fallback.
RBDOOM-3-BFG.exe has one prebaked environment probe that is compiled into the executable.
It's the light data from the Mars City 1 lobby in the screenshot above. Using this data for the entire game is inacurrate but a better compromise than using a fixed global light direction and some sort of Rim lighting hack like in version 1.2.0.
The default irradiance / radiance data gives the entire game a warmer look and it fits for being on Mars all the time.
Left: No extra ambient pass. Ambient is pure black like in original Doom 3. Right: Extra ambient pass with r_forceAmbient 0.5 using local environment probe for irradiance and radiance lighting.
It's usually rendered with Blinn-Phong specular with a fixed specular exponent.
Specularmaps are more or less Gloss maps.
In RBDOOM-3-BFG it uses the PBR GGX Cook-Torrence formular and you can vary the roughness along the material like in other modern engines and you usually define a texture with the _rmao suffix.
The engine will also look for _rmao.[png/tga] overrides for old materials and if it finds those it will render them using a better PBR path. Old school specularmaps also go through a GGX pipeline but the roughness is estimated from the glossmap.
The Ambient Occlusion will be mixed with the Screen Space Ambient Occlusion and will only affect indirect lighting contributed by the environment probes.
## Filmic Post Processing
TL;DR If you enable it then you play DOOM 3 BFG with the optics of a Zack Snyder movie.
This release adds chromatic abberation and filmic dithering using Blue Noise.
The effect is heavy and is usually aimed in Film production to mix real camera footage with CG generated content.
Dithering demonstration: left side is quantized to 3 bits for each color channel. Right side is also only 3 bits but dithered with chromatic Blue Noise. The interesting fact about the dithering here is shown in the upper debug bands.
The first top band is the original signal. The second shows just 8 blocks and if you dither the those blocks with Blue Noise then it is close to the original signal which is surprising.
***This is still very much Work in Progress and not supported by the official TrenchBroom.***
Mapping for Doom 3 using TrenchBroom requires an extended unofficial build that is bundled with the official RBDOOM-3-BFG 7z package.
You can find the customized version under tools/trenchbroom/.
Doom 3 also requires some extensions in order to work with TrenchBroom. I usually develop these things for RBDOOM-3-BFG but I created a seperated brach for vanilla Doom 3 so everybody can adopt TrenchBroom and the new Doom 3 (Valve) .map support.
The goal of the TrenchBroom support is to make it easier to create new maps. It doesn't allow to create bezier patches at the moment so you won't be able to edit existing Doom 3 maps.
You can only save maps to the Doom 3 Valve format but you can copy paste from the vanilla Doom 3 .map format into the Doom 3 (Valve)configuration and reset your texture alignment as you want.
TrenchBroom doesn't support brush primitives like in D3Radiant or DarkRadiant and if you are familiar with TrenchBroom then you know that the preferred .map format is some kind of (Valve) .map format for your game.
The Quake 1/2/3 communities already adopted the Valve .map format in the BSP compilers and I did the same with dmap in RBDOOM-3-BFG.
Here is an overview of the changes made to TrenchBroom:
* Doom 3 .map parser with brushDef3, patchDef2, patchDef3 primitives
* Doom 3 Valve .map configuration
* Quake 3 .shader parser adopted to support .mtr
* .mtr support includes support for Doom 3 diffuse stages and the lookup for them is like in idMaterial::GetEditorImage()
* New Doom 3 OBJ parser. My TB Interop branch automatically creates OBJ files to work with TB and it also allows seamless interop with Blender 2.8x and 2.9x with the need of additional model formats for func_static entities (like misc_model for Quake 3)
* It has no support for BFG .resource files and .bimage files. BFG only shipped for precompressed textures and no .tga files so people who want to mod for BFG have to copy the vanilla D3 base/textures/* and base/models/* to D3BFG/base/
* Many entities work differently in Doom 3 if they have an origin. Brush work in D3 is usually stored in entity space and not world space. This is a major issue and not solved. I couldn't figure out how to parse the origin first and then translate the brushes accordingly.
* Doom 3's primary model formats are LWO and ASE. LWO and .md5mesh model support is missing.
* Some ASE models can't be loaded and materials are usually all wrong if loaded
### Some Scenes of Mars City 1 loaded into TrenchBroom
* Changed light interaction shaders to use PBR GGX Cook-Torrance specular contribution. The material roughness is estimated by the old school Doom 3 glossmap if no PBR texture is provided
* Improved r_hdrDebug which shows F-stops like in Filament
* Turned off r_hdrAutomaticExposure by default because it caused too much flickering and needs further work. The new default values work well with all Doom 3 content
* SSAO only affects the ambient pass and not the direct lighting and the ambient occlusion affects the specular indirect contribution depending on the roughness of a material. See moving Frostbite to PBR (Lagarde2014)
* Added HACK to look for PBR reflection maps with the suffix _rmao if a specular map was specified and ends with _s.tga. This allows to override the materials with PBR textures without touching the material .mtr files.
* idMapFile and dmap were changed to support the Valve 220 .map format to aid mapping with TrenchBroom
* Added exportFGD [models] console command which exports all def/*.def entityDef declarations to base/exported/_tb/ as Forge Game Data files. TrenchBroom has native support to read those files https://developer.valvesoftware.com/wiki/FGD.
Using the models argument will also export all needed models by entity declarations to base/_tb/ as Wavefront OBJ files.
* To make it easier getting static models from Blender/Maya/3D Studio Max into TrenchBroom and the engine Wavefront OBJ model support has been ported from IcedTech
* Improved Screen Space Ambient Occlusion performance by enhancing the quality with Blue Noise and skipping the expensive extra bilateral filtering pass
* Updated idRenderLog to support RenderDoc and Nvidia's Nsight and only issue OpenGL or Vulkan debug commands if the debug extensions are detected. Reference: https://devblogs.nvidia.com/best-practices-gpu-performance-events/
* Allow _extra_ents.map files next to the original .map files so new entities can be added to existing maps or old entities can be tweaked with new values
