Eliminate game training, add motion vectors for better quality

Eliminate game training, add motion vectors for better quality

While NVIDIA’s annual GPU technology conference has been widely recalled and most of NVIDIA’s announcements have been made for another day, apparently, the company still has an announcement up its sleeve this week. And an announcement related to the game, no less. NVIDIA is finally removing DLSS 2.0 technology this morning, which the company is deploying as a major update of its previous AI upscaling technology.

Responding to both competitive pressure and the realization of its technological limitations, the latest iteration of NVIDIA’s upscaling technology is a rather significant revision of the technique. While NVIDIA is still performing AI level upscaling, DLSS 2.0 is no longer a pure upscaler; NVIDIA is now essentially combining it with temporal anti-aliasing. The results, NVIDIA is promising, are both better image quality than DLSS 1.0, and faster integration within individual games by eliminating training per game.

As a quick update, Deep Learning Super Sampling (DLSS) was originally released around the launch of the Turing generation (GeForce RTX 20 series) in the fall of 2018. DLSS was the first major effort by NVIDIA to use their rapid programming experience. AI and AI hardware to apply technology to image quality in video games. With all their GeForce RTX cards shipped with a core tensioner, what better way to put them into use than to use them to improve image quality in games in a semi-abstract way? It was perhaps a bit the case of a hammer looking for a nail, but the fundamental idea was reasonable, especially because 4K monitors become cheaper and GeForce 2080 Tis not.

Unfortunately, DLSS 1.0 has never lived up to its promises. NVIDIA has taken a very image-centric approach to the process, relying on a vast training program that involved creating a different neural network for each game at each resolution, training the networks on what a game should look like by feeding them at very high resolution, 64 anti-aliasing images. In theory, the resulting networks should have been able to recognize how a more detailed world should function and consequently produce cleaner, sharper images.

Sometimes this worked well. More often the results have been mixed. NVIDIA mainly launched the technology as a way to reduce the cost of rendering higher resolutions, i.e. rendering a game at a lower resolution and then upscaling, with the aim of combining the native resolution of a game with the temporal antialiasing. The final results sometimes achieve or exceed this goal, and other times an image would still be soft and lacking in detail, revealing its low resolution origins. And in the meantime, it took a lot of work to add DLSS to a game: each game and each supported resolution required the training of another neural network. In the meantime, a simple top level filter + sharpness could offer a not insignificant increase in perceived image quality with only a fraction of the work and GPU usage.

Enter DLSS 2.0

While DLSS 1.0 was pure, in retrospect it was perhaps a little naive. As NVIDIA clearly states now, DLSS 1.0 was difficult to work with because it was based on the idea that video games are deterministic – that everything would behave in a predefined and predictable way. In reality the games are not deterministic and although AI characters always do the same thing every time, second order effects like particles and the like can be turned off by doing their thing. As a result, it was difficult to train DLSS 1.0 networks, which needed this determinism to improve, let alone apply them to games.

So for their second shot at AI upscaling, NVIDIA is taking a different turn. Instead of relying on individual neural networks per game, NVIDIA has created a single generic neural network from which they are optimizing hell. And to compensate for the lack of information that comes from gaming networks, the company is compensating for it by integrating the information on real-time motion vectors from the game itself, a fundamental aspect of time antialiasing (TAA) and similar techniques. The net result is that DLSS 2.0 behaves much more like a temporal upscaling solution, which makes it more stupid in some ways, but also smarter in others.

The single big change here is obviously the new generic neural network. Trying to remove the expensive game training and the many (many) problems presented by nondeterministic games during training, NVIDIA switched to a single generic network for all games. This new neural network is based on a fully synthetic training set rather than single games, which in turn is completely deterministic, allowing NVIDIA to train the new network extensively exactly in the way they need to iterate and improve over the course of time. generations. According to NVIDIA, this new network is also faster to run on the GPU, initially reducing the overhead of using the DLSS.

In addition to the elimination of training time per game and the hassle of developers about determinism, the other result of NVIDIA is that the generic network offers them more resolution scaling options. NVIDIA can now perform upscaling of frames up to 4x in resolution – from 1080p input to 4K – both by allowing the use of DLSS 2.0 with a wider range of input / output resolutions, and by allowing a more decisive use, in lack of a better term. DLSS 1.0, by contrast, generally aimed for 2x upscaling.

This new flexibility also means that NVIDIA now offers multiple modes of DLSS quality, negotiating internal rendering resolution (and therefore image quality) for greater performance. These modes are performance, balance and quality.

If not, the current network training process has not been completely changed. NVIDIA is still training against 16K images, with the aim of teaching the neural network as much as possible about quality. And this is still performed as neural networks also through the tensor cores, although I’m curious to see if the DLSS 2.0 pins work equally well with the tensor cores like 1.0.

The trick of DLSS 2.0, however, is that this still requires the integration of game developers and in a very different way. Since DLSS 2.0 relies on motion vectors to reproject the previous frame and better calculate the appearance of the output image, developers must now provide these vectors to DLSS. Since many developers are already performing some form of temporal AA in their games, this information is often available within the engine and simply needs to be exposed to the DLSS. Nonetheless, it means that DLSS 2.0 has yet to be integrated on a game-by-game basis, even though game training has disappeared. It’s not a pure end-of-chain post-processing solution like FXAA or that combines image sharpness with upscaling.

In addition, it should be noted that NVIDIA is still defining DLSS resolutions as before; that is, they are talking about the output resolution rather than the input resolution. Therefore, 1080p Quality mode, for example, would generally mean that the internal rendering resolution is half the output resolution or that 1280×720 is enlarged to 1920×1080. And Performance mode, they told me, would be a luxury 4x.

In the meantime, it is obvious that the topic of upscaling and image enhancements has been a hot topic since the introduction of the DLSS, as well as AMD’s most recent efforts to counter it with Radeon Image Sharpening. So NVIDIA is hitting the ground, so to speak, in promoting DLSS 2.0.

Indeed, “promotion” is perhaps the key word for today. While NVIDIA is only announcing DLSS 2.0 today and outlining how it works, the company has already shipped it to game developers for a while. Both Deliver Us the Moon and Wolfenstein: Youngblood are already available with DLSS 2.0. And now that NVIDIA is satisfied with the state of the art field-proven technology, it is moving on to distributing it to gamers and game developers as a whole, including integration into Unreal Engine 4.

Along with the above games, both Control and MechWarrior 5 are receiving DLSS 2.0 updates. The check in particular will be an interesting case, as it is the only game in this set that also had a DLSS 1.x implementation, which means it can be used as a check to judge differences in image quality. NVIDIA also goes so far to demonstrate some of the quality improvements.

As for performance, NVIDIA generally promises performance similar to that of DLSS 1.0. This means that the comparable quality mode might be a little slower than DLSS 1.0 in games like Control, but overall that quality mode and its mid-rendering resolution should provide significant speed gains compared to native resolution games . In the meantime, the resulting image quality should be better than what DLSS 1.0 could offer. NVIDIA is even advertising DLSS 2.0 as it offers better image quality than games with native resolution, although for the time being putting aside the subjective nature of image quality, it may not be an apple to apple comparison what post-processing effects the developers are using (e.g. replacing a very blurred TAA filter with DLSS 2.0).

Either way, DLSS 2.0 is now officially available today. Updates for Control and MechWarrior 5 will be shipped this week and if NVIDIA makes its way, more games will follow soon.