Upscaling & Denoise filters causes browser sources to lose transparency #646

I'm unsure if I want to treat it as a bug so far, since these effects don't scale transparency as well as color, so the transparency channel is quite literally undefined. This is true for every single upscaling algorithm I know, none of them consider transparency in an image. So what would the expected result here be, since we have nothing to scale the transparency with? Just simply interpolate with linear sampling?

My understanding is that since the Nvidia upscaling technology upscales in specific intervals (1.5x, 2x, 2.5x, etc.) and StreamFX only upscales the source when it reaches those multiples.

You are correct that the underlying NVIDIA Super Resolution network only supports certain scale factors, to which StreamFX snaps. Any unsupported resolution will cause the network to fail initialization, which results in the out-of-bounds rendering.

I'm unsure if I want to treat it as a bug so far, since these effects don't scale transparency as well as color, so the transparency channel is quite literally undefined. This is true for every single upscaling algorithm I know, none of them consider transparency in an image. So what would the expected result here be, since we have nothing to scale the transparency with? Just simply interpolate with linear sampling?

Linear sampling could work, or the more complicated (and probably unnecessary way) would be to make it take the alpha channel, run it through a separate denoising instance, then apply it back as an alpha channel to the original source's output. I can only imagine this helps with something like pre-rendered video media sources like overlays and such, which are likely to have an alpha channel for transparency, and that transparency may be relatively complicated for the given source.

You are correct that the underlying NVIDIA Super Resolution network only supports certain scale factors, to which StreamFX snaps. Any unsupported resolution will cause the network to fail initialization, which results in the out-of-bounds rendering.

That makes sense. I'm not sure if there's a worthy tradeoff to making so the source is upscaled to the next available multiplying factor, then downscale it back to the defined percentage. IE: if the user specified a scaling factor of 140%, then the filter would upscale using the nearest scaling factor (which I believe would be 1.5x or 150%), then downscale the source back to fit into that 140% factor. With a given source at 1080p resolution, a 150% upscale would make it 2880x1620. Then filter can apply downscaling down to 140% of the resolution, which is 2688x1512.

The "how worth it is to spend resources upscaling and then downscaling again" thing is definitely an issue, although I presume in some scenarios it's worth it, while in others it's not.

That makes sense. I'm not sure if there's a worthy tradeoff to making so the source is upscaled to the next available multiplying factor, then downscale it back to the defined percentage.

The problem with that is that there is no defined list of resolutions explicitly supported, and the network fails if it's off by as little as 1px in any direction. Which is a bit difficult to detect without reloading the effect, and that takes a fraction of a second - so it would introduce a noticable hitch every time you move the slider.