RTX Video Enhancement support #11390

germn · 2023-03-01T06:42:25Z

There was an RTX Video Enhancement release recently.
https://nvidia.custhelp.com/app/answers/detail/a_id/5448

Basically, it allows real-time AI video upscaling on recent RTX video cards.
It would be nice to have this feature available in mpv as well.

There's already a release for MPC with support of this:
https://github.com/emoose/VideoRenderer/releases/tag/rtx-1.0

kasper93 · 2023-03-05T20:56:18Z

There is already vf_d3d11vpp.c so it is actually trivial to extend it to select correct post processing extension. I personally don't have hardware to work with and implementing it blindly is never fun, so let's wait for another brave soul to add this.

JoseHidalgo · 2023-03-06T14:56:01Z

Just wanted to support this idea. No doubt that a MPV hero will take on this mission. :) The new feature is supposed to work not only on 4xxx cards but on 3xxx cards also.

Caution, there may be some issues to tackle : memory leaks on some HEVC 10bit videos, or the enhancements not working on vertical videos. See here:
https://www.reddit.com/r/nvidia/comments/11e7e2h/comment/jacujw9/?utm_source=reddit&utm_medium=web2x&context=3
emoose/VideoRenderer#1 (comment)

askasys-code · 2023-03-09T09:43:47Z

I'm totally interessed to have this feature in mpv.
I tried RTX VSR on MPC-BE and almost completely beat many upscalers I use with mpv, like FSRCNNX. It is absolutely worth it.

cloud11665 · 2023-03-24T10:34:15Z

I can take a look at this after getting access to a 40 series GPU

Myaanee · 2023-03-24T10:49:29Z

I can take a look at this after getting access to a 40 series GPU

If you need a tester @ me, I have a 4080

NekuSoul · 2023-03-24T11:08:24Z

I can take a look at this after getting access to a 40 series GPU

As a side-note to avoid possible misconceptions: RTX Video is also available for the RTX 30xx series.

cloud11665 · 2023-03-24T11:12:38Z

I currently have a 2080, and I will have access to a 4070 in ~2-3 weeks

askasys-code · 2023-04-13T17:39:01Z

Tested on vlc-3.0.19 and it works. Now only mpv remain, I really hope it will be supported soon. Maybe with a lua script? A toggle option is mandatory in my opinion, because both MPC-HC and VLC have VSR always ON and it's annoying.

360hzz · 2023-05-08T11:56:00Z

yes we are waiting for MPV my RTX 3080

saha1914 · 2023-05-28T16:13:17Z

there is already a VSR for VapourSynth https://github.com/HolyWu/vs-basicvsrpp only if someone would have build it

This doesn't seem to have anything to do with the NVIDIA VSR based on d3d11vpp.

really sorry ooooppps

cloud11665 · 2023-06-05T16:31:16Z

update: I now have access to a 40-series GPU, just need to set up my devenv on windows, and I can take a closer look.

deama · 2023-07-14T21:03:16Z

update: I now have access to a 40-series GPU, just need to set up my devenv on windows, and I can take a closer look.

Any luck?

radozd · 2023-10-19T14:56:27Z

Aaaaand?

kasper93 · 2023-10-19T15:01:11Z

Aaaaand?

Aaaaand, I can implement that, just need a GPU. We can discuss shipping/payment details in email. You can find my email address in my profile. (my nick on gmail)

Holdo · 2023-10-24T17:02:17Z

@kasper93 RTX VSR 1.5 now works on RTX 2000 series as well, don't need 4000 series.

deama · 2023-11-23T21:03:00Z

Aaaaand?

Aaaaand, I can implement that, just need a GPU. We can discuss shipping/payment details in email. You can find my email address in my profile. (my nick on gmail)

So I guess this was a joke or were you trying to scam someone?

kasper93 · 2023-11-23T22:44:39Z

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

deama · 2023-11-26T10:04:16Z

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

I think it looks pretty good.

kenpuluma · 2023-11-28T05:40:52Z

So I guess this was a joke or were you trying to scam someone?

Scam of what? It has been months and looks like no one capable of adding this feature to mpv cares to do so. Like I said, I don't have NVIDIA GPU, nor I intend to buy one, so I won't work on this feature, unless someone contribute the GPU. And since there is constant nagging in here to get it implemented I offered my help and stated my price. I was mostly joking, because I don't expect you guys understand how contracting works, but messages like Aaaaand? won't persuade anyone to work on this feature.

Look at my recent commit history in here if you call out scams... If anything I'm scammed of my free time working on this project.

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

tliron · 2023-11-28T05:56:21Z

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

People like you make me hate volunteering my work for open source.

Your sense of entitlement is absurd. Nobody owes you this feature. And also nobody owes you an in-depth explanation of the technical or personal or monetary or whatever challenges keep it from being implemented as fast as you might like.

The feature was asked for. The issue was not closed, so it's welcome. If you want it so much, develop it yourself and submit a PR. Otherwise, sit yourself down and wait patiently and politely and gratefully for the project's volunteers to do the work for you, free of charge, while you type "And?".

kenpuluma · 2023-11-28T06:23:57Z

If you consider two messages between Jun and Oct as "constant nagging" and it bothers you, and you require sponsorship to start your work then probably you shouldn't spend your time for an open source project. Complains like yours won't persuade anyone away from asking for new features, explanations like @Andarwinux's will.

People like you make me hate volunteering my work for open source.

Your sense of entitlement is absurd. Nobody owes you this feature. And also nobody owes you an in-depth explanation of the technical or personal or monetary or whatever challenges keep it from being implemented as fast as you might like.

The feature was asked for. The issue was not closed, so it's welcome. If you want it so much, develop it yourself and submit a PR. Otherwise, sit yourself down and wait patiently and politely and gratefully for the project's volunteers to do the work for you, free of charge, while you type "And?".

I'm not begging you for development, I'm not even the one who typed "And". I'm just searching online and happened to found this issue, and it looks to me that the discussion was normal until somebody asks for a payment (I guess it's a joke).

If you hate volunteering your work for open source then don't do it, like you said nobody is forcing you.

llyyr · 2023-11-28T06:42:20Z

Complains like yours won't persuade anyone away from asking for new features

Patches welcome

N-R-K · 2023-11-28T06:58:46Z

[...] probably you shouldn't spend your time for an open source project

This is not your place to state what others should or should not do in their own time.

the discussion was normal until somebody asks for a payment (I guess it's a joke)

Perhaps you're from some communist utopia but asking for payment for your work is nothing abnormal around the rest of the world.

And if you're so bothered by someone not wanting to spend hundreds of dollars to implement a feature for free for randoms on the internet, then go ahead and be the change you wish to see in the world. Buy yourself a compatible gpu and send the patches.

kenpuluma · 2023-11-28T07:29:41Z

I really don't want to populate the Github issue with meaningless, unrelated messages in a but if you insist ok:

This is not your place to state what others should or should not do in their own time.

Agreed. Github issue is the place for tracking work and collaboration on the project, not for making jokes and spreading hates and stress.

Perhaps you're from some communist utopia but asking for payment for your work is nothing abnormal around the rest of the world.

Don't try to change the subject. It's definitely normal for asking payments for anything you have spent time on, lots of people work on open source projects for a living.
But the discussion went pretty off the track and became completely irrelevant about its original topic: "RTX Video Enhancement support" now, people started defending themselves on scam or not scam, be thankful to open source contributors or not , beginning from that payment joke.

If this is how you want this repository looks like, then it's fine, leave people there to fight against each others.

ghost · 2023-11-28T07:56:09Z

@kenpuluma

Hey.
Basically I'm not going to implement it (the NVIDIA VSR)
I know... UGH!! I know... I'm sorry!!!!
I'm just not gonna implement it is all
HAHAHAHAHAHA

kasper93 · 2023-11-28T08:37:45Z

VSR has no status feedback, so there is no reliable way to confirm VSR is active.

How the driver handles the scaling is indeed not visible. But adding generic d3d11vpp scaling is good either way. Even if not VSR, it would still scale, deepening on vendor with various quality.

Also note that passing frames through video processor can/will also apply other adjustments that they have in pipeline, which in most cases is not wanted.

This feature was designed primarily for Chromium, which is planning to move to D3D12, so D3D11VPP VSR will certainly be dropped eventually, or even removed from drivers at some point. Adding VSR may not be worth it at this time.

I think you infer too much. d3d11 or d3d12 is only an API, there is no need to remove the scaler from d3d11 video processor to make it work for d3d12 video processor. Unless there would be needed API extension.

Intel actually also support VSR (Arc only) but like NVIDIA there is no status feedback and Intel's VSR has no gpu control panel switches or options and the effect is much weaker, you never know if VSR is active or not.

Similar situation, we just request the scaling. It is on driver control panel to expose proper feedback/control.

haasn · 2023-11-28T09:10:30Z

Don't listen to paid AMD shills and false flaggers trying to subvert this issue

kasper93 · 2024-01-24T00:42:11Z

@kasper93 RTX VSR 1.5 now works on RTX 2000 series as well, don't need 4000 series.

AMD has it's own upscaling now. You don't even need NVIDIA gpu to implement this. Just the latest AMD ones. (at least for the AMD, probably different guids and so on)

AMD Video Upscaling – Advanced video upscale algorithm to improve video playback image quality for AMD Radeon™ RX 7000 desktop series GPUs.

https://www.amd.com/en/support/kb/release-notes/rn-rad-win-24-1-1

...

Hrxn · 2024-01-24T01:57:31Z

I assume this'll be the default setting for Chrome/Edge somewhat soon?

BTW, the description blurb for this flag is:

Enable/Disable the use of MediaFoundation for non-protected content playback on supported systems. – Windows

Implying that MediaFoundation is already used for Widevine content etc.

wadixx · 2024-04-19T18:31:09Z

MPC-HC, Potplayer and VLC has already started using RTX VSR. Can we expect to see this in MPV soon?

billxc · 2024-04-29T10:30:08Z

Having study the code of MPC Render, and chromium. The magic code is quite easy

    constexpr GUID kNvidiaPPEInterfaceGUID = {
        0xd43ce1b3,
        0x1f4b,
        0x48ac,
        {0xba, 0xee, 0xc3, 0xc2, 0x53, 0x75, 0xe6, 0xf7}};
    constexpr UINT kStreamExtensionVersionV1 = 0x1;
    constexpr UINT kStreamExtensionMethodSuperResolution = 0x2;

    struct {
        UINT version;
        UINT method;
        UINT enable;
    } stream_extension_info = {kStreamExtensionVersionV1,
                                kStreamExtensionMethodSuperResolution,
                                1u};

    hr = ID3D11VideoContext_VideoProcessorSetStreamExtension(p->video_ctx,p->video_proc,&kNvidiaPPEInterfaceGUID,
        sizeof(stream_extension_info), &stream_extension_info);

I tried to add this code to d3d11vpp.c and enable the the filter by using vf set d3d11vpp=deint=no.
But nothing happened, no super res, even no crash.
I am new to MPV, will work to find out if the codes are included in the compile.

bcaefa6

emoose/VideoRenderer@a684e0a
chromium/chromium@46f000a#diff-993a15e7a79f90944c791b0c616715be012bf762dd8f7a18bc27e8a597fd18fb

wadixx · 2024-04-30T17:32:29Z

Having study the code of MPC Render, and chromium. The magic code is quite easy

    constexpr GUID kNvidiaPPEInterfaceGUID = {
        0xd43ce1b3,
        0x1f4b,
        0x48ac,
        {0xba, 0xee, 0xc3, 0xc2, 0x53, 0x75, 0xe6, 0xf7}};
    constexpr UINT kStreamExtensionVersionV1 = 0x1;
    constexpr UINT kStreamExtensionMethodSuperResolution = 0x2;

    struct {
        UINT version;
        UINT method;
        UINT enable;
    } stream_extension_info = {kStreamExtensionVersionV1,
                                kStreamExtensionMethodSuperResolution,
                                1u};

    hr = ID3D11VideoContext_VideoProcessorSetStreamExtension(p->video_ctx,p->video_proc,&kNvidiaPPEInterfaceGUID,
        sizeof(stream_extension_info), &stream_extension_info);

I tried to add this code to d3d11vpp.c and enable the the filter by using vf set d3d11vpp=deint=no. But nothing happened, no super res, even no crash. I am new to MPV, will work to find out if the codes are included in the compile.

bcaefa6

emoose/VideoRenderer@a684e0a chromium/chromium@46f000a#diff-993a15e7a79f90944c791b0c616715be012bf762dd8f7a18bc27e8a597fd18fb

Looking forward to hearing from you.

billxc · 2024-05-04T17:22:37Z

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter.
@kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

wadixx · 2024-05-04T18:05:49Z

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter. @kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

You are a hero. Hope @kasper93 can merge this into mpv.

philipl · 2024-05-04T22:52:32Z

Update the status: Now the RTX Super Res works, but with a little hard code, and NOT ready for merge.

If you are curious about the code, just refer to https://github.com/billxc/mpv.

This relies on the D3D11VPP filter and seems that it cannot be used together with vapoursynth filter. @kasper93 could you help confirm if it is expected, and is there any way to make both filters work together?

The vapoursynth filter is a CPU filter, so it's never going to work directly. You would need to do a GPU upload/download (depending on the order of operations) to the frames into the right place for the other filter. I expect ffmpeg has upload/download support for d3d11va so it ought to be possible, but you'd need to work out how to construct the correct filter chain yourself, and you may still find additional limitations getting in your way.

A mergeable implementation would require extending the d3d11vpp filter to have a scaling mode with proper specification of the output frame size/format/etc and then a way to select the scaling mode (of which RTX Video is just one of many).

wadixx · 2024-05-13T15:51:35Z

@billxc Any updates on this?

billxc · 2024-05-14T02:46:29Z

@billxc Any updates on this?

https://github.com/billxc/mpv/releases/tag/MPV-RTX-SuperRes-RC

Not quite ready for merge, but ready for test.

Note: the filter usage is not the final version, and may be changed in the future.

wadixx · 2024-05-14T16:30:20Z

@billxc Any updates on this?

https://github.com/billxc/mpv/releases/tag/MPV-RTX-SuperRes-RC

Not quite ready for merge, but ready for test.

Note: the filter usage is not the final version, and may be changed in the future.

I downloaded mpv.exe and run it with following config. Mpv loads filter and triggers super resolution on NCP however it breaks video. See screenshot below. In some video files it doesn't trigger super resolution at all. Tried a few things but couldn't find out why.

vf-add=d3d11sr
vf=d3d11sr=mode=nvidia:scale=1440p

philipl · 2024-05-14T20:10:51Z

@billxc First off, it's great that you're working on this, and thank you for looking at my original feedback. However, after taking an initial look at your tree, it looks like you have written it as a separate filter. For it to be mergeable, the functionality should be added to d3d11vpp. Were you just doing it as a separate filter as an initial development step? I can't tell.

Also, the filter's configuration interface should not hard code the "Super Resolution" concept - as I understand it, it's just a generic scaling function and vendors or anyone can write and plug in a scaling algorithm (I see you have added Intel as a second example), so it should just be a scale configuration. It should also support taking arbitrary width and height for the scaling output, as that is supported in the general case (does RTX Video only work for fixed resolutions?). Thanks!

Feel free to open a PR - it will be easier to discuss there.

kasper93 · 2024-05-14T22:18:00Z

Also to make it scalable for the future we could extend options of d3d11vpp with arbitrary GUID that user could provide. Would avoid having to recompile mpv for different variant of the same. I think AMD has some sort of super res too, which has to be enabled, because normally this interface just scales without fancy features.

srk24 · 2024-05-16T01:17:14Z

@billxc Any updates on this?

https://github.com/billxc/mpv/releases/tag/MPV-RTX-SuperRes-RC

Not quite ready for merge, but ready for test.

Note: the filter usage is not the final version, and may be changed in the future.

so nice, performance better than ravu. thx !!!!

billxc · 2024-05-16T05:44:52Z

@philipl @kasper93
I have created a PR, we can move the discussion there
#14155

Adds `--vf=d3d11vpp=scaling-mode` to control which extensions should be enabled. Fixes: mpv-player#11390 Co-authored-by: xc <[email protected]>

tmarkov · 2024-11-03T23:02:09Z

Does this only work for Direct3d (i.e. Windows)?

kasper93 · 2024-11-03T23:07:37Z

Does this only work for Direct3d (i.e. Windows)?

Yes, it works for d3d11, only on Windows.

Laesx · 2025-01-03T13:17:38Z

Does this only work for Direct3d (i.e. Windows)?

Yes, it works for d3d11, only on Windows.

Hello, I tried to enable this but I don't seem able to do it, this is what I tried in my mpv.conf:

vo=gpu vf-add=d3d11vpp vf=d3d11vpp=scaling-mode=nvidia:scale=2.0

also tried without the vf-add, without scale=2.0, with different vo, but I don't notice any change in quality, and the nvidia panel says RTX super resolution is inactive.

The logs however say "[d3d11vpp] NVIDIA RTX Super Resolution enabled"

Is there any option I should set I'm missing perhaps?

germn added the meta:feature-request label Mar 1, 2023

Andarwinux mentioned this issue Jan 25, 2024

support for rtx video hdr #13352

Closed

billxc mentioned this issue May 16, 2024

d3d11vpp: add GPU scale option for intel and nvidia GPUs #14155

Closed

kasper93 closed this as completed in 2848af5 Aug 20, 2024

This was referenced Jan 16, 2025

RTX Video not enabling Zaarrg/stremio-desktop-v5#9

Closed

Mpv rtx upscaling / super resolution not working #15700

Closed

RTX Video Enhancement support #11390

RTX Video Enhancement support #11390

Comments

germn commented Mar 1, 2023

kasper93 commented Mar 5, 2023 • edited Loading

JoseHidalgo commented Mar 6, 2023

askasys-code commented Mar 9, 2023

cloud11665 commented Mar 24, 2023

Myaanee commented Mar 24, 2023

NekuSoul commented Mar 24, 2023

cloud11665 commented Mar 24, 2023

askasys-code commented Apr 13, 2023 • edited Loading

360hzz commented May 8, 2023

saha1914 commented May 28, 2023

cloud11665 commented Jun 5, 2023

deama commented Jul 14, 2023

radozd commented Oct 19, 2023

kasper93 commented Oct 19, 2023 • edited Loading

Holdo commented Oct 24, 2023

deama commented Nov 23, 2023

kasper93 commented Nov 23, 2023 • edited Loading

deama commented Nov 26, 2023

kenpuluma commented Nov 28, 2023

tliron commented Nov 28, 2023

kenpuluma commented Nov 28, 2023

llyyr commented Nov 28, 2023

N-R-K commented Nov 28, 2023

kenpuluma commented Nov 28, 2023 • edited Loading

ghost commented Nov 28, 2023

kasper93 commented Nov 28, 2023 • edited Loading

haasn commented Nov 28, 2023 • edited Loading

kasper93 commented Jan 24, 2024

Hrxn commented Jan 24, 2024

wadixx commented Apr 19, 2024 • edited Loading

billxc commented Apr 29, 2024 • edited Loading

wadixx commented Apr 30, 2024

billxc commented May 4, 2024 • edited Loading

wadixx commented May 4, 2024

philipl commented May 4, 2024

wadixx commented May 13, 2024

billxc commented May 14, 2024

wadixx commented May 14, 2024

philipl commented May 14, 2024

kasper93 commented May 14, 2024

srk24 commented May 16, 2024

billxc commented May 16, 2024

tmarkov commented Nov 3, 2024

kasper93 commented Nov 3, 2024

Laesx commented Jan 3, 2025

kasper93 commented Mar 5, 2023 •

edited

Loading

askasys-code commented Apr 13, 2023 •

edited

Loading

kasper93 commented Oct 19, 2023 •

edited

Loading

kasper93 commented Nov 23, 2023 •

edited

Loading

kenpuluma commented Nov 28, 2023 •

edited

Loading

kasper93 commented Nov 28, 2023 •

edited

Loading

haasn commented Nov 28, 2023 •

edited

Loading

wadixx commented Apr 19, 2024 •

edited

Loading

billxc commented Apr 29, 2024 •

edited

Loading

billxc commented May 4, 2024 •

edited

Loading