obs segmentation faults when loading streamFX (Linux, Arm64) #470

StreamFX respects CMake's CMAKE_BUILD_TYPE setting (or in case of multi-config generators, whatever the generator deems necessary). You can try disabling each feature one by one to see which one breaks your build.

(Partially) Solved: Seg fault is caused by a bug in ubuntu's Gcc-8/G++-8 compiler installation with C++ 'filesystem' package when Gcc-9/G++-9 is also present on the system, see https://bugs.launchpad.net/ubuntu/+source/gcc-8/+bug/1824721 https://stackoverflow.com/questions/56615841/passing-stdfilesystempath-to-a-function-segfaults

I switched the compiler to Gcc-9/G++-9 and the seg fault vanished.

Remark: The streamfx/CMakeLists.txt on line 1403 checks for GCC-8 and includes library stdc++fs . However that does not prevent the issue. Probably because obs itself does not use 'filesystem', so on the dynamic load of streamFX the wrong library gets loaded instead of libc++fs. Also this issue was supposed to be fixed (see first link) by G++ taking care of linking with stdc++fs to avoid this issue. This does not seem to work in this situation. I also experimented with adding -lstdc++fs at various location after the object files - but no dice. So in total I currently see no way how to compile and link obs with streamfx with Gcc-8/G++-8 without deeper analysis.

As StreamFX requires a fully compatible C++17 compiler, GCC-8 is out of the question by default, yes. There may be a way to get it to work with std::experimental::filesystem, but testing felt pointless as I do not support such old operating systems or tools anyway (and there's no reason to not upgrade either).

Edit: The building page incorrectly referred to gcc-8. I've never actually built with gcc-8, it should have said gcc-9.

I fully agree. The only reason why I was using gcc-8 is - as I pointed out in system description - that nvidia still provides only cuda 10.2 for their jetson family of machines, and cuda 10.2 is tied to gcc-8 to compile, it won't compile with gcc-9. Currently I'm not trying to enable the cuda support for streamFX, so that is not an issue.

"CUDA support" in StreamFX is purely for dynamic linking. It requires no actual CUDA to be present, only the driver libraries.

I fully agree. The only reason why I was using gcc-8 is - as I pointed out in system description - that nvidia still provides only cuda 10.2 for their jetson family of machines, and cuda 10.2 is tied to gcc-8 to compile, it won't compile with gcc-9. Currently I'm not trying to enable the cuda support for streamFX, so that is not an issue.

Hi, I'm somehow missing an implementation of ffmpeg::hwapi (base.hpp) for Linux for nvenc; I see d3d11 for Windows, but as I'm on Linux, ffmpeg::instance::_hwapi remains 0 for nvenc and streamFX switches from hardware to software encoding for it in obs. What am I missing here?

OBS Studio does not implement texture sharing for OpenGL, so there's no point in adding a texture sharing implementation for it.

So when using the nvenc encoder in obs/streamFX, in file encoder-ffmpeg.cpp:122, _hwapi==0 and the runtime error "Failed to create acceleration context." is thrown, and the encoding will then use the sw-encoding path in obs?

As explained above I'm currently trying to integrate the nvidia nvmpi hardware encoder that is available in ffmpeg/libavcodec on the nvidia jetson-nano into obs. For this I've taken your nvenc integration as a starting point. It actually throws the run time error in line 122 and continues as a sw-encoder (is_hw=0). So that would be correct behaviour, and nvenc would also do it? Thx for any hint.

Yes, what you've read is correct. If you throw an exception while it's trying to initialize as a Zero-Copy/Hardware encoder, it will catch that, and then fall back to software encoding. If you then throw an exception while it's trying to initialize N-Copy/Software encoder, it will hard fail and report that back to OBS.

You can actually entirely disable the attempt to initialize as a Zero-Copy encoder by returning false from handler::is_hardware_encoder(...)

So in Linux, nvenc is actually not working as a zero-Copy encoder in obs? How much does that degrade performance? I'm asking because I get nvmpi to actually run in streamFX/obs, but in sw-encoding mode and that is so slow that it is of no use. So I had assumed that nvenc was integrated as an hardware encoder/Zero-Copy encoder.

Encoding speed wise, it's the same. It's just that you have an added transfer of the buffer, which has a non-zero 3D/Copy usage cost.

Ok, making progress. But there is a performance killer somewhere in obs/streamFX:

1) I have verified that "ffmpeg -i /dev/video0 -c:v h264_nvmpi out.mkv" works great with low CPU. So the nvmpi encoder is reasonably well integrated into ffmpeg/libavcodec. 2) When recording the webcam in obs/streamFX using the nvenc wrapper modified for nvmpi I have verified it is actually using the hardware encoder of the GPU. So that works! 3) But the GPU is mostly idle, because the data transfer within obs to libavcodec and back is terribly slow, CPU usage is 90+% of 4 CPU cores. So probably have to dig into the whole data processing chain in obs to libavcodec.

SO: Any idea where I should look first? Any SIMD or x86/x64 optimized code in there which might behave unexpectedly on ARM64?

StreamFX pushes the responsibility for GPU->CPU and CPU->GPU traffic to OBS Studio and FFmpeg, so your best bet is probably within either of those. Everything else is contained within encoder-ffmpeg.cpp

Ok, I'm zooming in on the issue. Good news first: When I run obs from the console x-terminal, its fully functional and high performing based on nvmpi hardware encoder via streamFX / ffmpeg. So I have an arm64 nvidia jetson with nvmpi encoder in obs-studio. This is a simple reuse of the streamFX nvenc integration, and I will document it in due course.

BUT the bad news: If I run obs-studio from a remote x-terminal, it has high cpu load, little fps, nvmpi underutilized. Same differing behaviour if I replace nvmpi with the standard sw encoder x264, and also with the standard obs-studio from ubuntu 20.04. So it has nothing to do with StreamFX and nothing with nvmpi, but with the different terminals. It looks like obs relies on the openGL capabilities of the x-terminal is executed from, instead of the capabilities of the machine it is actually running on. Hopefully you can also shed some light on this, see my question below.

Why this assumption: Obs loads different OpenGL support at different terminals:

1) on the console, the X-terminal exposes OpenGL with full nvidia GPU-Support, 2) while the remote X-terminal client just offers the standard MESA llvmpipe software OpenGL support - which is more than powerfull enough to operate the obs-studio GUI perfectly well. If obs also uses this particular OpenGL library for its internal frame handling to/from the encoder that might explain the performance breakdown on the remote x-terminal.

Q: Am I right with this assumption, or what else comes to mind that could explain this differing behaviour between console and remote x-terminal? Are there options in obs to use separate libraries for GUI-display and for its internal processing?

I haven't done a lot of in-depth research on the Unix/Linux side of things, but that conclusion sounds correct. OBS should rely on whatever OpenGL driver the underlying X-system provides, which may happen to be a software one. As far as I'm aware, there are no options in OBS itself to switch that - it would be up to the X-system provider.

Your are right; I've found multiple indication for this, the most clear one https://obsproject.com/forum/threads/obs-on-non-gui-linux-server-multiple-instances.95580/ . Up to now I had thought that OBS would consist of two components: a rendering+encoding kernel (similar to ffmpeg or gstreamer) and a GUI around that for settings and preview. The kernel might then use OpenGL or whatever for rendering using whatever hardware support is available on the machine, but obviously not being tied to the display that the GUI is running on (which might even switch off preview). But it seems that OBS, coming from the Windows world, has not been designed with that differentiation in mind, so display capabilities = machine capabilities. Or maybe simply OBS does not find the "right" openGL provider, just sticking to the display and not searching on. So one would have to trick the kernel of obs into believing it is running on the console terminal while the Qt-based GUI stays unaltered, or bring the hardware acceleration to the remote terminal, using something like www.virtualgl.org . I will give that a shot.

As I said above, I managed to add the nvmpi encoder to streamFX, just by starting from nvenc files and doing minor adaptions, and I now have a working OBS solution with nvmpi hardware encoder for the task at hand (also working remotely, see 1+2 below).

As this was a one time exercise and is way off my key focus areas I won't maintain this.

So my question: Are you interested to add nvidia nvmpi support to your streamFX code base? If yes what would be the best way to hand over the nvmpi code to you? (its about 60 KB in 6 files, plus some include lines added in encoder-ffmpeg.cpp and CMakeLists.txt)

PS.1: I took the features from the nvmpi_enc.c in the ffmpeg/libavcodec build that I use; the features list can be seen from line 544 in here. As you can see from the issues list this ffmpeg integration for jetson is work in progress.

PS.2:

1) Encoding core and GUI separation of OBS I managed to trick OBS into using the OpenGL GPU resources even when running over a remote xrdp screen, just by modding OBS to use display :0 for its internal rendering independent from the remote display where the OBS GUI and the preview are displayed. However it seems that OBS mingles internal rendering and preview more closely than I could resolve with reasonable effort, so I resorted to 2): 2) OBS and VirtualGL I found that OBS runs reasonably well on nvidia GPU and nvmpi encoding over a remote xrdp session using VirtualGL (www.virtualgl.org), see my comment on xrdp

Are you interested to add nvidia nvmpi support to your streamFX code base? If yes what would be the best way to hand over the nvmpi code to you? (its about 60 KB in 6 files, plus some include lines added in encoder-ffmpeg.cpp and CMakeLists.txt)

Do you have a fork of StreamFX with the additions included? And are you willing to continuously maintain the nvmpi encoder? If not, I will have to refuse.

Unfortunately I'm unable to maintain this, as I said above. I've just made it work for me and the task at hand. So, for the benefit of all who want to use OBS with nvidia nvmpi e.g. on the nvidia jetson family I will just open up a repository https://github.com/Costor/nvmpi-obs-streamFX in my github account and put the files there for download, together with some explanation and due references to streamFX and obs.

Unfortunately I'm unable to maintain this, as I said above. I've just made it work for me and the task at hand. So, for the benefit of all who want to use OBS with nvidia nvmpi e.g. on the nvidia jetson family I will just open up a repository nvmpi-streamFX-obs in my github account and put the files there for download, together with some explanation and due references to streamFX and obs.