Obstacles to build on Linux / arm64 [solved, need permanent fix] but core dump on start of plugin #465

Hi, I successfully compiled and installed StreamFX (bundled with obs-studio) on Linux-arm64 (details below). Doing so I encountered and temporarily fixed three issues with the Linux build process that imho require a permanent fix (see below). However the plugin core dumps when loaded by obs, which might be a result of hidden Linux/arm64 compatibility issues of the obs<->streamfx plugin mechanisms. So my issues are

1) permantly fixing the build process for Linux / arm64 which currently seems borken 2) Making sure my fixes didn't introduce errors that cause the seg fault of streamfx 3) fixing the seg fault of streamfx

Remark: My ultimate goal is to use the hardware encoder h264_nvmpi which replaces h264_nvenc on the nvidia jetson and which I have already compiled into ffmpeg. I will probably need help on this but will start with this only once streamfx works at all.

Environment details

Jetson Nano 4GB RAM (ARM V8 64-Bit, Nvidia GPU), Ubuntu 20.04 LTS, Cuda 10.2, obs-studio 26.1.2 (build from source with ffmpeg 2.4.4 build from source, all running stable), Gcc/g++ 8.0 (must be 8 for compatbility with cuda 10.2), streamfx 0.10.0.0a1-gfbe35d15.

Overview of build obstacles for Linux and/or arm64 (details and my fixes see below):

1. Just an addition to the instructions: After adding the git submodule streamfx there should also be 'git submodule init && git submodule update' from the streamfx folder to update the submodules of streamfx
2. Disabling of nvidia arsdk for non-Windows environments does not work, so compile of arsdk files necessarily fails
3. invalid static_cast for os_dlsym(..) when loading from CUDA lib: discrepancy of gnu g++ compiler to MSC
4. Use of xmmintrin.h for SIMD instructions for non x86/x64 architectures

My analysis and temporary fixes

• Concerning 2) The CMakelists.txt for streamfx has a check for "Nvidia Face tracking" in Line 562. However that does not work - though cmake gives the warning "Face tracking Windows only: disabling..", the lines from 1148 on are still included by cmake. Moreover the lines 872ff include the nvidia-ar.* sources unconditionally if only the submodule arsdk is present. So I had to comment out all this lines to prevent arsdk files from being included in the build. Probably it is best to not install the whole submodule arsdk if nvidia face tracking is not wanted or not available for Linux.

• Concerning 3) streamfx/CMakeFiles/StreamFX.dir/source/nvidia/cuda/nvidia-cuda.cpp contains lines "NAME = static_cast<type>(os_dlsym(..))" to load symbols from dynamic CUDA libraries. os_dlsym() returns void*, which is cast to a function pointer. g++ throws an error on this casts, as ANSI C does not permit casts from object pointers to function pointers. MSC is probably more forgiving here. So the solution for g++ seems to either use a C-type cast (that is what the other c++ code in obs-studio does) or to use reinterpret_cast (what I did)

• Concerning 4) streamfx/source/obs/gs/gs-vertex.hpp unconditionally includes xmmintrin.h which is a x86 specific include for SIMD instructions and does not exist on other architectures. obs-studio has build in architecture independent support for SIMD instructions by using the sse2neon solution. This is already <included> via obs.h->obs-studio/libobs/util/sse-intrin.h ->simde/x86/sse2.h (Note that sse2.h is architecture aware and includes ARM64 support though for historical reasons it is placed in x86/ folder). So I put #if defined(MSC) .. #endif around the #include <xmmintrin.h> (probably the include is superfluous anyway as it also might be handled by sse2.h under Windows).

streamfx segmentation fault

With this patches, obs including streamfx build successfully only displaying marginal compiler warnings about unused variables etc. Also the install process was without warnings. Remember that obs ran ok without streamfx so I assume streamfx is causing the segmentation fault:

output from obs before segmentation fault

Attempted path: share/obs/obs-studio/locale/en-US.ini
Attempted path: /usr/share/obs/obs-studio/locale/en-US.ini
Attempted path: share/obs/obs-studio/locale.ini
Attempted path: /usr/share/obs/obs-studio/locale.ini
Attempted path: share/obs/obs-studio/themes/Dark.qss
Attempted path: /usr/share/obs/obs-studio/themes/Dark.qss
info: Physical Cores: 4, Logical Cores: 4
info: Physical Memory: 3956MB Total, 1113MB Free
info: Kernel Version: Linux 4.9.140-tegra
info: Distribution: "Ubuntu" "20.04"
info: Session Type: x11
info: Window System: X11.0, Vendor: The X.Org Foundation, Version: 1.20.9
info: Portable mode: false
Attempted path: share/obs/obs-studio/themes/Dark/no_sources.svg
Attempted path: /usr/share/obs/obs-studio/themes/Dark/no_sources.svg
QMetaObject::connectSlotsByName: No matching signal for on_advAudioProps_clicked()
QMetaObject::connectSlotsByName: No matching signal for on_advAudioProps_destroyed()
QMetaObject::connectSlotsByName: No matching signal for on_actionGridMode_triggered()
QMetaObject::connectSlotsByName: No matching signal for on_program_customContextMenuRequested(QPoint)
QMetaObject::connectSlotsByName: No matching signal for on_transitionRemove_clicked()
info: OBS 26.1.2-161-g36ab7b97c-modified (linux)
info: ---------------------------------
info: ---------------------------------
info: audio settings reset:
        samples per sec: 48000
        speakers:        2
info: ---------------------------------
info: Initializing OpenGL...
info: Loading up OpenGL on adapter Mesa/X.org llvmpipe (LLVM 11.0.0, 128 bits)
info: OpenGL loaded successfully, version 4.5 (Core Profile) Mesa 20.2.6, shading language 4.50
info: ---------------------------------
info: video settings reset:
        base resolution:   1280x1024
        output resolution: 1024x818
        downscale filter:  Bicubic
        fps:               30/1
        format:            NV12
        YUV mode:          709/Partial
info: NV12 texture support not available
info: Audio monitoring device:
        name: Default
        id: default
info: ---------------------------------
warning: Failed to load 'en-US' text for module: 'decklink-captions.so'
warning: Failed to load 'en-US' text for module: 'decklink-ouput-ui.so'
libDeckLinkAPI.so: cannot open shared object file: No such file or directory
warning: A DeckLink iterator could not be created.  The DeckLink drivers may not be installed
info: No blackmagic support
warning: v4l2loopback not installed, virtual camera disabled
info: [obs-browser]: Version 2.13.1
info: [obs-browser]: CEF Version 88.2.5+gcab058b+chromium-88.0.4324.150
info: FFMPEG VAAPI supported
info: [StreamFX] Loading Version 0.10.0.0a1-gfbe35d15
Segmentation fault (core dumped)
================ end of output ===========