As VirtualCast itself heavily uses GPU power, using another application that utilizes GPU simultaneously may cause VirtualCast to fail.
Change the encoding of the streaming software to software encode
(To be specific, don't use NVENC instead use x264)
It could be due to insufficient memory. We've seen a case where turning on Windows' virtual memory on and setting it properly solved the problem.