NVIDIA’s new SDK offers AI-powered greenscreen, augmented reality and style filters, and will be in action at TwitchCon this weekend.
NVIDIA’s new SDK offers AI-powered greenscreen, augmented reality and style filters, and will be in action at TwitchCon this weekend.
TwitchCon 2019 – Amazon’s annual convention celebrating streaming and their dominant creator platform – is held this weekend in San Diego, and what better event to showcase new streaming technologies and platform capabilities. The popularity of Twitch streaming – both videogames and ‘real life’ content – has been a massive growth market over the last few years, and it continues to go from strength to strength as broadcast and content innovations are developed by pioneers in the industry.
As the biggest discrete performance GPU manufacturer and supporter of independent content creators, NVIDIA are implicitly already a major stakeholder in streaming’s success. Their huge software development team continue to roll out GPU hardware accelerated features that render higher quality broadcast videostreams, improve the viewer experience with better frame rates, and reduce the barrier to entry for streaming as a hobby and profession. GeForce Experience already offers baseline game streaming functionality with no 3rd party apps required, and now they’re finally stretching the legs of cutting graphics processors.
Unveiled today and set to be showcased for the first time at TwitchCon 2019 is the RTX Broadcast Engine, an SDK which leverages the advanced AI capabilities of GeForce RTX hardware. Tools developed with this SDK will offer features without the need for additional expensive and purpose-built hardware, instead utilising RTX-powered deep learning and inferencing through plugins to popular broadcast software supported by Twitch as well as Youtube, Mixer and other platforms.
Included in the new SDK are:
RTX Greenscreen, delivering real-time background removal of a webcam feed so only your face and body show up on the livestream. The RTX Greenscreen AI model understands which part of an image is human and which is background, granting the benefits of a greenscreen without needing to buy one.
RTX AR, which can detect faces, track facial features such as eyes and mouth, and even model the surface of a face. With this data the toolkit enables real-time augmented reality effects such as dynamic face overlays using a standard web camera.
RTX Style Filters, which use an AI technique called style transfer to transform the look and feel of a webcam feed based on the style of another image.
RTX AR, which can detect faces, track facial features such as eyes and mouth, and even model the surface of a face. With this data the toolkit enables real-time augmented reality effects such as dynamic face overlays using a standard web camera.
RTX Style Filters, which use an AI technique called style transfer to transform the look and feel of a webcam feed based on the style of another image.
NVIDIA, in partnership with OBS, will be demonstrating RTX Greenscreen integrated into their broadcasting software. The technology should be available through OBS in the coming months, and other apps leveraging aspects of the SDK are being developed for the popular Streamlabs platform. OBS competitor XSplit also have their own plans for RTX Broadcast Engine.
By implication the RTX Broadcast Engine will be a technology suite exclusive to GeForce RTX hardware. Its install base of supported hardware will be limited at first, but over time will become much more ubiquitous as RTX is rolled out to lower price points. It’s also an important additional use case for the unique aspects of RTX hardware which are currently only utilised to any great degree in ray-tracing and DLSS upscaling.
Developers can learn more about the RTX Broadcast Engine and apply for early access at developer.nvidia.com/broadcastengine, and NVIDIA representatives will be on hand at the OBS TwitchCon booth, booth 1823, showing off RTX Greenscreen in OBS, new RTX Studio laptops and upcoming RTX games.
SOURCE: NVIDIA Blog
