NEWS

Mixed Reality for Film, TV and Event

Will the Green screen soon be a thing of the past? Video walls are everywhere, in film and in television. The resolution of LED walls and the performance of media servers has increased rapidly in recent years and it is now possible to display photorealistic 3D computer animations in real time.

Under the initiative of Frank Junghahn and his production company Youngrooster, which has been active in the TV, stage and event sector for over 20 years, a special kind of LED installation has been set up at TMT Studios in Munich. This is in cooperation with LEDitgo Videowall Germany GmbH and rtleaders.

More infos and videos at Facebook https://www.facebook.com/RealtimeWizzard/videos/330685894499709/

TMT Studio 1

Thomas Berz, one of the managing directors of TMT Film- und TV-Produktions-Service GmbH and his sign gaffer and cameraman commented: ‘TMT has been providing stage lighting and camera for film and TV for the last 28 years, 10 years with its own studios.

For the past year we have been able to offer a Stäubli RX 160 robot. Originally, we focused on high-speed recording and motion control. Through communication with interested customers and creative colleagues, we constantly discover new areas of application. We are therefore grateful to Frank Junghahn for his initiative and look forward to the exciting times ahead. We can imagine establishing a hotspot for these innovative technologies here in Munich.’

In order to test the various possibilities of the latest presentation and show technology, two 2x2m rXone LED walls from LEDitgo with a resolution of 1.9 mm were first placed on a small scale at an angle of 90 degrees and connected to the LED event floor sB6ef with 6.25 mm resolution.

Construction and calibration were done by the experienced LED technician Hendrik Wallbaum. The type, positioning, pixel pitch and scaling of the LED surfaces can of course be extended at will and are planned for the future on a large scale.

The live event programmer Frank Junghahn is responsible for the video content. The device of his choice is a high-performance media server on which the software Notch and D3 from Disguise are used. All content is rendered in real time.

TMT Studio 3, playground of the innovative technique

The aim is exploiting the possibilities of technology in order to offer new creative approaches to customers in the film, advertising and TV sectors.

Besides many other planned tests, Stephan Grune from rtleaders and Frank Junghahn from Youngrooster first set themselves the task of tackling one of the most innovative ideas by presenting a virtual studio, without having to spend time building and illuminating a perfect green screen cyclorama, still giving the impression of standing in a large studio.

‘The special advantages of using LED surfaces as backgrounds are obvious,’ Frank Junghahn reports from his own experience as a cameraman and director. ‘It is often necessary to be able to correctly assess camera angles and perspectives on the set or give the actors orientation. With the combination of LED technology and media servers, we can guarantee this reacting immediately to all conditions and wishes on set without having to render new image content.’

Frank Junghahn inside the Mixed Reality

Thomas Berz from TMT Studios adds: ‘The new technology gives us completely new opportunities to work on set. Now we can use the contents on the LED surfaces both as light sources and dramaturgically. Problems with reflections and refraction in the scene due to the green scattered light caused by the green box illumination are a thing of the past. What you see is what you get. Expensive post production can be avoided in many cases.’

Stäubli Roboter RX 160 and Stephan Grune from rtleaders

In order to guarantee the impression of the fusion of real camera recording and virtual reality in the so-called Mixed Reality, it is necessary to capture all data of the real camera. This includes primarily the cartesian position in space, the rotation data around all three axes and the values for zoom, focus and aperture.

In the first test, the developers of this special installation decided to use a Stäubli industrial robot weighing 250 kg and with a range of 1.6 m. Stephan Grune of rtleaders with years of experience as a robot planner, was responsible for this job.

Using RoboKam, self-developed software from rtleaders, it is possible to move the camera three-dimensionally in space and send all values as network stream to the media server.

Some will wonder why use a robot when there are camera motion tracking systems from MoSys, Stype or Ncam. The advantage of using a robot is its absolute precision and repeatability. For example, it is possible to take high speed pictures of food shots with the same camera movement over and over again, shooting multiple exposures for dramaturgical use in film or creative music videos.

The special feature of this LED setup is the detail and now for a little brain jogging. The challenge was to synchronize all cameras, virtual and real, with each other, to define scaling, to adjust all coordinate crosses and to find a zero point as reference.

RoboKam in action

D3 scene on the media server

The virtual studio was saved with the software Notch as a real time scene with a selectable camera and exported as a so-called Notch Block. This was then imported into Disguise’s D3 program on the media server. In the D3 scene the LED corner was created in the correct scale and orientation as in reality and the surfaces were assigned to the respective graphic outputs. The perfect impression of the virtual studio is only possible if the Notch scene is projected into the three-dimensional LED corner in the D3 scene from the view point of the D3 camera.

The D3 camera has to be synchronized with the real camera via the network stream of RoboKam. If focal length, field of view and all other data match, the camera on the robot should show exactly the image that is sent from D3 to the LED controller. If there is a moderator in the real LED corner, the perfect illusion should be created. Check?

Another feature of Notch worth to be pointed out is the possibility to make any content or desired values in the Notch scene externally influenceable, to expose them. Thus, it is very easy e.g. to adjust the number of particles, colours of lights or the positions of 3D elements interactively with the connected MIDI controller like the Ableton APC40MKII.

In this particular case the position and orientation of the virtual character was adjusted to stay next to the real moderator.

MIDI controller Ableton APC40MKII

As a next step, the specialists at TMT Studios plan to use Perception Neuron’s Motion Suit to breathe life into the virtual character. For example, you could have a dialogue between the virtual character and the presenter, place unusual interactions in music videos or enhance ballet performances with a holographic projection.

Motion Suit from Perception Neuron and the Kinect 2 in the foreground

project-polunin-satori-ph-roberto-ricci-02

‘We are also planning to integrate handheld cameras into our Mixed Reality with camera motion tracking systems from Mosys and Ncam in the near future and are also eagerly awaiting the new Kinect 3 from Microsoft. When we mount the new model on the real camera, we can read even more precise depth information and create more perfect illusions than ever before.’ Says Frank Junghahn.

It remains exciting.

Scene from the ballet Satori with Sergei Polunin in Prague 2019

More infos and videos at Facebook https://www.facebook.com/RealtimeWizzard/videos/330685894499709/