Back in the olden days when movies were shot in single-lens cameras, a 5-second scene could take hours to film. The very first motion picture in history shot in 1888 used images of moving objects printed onto paper, hence the term “motion picture”. Despite being just 2 seconds long at 12 frames per second, it was incredibly time-consuming to make at the time.
Less than two decades after the feat, filmmaker Edwin S. Porter discovered a new trick that fused two different images to create an imaginary location. The 1903 film The Great Train Robbery was the first to use a green screen to lay over a moving train clip over a window. This created the illusion that the train was leaving the station.
For decades, chroma key compositing was used by film producers to trick the human eye. By shooting scenes with a green screen, filmmakers can stitch them seamlessly with a virtual background. However, when LED screen technology came, LED blue screens took over the reins, beginning with the production of Star Wars Episode V in 1980.
Today, with the prevalence of fully immersive, high-definition cinemas, filmmakers are also stepping up their game by incorporating new technology that could make the background even more realistic: virtual production LED walls.
What is Virtual Production?
Virtual production technology is the use of LED display screens and motion tracking technology to create highly immersive content without the need to shoot on-site or use outdated green screens. Instead, LED walls are installed in the virtual production studio to display sceneries or production sets that would otherwise take days to build and involve a lot of personnel.
Think of a virtual studio as Extended Reality (XR). With the high-definition LED displays working with the camera, the director can recreate a real-life setting–either pre-filmed or animated–and film the actors in the studio. This bypasses chroma key compositing employed in traditional green and blue screens.
Difference Between AR, MR, VR, and XR
VR – Virtual Reality creates a virtual world where the subject is immersed. The surroundings are 100% make-believe, such as wearing a VR headset where the “player” is transported to a different world.
AR – Augmented Reality is the combination of real-world and virtual information. Integrates virtual elements into real-life scenarios, in real-time, such as superimposing objects or characters into the viewer’s view.
MR – Mixed Reality is the combination of the virtual and real-world elements, creating an entirely new environment.
XR – Extended Reality is the combination of VR, AR, and MR. It requires computer-human interaction to create a new environment.
How Does It Work?
While the chroma technique requires massive green backgrounds for filming, virtual production requires floor-to-ceiling, large LED walls to create the illusion that the characters are in a different location and not inside a studio. A virtual studio would typically have two LED video walls forming a right angle and floor LED tiles. This combination ensures the virtual worlds created inside the studio look true to life in front of the camera.
The XR controller utilizes a real time rendering engine to ensure the LED screens will display the correct content at the right time. Real-time rendering applies the adjustment on the spot and reduces post-production work and related costs.
The final component of a virtual studio is the camera equipped with a tracking device. It follows the subjects and sends the information to the rendering engine, allowing it to adjust the LED screen display according to the subject’s movements.
The Importance of LED Screens in Virtual Productions
The most important role of LED screens in virtual production is that the captured scene is often close to if not already the final cut. Unlike traditional green screen that requires filming the actors separately and superimposing them onto a fictional background, virtual sets allow filmmakers to shoot the scene directly in the fictional setting, complete with the visual effects.
With the LED video wall already in the background, the director and the crew can see the final result in real-time, rather than having to imagine what certain scenes would look like once the effects are finalized in post-production processing.
Creative Flexibility
LED displays allow the production design team to edit the background in real time. The virtual sets can be adjusted according to the director’s specifications without taking too long. Unlike physical sets that require manpower and can take hours to tweak, LED XR displays can be edited in minutes. Whether you want a giant piece of rock moved or the clouds darker, the production team is in full control.
Limitless Location Options
Finding shooting locations is one of the most time-consuming steps during pre-production. The team would have to go from one place to another to find a house or an outdoor spot that would suit the specific scene they’re going to film. Add to that, they’re also playing against time and weather. With virtual production techniques, designers can recreate any location–indoor or outdoor–in the studio and make it appear like it’s sunny or rainy with just LED panels.
Realistic and Immersive
Virtual production creates realistic and immersive environments using real time rendering. It incorporates the principles of green screen filming with the practicality of on-site filming. The actors can be in any environment, be it real-life or fictional, and still have natural-looking lighting while being indoors. This is crucial in ensuring the scene looks as real as possible by casting the appropriate light and creating reflections on the actors’ faces. Replicating this in post-production can be time-consuming and expensive.
More On-Screen Work Over Post-Production
Some actors and filmmakers find working with green screens challenging as they can’t see the final look of the environment they’re working with. Visual effects are done in post-production, and actors and directors would have to make their imaginations work to film a convincing scene. Virtual production eliminates this challenge as the characters can interact with the setup as if they are in a physical set, resulting in a more convincing execution, fewer takes, and lower production costs.
Types of Virtual Production LED Display
While the most common LED configuration in virtual production studios involves three flat-screen displays, more advanced setups involve LED volume walls, which are a series of LED displays with high-end components. Virtual production can be compared, in essence, to filming a movie on an ultra-HD LED floor-to-ceiling TV.
Curved Display
In a virtual production, curved XR LED screens function as the primary background. Instead of two vertical LED walls making a right angle, as the name implies, it curves around the studio to give the set a more realistic and immersive appearance. Flexible LED panels are the major focus point of sceneries, so high-quality modules are needed to provide crisp, high-definition visuals.
LED Floor Display
Scenes are completed in the foreground by using floor LED tiles. Because of their strong load-bearing capacity, actors are able to stand on them for extended periods of time. Additionally very interactive, LED floors can detect force and send signals to particular areas of the display. A three-dimensional image is produced by combining floor LED tiles with a curved LED display, giving the foreground and backdrop a realistic appearance.
LED Ceiling Display
The three-dimensional setup is finished with ceiling LED screens mounted on the ceiling. They provide natural lighting and serve as a spectacle for the sky. A ceiling LED panel connected to the LED wall can give the impression of continuity.
Working with Virtual Production LED Walls
In order to work with a virtual production LED wall, the following software and hardware are required:
LED Displays
The most crucial element in a virtual production are the LED panels. They act as the background of the movie and showcase the set design. When selecting LED panels, keep the following in mind:
- Color accuracy and refresh rate: color accuracy guarantees that the colors on the display are reproduced exactly as they were intended, while refresh rate modifies the sharpness of the images. A low refresh rate will result in pixelation and blurring of the display’s images.
- The distance between two pixels is known as pixel pitch. Images with a low pixel pitch have more detail.
- Heat dissipation and power supply: The power supply controls how much electricity the display requires. Power produces heat, therefore heat dissipation is crucial to keeping LED displays working at optimal temperatures.
- Type of LED : ceiling, curved, or flat.
Camera Tracker
With the help of camera tracking technology, the camera can move in sync with the virtual environment, ensuring that images on the LED walls adjust accurately as the camera shifts its view.
Genlock System
To avoid scan lines in the background and maintain consistent frequencies, the genlock system synchronizes the LED panels, rendering machines, and camera.
Rendering Machines
Rendering machines render each pixel in real-time, adjusting tiny details on the spot to avoid post-production issues. Because virtual production LED walls involve millions of pixels, the rendering is divided into multiple computers that work in sync.
Virtual Production LED Wall Price
While filming movies using virtual production LED walls can be more cost-effective than bringing the entire crew and equipment to a physical location, the upfront costs for putting up a virtual production studio can be steep. For instance, the Mandalorian LED wall built for the first live-action Star Wars TV series was estimated to cost more than $10 billion. The 20-foot curved LED wall had 1,326 P2.84mm LED screens and can be lifted in and out of set.
The virtual production LED wall cost can be affected by several factors:
- Screen Size: Larger screens are naturally more expensive than smaller ones. The screen size also depends on the size of the studio. Larger displays involve more individual modules, adding the LED wall’s final cost.
- Pixel Density & Resolution: Pixel density and resolution directly affect the quality of the display. A low pixel density and a higher resolution result in more detailed images, and vice versa.
- System Requirements: The cost of the software responsible for real-time rendering can significantly add to the LED display’s final cost.
- Customization: Virtual production LED walls can be customized according to your specifications. Some productions, such as The Mandalorian, could require a curved LED screen for an immersive experience. The shape is special, and the software would have to be customized to fit the display specifications.
Price Range
Small applications involving 10-20 sqm LED walls can range between $50,000 and $100,000, depending on the quality. Medium-sized applications involving 30-50 sqm LED walls can start at $100,000 to $250,000, while large projects involving displays more than 50 sqm can start at $300,000 to millions.
Challenges and Limitations of Virtual Production
Virtual production is the future of filmmaking. However, its application is not as common as its proponents would like it to be due to several factors:
Learning Curve
Virtual production technology is still in its infancy, and only a select few is well-versed with it. Traditional filmmakers may find it difficult to adapt, especially those who are accustomed to the simplicity of green screens. They may also encounter technical difficulties as the technology is not yet polished.
Virtual Production Techniques
Virtual production is not intended to completely replace green screens. Instead, directors are encouraged to integrate existing traditional techniques with virtual production techniques and get the best of both worlds.
Ethical Concerns
Traditional roles such as set designers may become irrelevant once virtual production is more widely-used. With graphic artists responsible for designing virtual sets, location scouts, production designers, and carpenters may become less in-demand.
Conclusion
Virtual production technology is changing the film and television industry with its undeniable benefits. However, there are ethical concerns in the use of virtual sets, considering they can possibly take away thousands of livelihoods. In this regard, it is similar to Artificial Intelligence. For both technologies to really be of help rather than hinder human development, they should be used in moderation–to aid rather than to replace.
