How Do You Map the Metaverse? Ask Ted Schilowitz

If you believe that the future of the internet is the Metaverse, then walk this way. ViacomCBS has officially announced it is exploring ways it can take its entire canon from divisions like Paramount, as well as future IP, into mixed reality (MR) to be consumed on next-gen immersive displays.

Heading the charge is Paramount Studios’ resident futurist Ted Schilowitz who has been tasked with exploring ViacomCBS’ interface with the emerging forms of visual computing.

In an exclusive interview for the Makers Pop-up virtual conference Schilowitz reveals, “Unofficially for the last couple years I’ve been covering all ViacomCBS in terms of the pursuits around the Metaverse. We’re not looking at the next 10 year journey to 2030. Officially now my purview is across all of ViacomCBS enterprises of which Paramount is one division.

“It’s exciting to have a belief from the company that we really need to pay attention to where things are going.”

The Metaverse (variously trademarked as Omniverse or Magicverse by Nvidia and Magic Leap) is a term gaining buzz in Hollywood and Silicon Valley to describe the evolution of our relationship to the internet.

It is understood to be a digital mirror for real world activities including work, social, health, fitness as well as entertainment.

READ MORE: What is the Metaverse and Why Should You Care? (NAB Amplify)

“It’s about being inside an immersive interactive environment that starts to feel real to you,” Schilowitz says. “That is the next big step in entertainment and it is starting to take off. New VR/AR devices will enable us to enter these entertainment worlds and take you to the holodeck,” the VR environment famously from Star Trek, part of the Paramount universe.

Star Trek aside, ViacomCBS owns huge brand assets from Nickelodeon to MTV that attract a younger audience. It’s the same demographic most likely to engage with experiences more familiar to gaming and who expect to interact with multiple digital touchpoints in their daily lives.

“It’s important for Paramount and other studios to constantly look at youth culture, how they absorb their information and how they expect to interact with digital worlds,” he says.

“We are mostly locked today into looking at blocks of pixels on flat, rectangular screens but the next screens we are experimenting with — as are companies like Sony, Facebook, Apple, Google, Microsoft — take the visual compute experience to the next horizon. We will start to take these static pixels that we currently create in a grid and park in front of our faces and change them to dynamic spatial pixels that move with us and track with our body and our eyes.”

— Ted Schilowitz

“My [responsibility] is to take all the hit content created over the 100 plus years at Paramount and the 45-50 years of the Viacom universe and see how we propel that forward into what the customers of tomorrow are starting to do today.”

That doesn’t necessarily mean producers should pivot pitches into the MR space. Schilowitz’ advice is to “pitch what you are passionate about, don’t try to lock into one platform or another. Speak to the creative core of the business not the tech side. The tech core should support the creative essence, not the other way around.”

The Digital Theme Park

The final picture may be indistinct, the puzzle has a zillion pieces, but Schilowitz insists the outline of the edges are already here.

“From a technology perspective we are extraordinarily close,” he says. “We are already [experiencing the Metaverse] in various ways albeit mostly using traditional screens. Overt examples would be games like Fortnite where people gather in a virtual space and spend hours of their day. It’s not just trying to achieve a game objective. It’s about socializing in a more avatar-oriented three-dimensional space.”

Earlier examples are the multiplayer world building games like Second Life, Star Citizen and The Sims. Minecraft and AR game Pokémon Go are other pointers. Schilowitz takes his own inspiration from Disney World in Florida, where he grew up. Until recently, theme parks were the only place you could get a sim experience at scale.

“The original metaverses were theme parks — amazing environments that didn’t come from the natural world but from creative people’s brains,” he says.

READ MORE: In Conversation With Paramount Pictures Futurist Ted Schilowitz (IBC 365)

“As someone who feels younger than they are [Schilowitz is in his late 50s] somehow my DNA and brain chem still works like a teenager when I view entertainment. I view interactive entertainment as a theme park style experience where it’s less about the hardcore adrenalin rush of playing a game — though that’s a part of it — and more about living in a simulated environment.”

What is changing rapidly is the form factor and audio visual acuity of immersive displays. Google, Magic Leap, Facebook and the entire VR entertainment industry may have gone back to the drawing board after early less than successful iterations of head-mounted gear — but lighter, less cumbersome, higher fidelity and more affordable wearables (combined with 5G connectivity) will usher in a new wave of immersive content experiences.

“There will be devices we wear on the street like glasses, those designed for home entertainment and those designed for special places to connect with other people in real and digital worlds. All of this is the Metaverse.”

— Ted Schilowitz

“We are mostly locked today into looking at blocks of pixels on flat, rectangular screens but the next screens we are experimenting with — as are companies like Sony, Facebook, Apple, Google, Microsoft — take the visual compute experience to the next horizon,” the futurist says. “We will start to take these static pixels that we currently create in a grid and park in front of our faces and change them to dynamic spatial pixels that move with us and track with our body and our eyes.”

Emerging displays will ditch the remote for more intuitive ways to interact with the content.

“When we start to get a touchscreen interface, or one we navigate just by our eyes, or with haptic feedback, you start to blend the branching narratives of games into the world of a mainstream audience narrative,” he outlines.

Netflix, which financed the pioneering branching narrative show Black Mirror: Bandersnatch, is reportedly exploring an expansion into gaming which would put interactive content development on the fast track.

“Interactive branching entertainment will really thrive with the next-gen of wearable headsets. They will bring story and interactivity together in a way only a large theme park simulator could have done in years past.”

— Ted Schilowitz

Schilowitz says ViacomCBS streaming platforms like Paramount+ and PlutoTV are ripe for similar exploitation. “What we’re doing is exploring what we should do with those platforms in the future as we move them past just a passive entertainment place to a space where people want to truly engage,” he says.

“Gamers who have played largely narrative driven games know what a true branching narrative means and that it doesn’t hold up on a regular TV screen with a remote. You need to be much more engaged connection to the characters and the story.”

None of this means the going to the movies will go out of fashion. On the contrary, he believes theatrical will live on just as we all enjoying see a play on Broadway.

“Seeing a motion picture with other people on a large screen won’t go away in the same way that live theatre and live music hasn’t gone away despite our ability to experience these events digitally. People will go to the movies because they want to have that experience and not because it’s the only place to see that movie.”

Building Video With Spatial Pixels

Studios like Paramount however, want to be ready with the content when new devices capable of reaching mass audiences in AR/VR/MR arrive.

“Interactive branching entertainment will really thrive with the next-gen of wearable headsets,” Schilowitz says. “[They] will bring story and interactivity together in a way only a large theme park simulator could have done in years past.”

The building blocks of MR content creation are here in the form of real time pixel generation from games engines and graphics cards.

“That’s on the high-end content creation side but there are platforms for people without traditional coding skills to build their own universes and make money with it.”

Examples of such virtual sandboxes include Roblox, Rec Room and Manticore (in which Epic Games has invested).

“We will see a YouTube for interactivity,” he predicts. “The way we create content doesn’t mean you have to have a traditional computer science background. A wider audience with almost no computer programming skills can start to build things in a way that traditional video tools have been democratized. Everyone has a high-end camera on their phone now. We are starting to do that with real time graphics and animation at scale.”

“The pandemic has shown us that people are comfortable in their digital individual lives. By 2030 there will be a lot of creative experience that have come to scale. We can see the edges of it now.”

— Ted Schilowitz

That’s why a lot of Schilowitz’s R&D is focused on tech and creative start-ups just as much as the well-resourced studios and tech giants.

“You learn where things are headed from that start-up companies,” he says.

He is an investor or founder of several, including 3nfinite (formerly HypeVR) which has developed an app designed to enable the viewing of volumetric video over 5G connections.

He thinks we need ways to “build” video for the Metaverse. “Currently video has a 2D representation. What we need is to create new styles of video. Volumetric video [the capture of scenes using multiple imaging devices] broadly blends the idea of gaming-style spatial pixels with true video pixels. Lots of companies are working on this.”

The timeframe for these developments is startlingly proximate. The Metaverse is already here if below the radar, according to Schilowitz, and a more evolved version will arrive within the decade.

By 2024 the next set of consumer AR/MR wearables will be on the market. By 2027 this ecosystem will be mature if still at the early adopter phase. 2030 will see the beginning of mass adoption.

“By 2030 we will still be using smartphones for certain things but also other more evolved tools used in tandem with or as replacement for smartphones,” he says.

Rather than a single all-in-one device, we will interface with the Metaverse from multiple devices — just as we do today via the 60-inch living room TV, laptop, smart watch or pocket phone.

“There will be devices we wear on the street like glasses, those designed for home entertainment and those designed for special places to connect with other people in real and digital worlds. All of this is the Metaverse.”

He adds, “The pandemic has shown us that people are comfortable in their digital individual lives. By 2030 there will be a lot of creative experience that have come to scale. We can see the edges of it now.”

The Metaverse is poised to become a multi-billion dollar playing field for every company from Roblox to Nvidia, Facebook to Microsoft, Google to Epic Games. They all want to own a slice.

“There are major questions about how we do cross collaboration and how we make sure there’s an open ecosystem,” Schilowitz says. “And also that profit should not be the only motivation.”