Medal of Honor: Above and Beyond –Using WebGL to craft an immersive hand-painted 3D battlefield

Jam3 partnered up with Oculus, Electronic Arts and Respawn to create a web experience for Respawn’s Medal of Honor: Above and Beyond.

The site has won Site of the Year at CSS Design, and both Site of the Year and People’s Choice Award at the FWA.

The immersive site for Respawn Entertainment’s Medal of Honor: Above and Beyond brings the 3D feeling of Oculus VR gaming to players on 2D surfaces. Step into the shoes of an OSS agent during WWII in this new installment of the beloved AAA franchise the way it was always meant to be played: in 360º reality.

The campaign was divided into two phases. A teaser website which featured dev diaries from Respawn and the main WebGL Experience, which would launch alongside the gallery trailer.

The WebGL experience turned the game’s painterly key art into an immersive 3D battlefield. By scrolling through the action, you can experience selected features and gameplay moments. By the end, you have a crystal clear idea of what to expect from the game: a stunning first-person experience, gripping moments of heroism, and innovative combat mechanics.

This article will cover our development approach for the WebGL experience, including:

  • Our process for developing the concept and art direction
  • Our early storyboarding and prototyping process
  • How we managed our 3D Asset pipeline
  • How we developed and executed our lighting, textures, and visual effects for optimal performance
  • How we integrated our scene with WebGL and accommodated a range of devices
  • How we brought the scene to life with camera movements, positional sound, scene transitions, and post-processing effects
  • How our visual design language took inspiration from war era dossiers

Concept

Medal of Honor: Above and Beyond is the first game in the series to come to life in Virtual Reality. We were fortunate to have access to an early version of the game, which helped us find our creative direction. During our play through we captured locations and narrative moments we could potentially translate into story beats for the web experience.

One of the locations that stood out to us was the French Village of Dubuisson. Here the player assumes the role of a WWII resistance fighter.

Art Direction

With the location set, we worked to develop a visual style that would reflect the era and themes of the game, while remaining visually distinct from in-game graphics.

Our moodboards were inspired by the impressionistic style of the key art featured in the game. This painted texture mirrored iconic WWII propaganda posters that perfectly captured the tone of the game.

Our creatives painted a style frame to help our team to align on key elements the site would feature. It also helped us demonstrate to Oculus and Respawn what our vision for the website could be.

Next, we experimented with some parallax animation to get a feel for how the camera could move through z-space.

Prototypes

Impressionistic style rendering has been used in several games, and is well-documented in articles like this one by Disney Animation. We began to explore how we could use a painterly style effect in real-time using WebGL.

We liked the direction this treatment was heading in, but we weren’t sure how this effect could scale and how much detail we could push across a large-scale world.

Creating Dubuisson

Medal of Honor: Above and Beyond was still in development at the time of this project. Therefore, we weren’t able to use any of the game assets for the website. It became clear that creating the assets ourselves would be more time efficient and would give us greater control for the art direction we wanted to achieve.

Before any asset creation could take place, we drafted some plans for the town layout and camera path. This would inform what we needed to model and where we could create action moments.

Our creatives mocked up a camera path animatic in Blender to help visualise our story in 3D space.

The main benefits of this approach were that it helped Jam3 get client approval, it informed our creative team on what to model, and it helped to visualise the pacing between narrative moments.

3D Asset pipeline

Establishing a strong art pipeline was essential for our 3D artists and developers. We started off by assembling a basic layout of the town and applying some first pass textures on the models.

Textures were hand-painted and iterated through several times, adding in more and more detail with each pass. It was a very time-consuming process. Sometimes models or UV’s would change, requiring the texture to be repainted.

The town looked very flat initially, without any shadow or ambient occlusion detail. To solve this, we set up a lighting rig in blender to mirror the game’s dusk lighting. Shadow maps were baked and applied to groups of 3D objects, such as multiple trees and foliage. For single-instance models, we added the shadow maps directly into the diffuse texture to reduce the amount of shadow maps needed.

Timelapse of the 3D level creation

Exporting to WebGL

Our next challenge was to export our scene, compatible and optimised for mobile devices. There were a variety of ways we could do this including:

  1. Export the entire scene including textures as a glb file
  2. Export the scene as an FBX file and load the textures separately

All of our textures were painted at very high resolutions, so the first approach wouldn’t be feasible. We also wanted to deliver two variations of each texture, based on the device’s capabilities. So we chose the latter approach.

The glTF format contains mesh nodes that map to materials and textures. Using the texture indices, we can look up which meshes use which texture.

We wrote a nodejs script to generate two data files. The first was mesh-to-material mappings, which would be used at runtime to apply the corresponding textures to the meshes.

The second script generates a config file so we can customize the resolutions we want per platform.

It became clear early on in development that PNG and JPG files weren’t going to be performant with the amount of textures that would be required for the entire scene.

Creating a 3D world at this scale, with this many textures, would require a different optimisation approach than WebGL experiences we’d developed in the past.

Basis universal texture format is a compressed texture format for GPUs. It’s a highly optimised and universal format that works across a variety of GPU’s which enables us to use higher resolution assets with a lower memory footprint. Threejs has built in support for it too, with the BasisTextureLoader.

File size reductions for the town model and textures

Using basis universal compression for textures and the Draco compression for glTF gave us a huge reduction in file size and GPU memory. This allowed for both greater fidelity and faster loading times.

Colours were used to visually debug texture resolutions in the scene.

With the scene being completely static, we also disabled matrix calculations to speed up rendering performance.

Discovering game content

Collecting dossiers was a game objective used through several of the Medal of Honor games. We liked the idea of collecting intel on the enemy, so we used this mechanism to reveal game content to the user.

Hotspots would appear at key moments throughout the scene. Opening them would showcase footage from the game.

We used threejs CSS3D renderer to anchor the UI hotspots to the world.

Lights, Camera, Action

We started off with a simple mobile prototype that used the device’s gyroscope to control the camera, letting you look around the environment.

We later added the ability to keyframe field-of-view, scroll speed, camera rolling and dynamic post effects. These effects are interpolated along the path as the user scrolls.

The camera path manager is editable in the browser, allowing us to preview the camera path in real-time.

Screen space motion blur is applied for the high graphics setting.

Mobile uses animated transitions from hotspot to hotspot. The camera movement is also enhanced by the gyroscope.

Scene Transitions

The site consists of multiple 3D scenes: preloader, home, intro, experience, and end. To create a seamless transition between these, we used a mask sprite sheet animation in our post-processing stack to blend one scene into another. We also Pre-render scenes for smooth transitions

On the experience scene, the navigation timeline acts as a quick menu to jump from hotspot to hotspot. The transition mask effect didn’t account for this initially, as it required the same scene to be rendered twice. We modified our post-processing stack to account for this new behaviour.

Positional Sound

Sound always plays a crucial role in immersion. We used threejs web audio capabilities to add positional sound to the scene. Sounds play around you, anchored to objects and also triggers as you scroll past. We created utilities to move and preview them in context. Each sound had its own config, allowing for greater customisation.

Post Processing

With the scene at 500k triangles, we needed to be conservative with post-processing effects to ensure that we wouldn’t need to re-render the scene multiple times. We created a minimal post-processing stack that turns on effects when required by the scene.

Passes

  • Transition Pass (active during scene to scene transitions)
  • Motion Blur (enabled for high graphics mode, active during scrolling)
  • Depth of field (active when the user sees a hotspot)
  • FXAA, Vignette, Noise (active during the experience scene)

We experimented with a painterly pass too. This used the world depth position and triplanar mapping to distort textures in world space. The downsides to this approach were loss of performance and texture detail.

Threejs does a good job at sorting depth transparency in most cases. However, we noticed a few issues with transparency clipping on models. We manually set the render order for these transparent models to fix any visual glitches.

Graphic modes

We always strive to deliver the best graphical experience for the end user but we also have to be realistic for the range of devices we try to support. Two graphics modes are available: high and normal. We use GPU Detect to profile and assign a tier to the gpu. This allows us to optimize the experience on a wider range of devices.

Other areas we optimise include:

  • Particle count reduced
  • Triangle count reduced on certain meshes
  • No motion blur
  • Lower resolution textures
  • Mobile optimised videos

User interface

To pay homage to the series’ history, we decided to uphold and evolve the dossier visual language that originated in the first game. We wanted every element to feel like it was designed in the 1940s. This meant integrating textures that looked like real materials you could touch and feel.

Going overboard with paper texture can border on cheesy, so our aim was to find a balance that kept the UI feeling light, but still reminiscent of a physical briefing document. Bold brand fonts were grounded with light textured rules. We wanted the hotspots to feel like they were pulled out of a soldier’s notebook, so we designed them as quickly rendered pencil marks.

The UI layer was built with React, GSAP, StyleX — a custom inline JavaScript stylesheet library, and a custom ReactTransitionGroup-like library to animate the StyleX classes.

Given the design focus on the painterly style for the 3D scenes, the UI layer needed to match the more tactile look and feel of the experience.

We ruled out using pure SVG elements or coloured DOM elements to populate the design touches, instead choosing to use background images to create the textured look we were aiming for. We created an SVG button component with the paper-like texture and added the icon options as a mask to keep the button scalable and transparent.

Typewriters were considered an essential tool within the military during WWII. We chose DIN Next Slab as the global font choice for all the user interface copy, also within the game development diaries. Combining this font with paper-like textured mirrored WWII documents more authentically.

The motion chosen for the design elements also spoke to the skeuomorphic nature of the design choices. Such as stamps pressing into the page, lines being drawn, and character profile images flashing in as if it was a photograph developing. We achieved this with the use of GSAP library and the custom transition group library.

We also used a sprite sheet animation on some elements to give the motion a more unique feel. The animations were created in After Effects, translated to sprite sheets and loaded in

individually instanced 2D canvas elements and attached animation triggers to DOM events.

Final thoughts

Peter Hirschmann, the creative director of Medal of Honor: Above and Beyond and the original 1999 classic, said their goal was to make the experience feel “grounded and emotionally authentic”.

While technology can give us new and innovative ways to make and deliver experiences, the real experience is always the story. In this case, our work was grounded in the stories of those who fought and experienced the war first-hand. The story is the real hero and the technology is how we bring that story to life. The two should never be considered separately.

At Jam3 we take great pride in making sure the creative and technical choices we make work invisibly to enhance our storytelling and provide the most authentic web experience possible.

Medal of Honor: Above and Beyond is now available on available on Oculus Rift, Oculus Quest with Link, and SteamVR.

--

--

--

Senior Creative Developer @Jam3

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

I Can’t Hate on Borderlands 3

Do Game Reviews Influence Sales?

Flashback: Friday Rat Queens: To the Slaughter Preview

Limited Edition Devotion Oathsworns VIM Mission Event!

Why I quit Google to become an indie game designer

Nft + AI Games, Unlock New Game Ecology

CSC November Update #1

Play2Live in Asia — understanding gaming in 12 Asian countries (part 2)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Amélie

Amélie

Senior Creative Developer @Jam3

More from Medium

How to Make a Run Cycle in Pixel Art

2 — Semester One: Welcome Back Student

Mobile Applications In Unity: Rainy Day Insurance — Anchor Points and Preserve Aspect Ratio

Augmented Reality Meets Space Exploration