fbpx

I’m Dreaming: A Mixed Reality Video Odyssey

What do you get when you add 40+ hours of mapping and lighting, 30+ hours of custom platform development, at least 25 full level play-throughs, and 100+ hours of editing and post-production?

A mixed reality masterpiece… that’s what. I had the opportunity to sit down with mapper/lighter Skyler Wallace, 3D modeler AkaRaiden, and mixed reality video guru TougeVR to talk about their until now top secret project to create an immersive Beat Saber experience based on the music video for I’m Dreaming by INZO, video by Lowly.io.

HelenCarnate: How did the project get started? Who was the brainchild behind it all?

TougeVR: That would be Skyler!

Skyler Wallace: So usually I start my mornings and my nights the same way – by listening to music and one morning I woke up and was listening to music on YouTube Recommended and I heard “I’m Dreaming” by INZO playing the day it came out and it was me and my girlfriend in my room together and I was playing it through my speakers and it was so powerful and atmospheric. At the same time, I had a lot of ideas going through my head of things that I wanted to do. I remembered that Aaltopahwi had posted in the BSMG Discord this flashbang lighting effect. I was also thinking about how cool the Anniversary Campaign “boss fight” that me and Raiden put together was and I thought maybe we could go even higher and incorporate some of the cool lighting effects that I had seen Aalto do with Chroma.

So I approached Raiden first and I asked him if it was possible if we could do this project and he said it was. I wanted to take it a step further and see if I could promote it with Touge, so I reached out to him next and it just kind of went from there.

purple pyramid in a forest
Screencapture from the actual I’m Dreaming music video by Lowly.

HC: Do you listen to a lot of other music from INZO or did this just kind of come up and you thought it was cool?

SW: Most people know INZO primarily for the Overthinker song, me included.  This one got my attention on Youtube on its release day.  The idea to do this project was mostly coming off of the Anniversary platform that me and Raiden had done – just thinking about that crystal design and naturally that extended to the prism from INZO. it felt very similar to me and it felt natural to do next. I like all kinds of EDM and all of it just kind of shows up in my YouTube Recommended at some point.

HC: So Raiden, what drew you to this project? I know you and Skyler have worked together in the past. The boss fight platform was super cool in the Anniversary Campaign. What was it about this project that had you intrigued?

AkaRaiden: Honestly? Any chance I can get to work with a mapper to make something  specific for a map – I’m all in right away. Especially this, because it just looked awesome in the video.

HC: Touge, same question for you… Skyler made his pitch. What piqued your interest?

TVR: First of all, the song. I listen to a really broad range of music and it’s not typically what I would listen to, but the song combined with the video just immediately grabbed me. Working with Skyler in the past with the Doom video that we did together I was just like “This is going to be amazing. I’m in!”

HC: So what was it about this particular project with everything that it entails – the map, the lights, the platform, etc. – that made it so ambitious?

SW: It was a lot. That was mostly my fault, though, because I would ask Raiden to do something and he would do it perfectly. And then I’d say, “Hey, can we do this ,too?” and then he would do that so I just kept telling him to “do this” and “do that,” but I think the end result was really good.

There were a lot of challenges we had to overcome. We had to learn a lot about the lighting engine of Beat Saber and we actually found out it was really limited. Me, Raiden, and the current maintainer of custom platforms, Uialeth, had conversations about how the lighting engine works and we had some challenges getting a very large object to light up and fade in a pleasing way.

AR: Yeah, I pretty much had to script in special events to create the fades for everything so that they would work somewhat normal because, at first, it was just a nightmare. In custom platforms when you create a light, if you don’t do anything extra to it it basically looks really bad. They basically just light up and turn off – there aren’t really any fades or anything like that. They don’t go completely black so I had to make all of that special, just for this.

HC: I went back and watched the original INZO music video which has a lot of flickering and flashing. Considering that you were up against full-blown CGI, you really captured it well!

SW: It was really tough. I think the main issue was getting fades to work properly. So we basically had to combine the fade event with the new Chroma 2.0 gradients to get that smooth effect. If I didn’t have the Chroma gradient there, what the light would actually do is go from pink to dark red to white and then black. Raiden actually worked with Tachrayonic, who is another platform and saber maker, to get a shader that would work well for what we wanted. So we had to create a new shader that could help do that transition a lot smoother. And after that I combined it with the Chroma gradient to make it even smoother, but even then it’s going from pink to grey to black which is better than what we had before. Even then, we still had to do a lot.

When we sent it to Touge he was getting the fades that basically strobed for a second and then turned off so I had them make an alternate lighting version just for him with more events to help that transition.

HC: Touge, you have done some absolutely gorgeous, energetic, really beautifully produced mixed reality videos. Was there anything special that you had to do on your end with this particular platform to make it look really good?

TVR: Yeah, so the scale of this platform is obviously very large which will become apparent when you play it. The size of that pyramid is just huge. I’m fortunate to have a fairly large room to shoot all of this stuff in so there are places where the camera is like eight meters (editors note: that’s ~26 feet for the ‘Murican readers) from where I’m stood and I’m just this little guy in the corner in front of this enormous pyramid and it still only just fits.

INZO pyramid in Blender
An early look at the pyramid platform element in Blender.

HC: That’s impressive! When you’re putting together a masterpiece like this, with all of the rotating cameras and multiple angles, how many takes do you usually have to do on a map to capture everything you want to put this together?

TVR: That varies, really. For example, the Doom one was about 35 or 40 takes. There was a lot of really unusual things happening in that like the walls were absolutely next level so there were a lot of different angles that we needed to consider. The majority of those were full play-throughs of the map as well. So it took about two full days shooting that. There’s videos I’ve done which have been more than that. Rustic did an End Game map shortly after that Doom one and that was basically 10 days of shooting and editing.

Clapboard showing 21 takes against a green screen background
21 takes later… shout out to Mrs. Touge working the camera!

HC: And do you do all the editing yourself?

TVR: I have support from other editors around me, you know, asking questions and things but yeah, I do everything here.

HC: Was there anything about playing through the map that you wanted to capture but weren’t quite able to, either through technical limitations or size limitations?

TVR: At this very minute the shoot is still in progress.. I’ve literally just stopped to do this interview so as soon as we finish I’ll be back into it. One of the things I really want to capture – and it’s a shot that features quite heavily in the INZO music video by Lowly – is this incredibly wide shot that’s like 50 meters (164 feet) away, way up in the sky and the pyramid just looks like it’s small, just in the trees. With LIV and the avatar function I believe I’ll be able to do that so that’s my next challenge before release is to get that shot.

The video that I’ll be making will be somewhere between the video from Lowly and a conventional Beat Saber video. There will be game play that you’ll see but it’s going to be quite stylized compared to a regular Beat Saber video.

HC: Was there anything special about the way that you put this video together?

TVR: Part of the reason that I have this studio at my house is that I make mixed reality trailers for VR games. Chances are if you’ve seen games like SynthRiders, OhShape, Shooty Skies, and others in the Oculus store, the trailers have been shot here. The technique we use for making trailers like that is much more involved than the typical Beat Saber video. The method we used is called “dump mode” from LIV which separates out all the individual layers from the mixed reality capture so the background, the foreground, the alpha key, and the camera footage is recorded separately. So what’s different about this I’m Dreaming project is that we’re applying that technique to this and that opens up the ability to control lighting on the subject as well. The lighting of the environment will be reflected on the player. This is a manual process in post-production.

A look inside the multi-layered mixed reality composition process.

HC: And so, Skyler, I assume that was your vision? That you were trying to recreate the feel of the music video as much as you were trying to make a map?

SW: Yeah, exactly. When I heard it for the first time, the experience of the song just surrounded me. I wanted to recreate that feeling for others and I thought Beat Saber would be really good for that. I wanted to create something huge, something massive in scope to really wow people more than what’s been done in the past. I know a lot of amazing things have been done with Mapping Extensions and Noodle Extensions but I think platforms are really the ultimate path forward to creating amazing experiences in Beat Saber.

HC: In terms of a massive platform, Raiden, how did this compare to the big Anniversary campaign boss fight platform that you did?

AR: It’s about on the same level, really. The hardest part was trying to set everything up. I went back and forth with Skyler a lot about the scale of things, trying to get everything sized properly and positioned properly that it would feel right and that was a big part of the challenge.

HC: A+ on the aurora borealis lights in this because that is such a cool effect.

TVR: It’s incredible

AR: Yeah, that’s all done with a shader.

HC: I’m pretty much in awe of the incredible things that modelers like you are able to accomplish with these environments because it’s visually stunning, and Touge, I have to imagine that it’s equally stunning to be standing there in the middle of a forest with a giant pyramid in front of you in Beat Saber.

TVR: It’s totally unique. Obviously wall maps and that sort of thing have come such a long way… there are no walls in this and it’s still, by far, the most stunning Beat Saber experience that I’ve had, playing this map.

SW: That was a conscious decision as well, to have no walls. We were already pushing the performance cap really hard with this platform and me and Raiden even went through multiple revisions just trying to get performance in check. So it didn’t really make sense to have walls because of performance. Originally I did contact SpookyGh0st to see if he’d be interested but pretty early on we realized that adding walls to it would just crush performance. I did use some basic walls at the start but it seems like it distracted from the map, more than it helped the map. There was also a heavy use of bombs as well, especially during the Post-Drop, but those were removed for the same stylistic reason.

HC: There are two parts in the music video where you get pulled away from the forest where it’s like you’re looking through rain into a more abstract where it looks like there’s crystals breaking apart and stuff. How did you approach what you wanted to do visually for that part?

SW: I did toss around a couple of ideas with Raiden and my girlfriend actually came up with the final idea, and that was to have the prism shoot out pieces of rain from it. Ultimately what you would want to do is have the prism explode or something but with the time constraint of this and the amount of performance overhead we had it’s just not possible to do that. We did pretty much the best thing that we could do within Beat Saber. The main thing besides the rain that we have is that subtle flash bang effect that happens had to be bright enough to be noticeable but also dim enough where you can actually play along to the map. If you actually watch the video you’ll see as the pyramid starts to glow it gets so bright that even your monitor or headset won’t be able to detect the difference so me and Raiden had to explore other ways to show the scope of just how bright the pyramid was getting. The note highway actually plays a big role in that. You’ll see it filling up like a finisher meter in a fighting game. You’ll also see that the trees light up during the drop which was a huge obstacle for Raiden and I to do.

AR:  The trees lighting up and making a lighting system for them was just horrible altogether. It was a nightmare. In the end I had to use some really wonky shaders and find a way to make them show the default Unity lights because they don’t actually show up within Beat Saber. So I had to find a shader that would actually show them and then we had to script an entire lighting system so that he could control brightness and everything.

trees in unity
A look at modeling the forest for the custom platform in Unity.

HC: Touge, did you get a change to go in early on before there was a map and just stand there in the forest and experience it?

TVR: I did, but at the time I got the opportunity to try a few versions along the way. Some were missing certain elements that we were talking about so it wasn’t until the version we have now that it all really came together.

HC: I assume, because of the secrecy around this project, that you were probably the primary playtester for this. Did you play through multiple revisions of the map itself as Skyler was working on it?

TVR: I think I’ve uploaded four, maybe five test videos for them to look at so there are a few versions that I’ve tried.

HC: Skyler, I know you are one of the masters of the “full spread” so when this experience drops will it be a full spread or, because of the special circumstances, have you kept it a bit more limited?

SW: There’s no reason for it not to be full spread. The only compromise that I’ve made in regards to that is that there will not be an Expert+ difficulty just because the current map for it is so sparse in notes that it would be disingenuous to call it Expert+ and it will be missing 360 for obvious reasons.

HC: Let’s talk about the map. Initially listening to the song I thought that this is an “interesting” choice for Beat Saber since it isn’t until you get to the peppier part in the middle that it picks up a little. It’s very atmospheric. What was the hardest part of the mapping itself for you to get right – for you to feel like you were mapping the energy of the song and the video?

SW: I took my inspiration from Rustic for the mapping. I originally had a couple of revisions of the map where I had more traditional 2020 flow but as a very visual experience I decided it would be better to have a more angular look to the map and use less diagonals overall and use more cardinal directions. So for the drop, what I did was I took that idea and I made the swings a lot wider and a lot more abstract in nature. Within like two seconds of the drop you have to hit this really weird pattern where your right arm is over your left shoulder and your left arm is under your right armpit so I just kind of went for really strange, alien-feeling motions during the drop. During the softer parts I went for something a lot simpler.

Skyler at work mapping and lighting in the ChroMapper beta by Caeden117

HC: Technology issues with lighting aside, think about lighting now from the mapper’s perspective. What was the most challenging thing to get just the way you wanted?

SW: It was probably a tie between getting the prism to spin smoothly and also getting the gradients to work well in Chroma. The issue I had with getting the “rings” to spin is that in custom platforms there isn’t currently a way to get things to continuously spin in a ramping motion. So what Raiden had to do was find a way to give different speed values for the spinning. By default it’s at 1x speed and it goes all the way up to 50x and we had to get different intervals between 1 and 50 to where it’s not too many values so it’s overbearing but also is close enough together that it feels smooth. We have values for 1x speed, 1.5x, 2x, 3x, all the way up to 10x in intervals of one. Then 10-20 is intervals of two, and then 20-50 is intervals of five and we found that to work really well while also not being super painful to map.

I also had to create a very complicated Desmos calculator and graph in order to create basically an easing function for time values, so it doesn’t just ramp up in a linear fashion but in an exponential way.

HC: This is all so exciting and I really can’t wait to see how the finished project came out. Thanks for sitting down and chatting with me about it! Any last words before this gets revealed?

SW: I want to give a shoutout to Aeroluna – thank you for making Chroma! Shout out to Caeden117 – thank you for making ChroMapper, and also helping to fix the gradient system in ChroMapper 0.6 Beta. if it weren’t for you fixing that this map would take probably another week to finish. Shout out to Uialeth and Tachrayonic for helping get us shaders that would work on this special project. Shout out to my girlfriend for supporting me, and also to the mappers… me and Raiden did design this with the intention of it being possible for other mappers to use the platform without too much effort so we’re going to give resources to other mappers so that they can build upon this platform.

TVR: If anyone wants to learn more about advanced mixed reality techniques, I’m releasing a tutorial series called “Mixed Reality Masterclass” that will be available on YouTube soon at https://www.youtube.com/tougevr

You can find more from these talented members of the community here:

Watch this space for ModelSaber and Map links later this week so you, too, can play this incredible experience!

Comments (5)
  1. MoncimooV says:

    cant wait to experience it, OMG….

    (1)
    1. helencarnate says:

      Map and platform should be available on the weekend. I’ll add links to this post and also update my announcement in BSMG.

      (4)
    2. Songs about great songs

      (0)
  2. Thomas says:

    Love the song

    (0)

Leave a Reply

Your email address will not be published. Required fields are marked *