Full Body Avatar | Memory Optimizations
How we fit more limbs into the same amount of memory!
June 26, 2024
Fitting more limbs into the same amount of memoryWhile we were developing the full body avatar, we recognized that our crash rate due to memory for all of Rec Room, especially on mobile devices, had gotten really bad. Every developer at the company was tasked with making memory optimization their top priority. One of the challenges that made it difficult to ship full body avatars within our original timeline is that we decided we needed to get to a place where the full body avatar, with all of their extra limbs, animation systems, etc., had to use the same (or less) memory as the floating bean. We called this “getting to memory neutral.”Here’s where we were when we started the optimization work.Our benchmark floating bean avatar had 6,095 vertices (which define the geometry), and cost about 4 MB per avatar (which includes all 3 levels of detail meshes). The floating bean also had a total of 18 bones and 4 weights per vertex.
This is the benchmark avatar we used for comparison of our bean geometry cost vs our full body cost
The equivalent full body avatar (which added default pants and shoes) had 11,384 vertices, and cost about 8 MB per avatar before optimization. The full body skeleton currently contains 102 active bones with 4 bone weights per vertex.We needed to find a way to cut our full body avatar cost in half in order to hit our goal of being “memory neutral.”We got there with 3 key innovations.
In the right image, you can see skin culling reduces the polygons under the clothes compared to the left image. This technique let us remove 1k to 2k vertices on average.
Memory Optimization Innovation #1: Skin Culling Masks
For our first innovation, we built skin culling masks. This technique allows us to throw away some of those expensive vertices in memory by not building parts of the avatar’s skin that are not visible based on what they’re wearing. This means that for every clothing item our artists and creators will author, they can select from a set of existing “skin culling masks” to inform the avatar system which skin faces are hidden when an item is worn.Whenever you change what you’re wearing, we look through the set of items you’ve equipped and determine which geometry faces can be completely discarded. This comes with the added benefit of avoiding clipping bugs by just removing the problematic underlying skin!You’ve probably noticed during our beta that when authoring our huge catalog of avatar items, some culling masks weren’t set correctly, or needed to be adjusted. While we fixed many neck and wrist gaps, missing fingers and hands there’s still work to do. We are continuing to clean this up during our polish work.
Memory Optimization Innovation #2: MeshData, and Vertex Data Compression
For our second innovation, we re-wrote how the avatar mesh was built using a new primitive called the MeshData feature. This dramatically sped up how our avatars are built, and removed an annoying frame hitch that sometimes happened.Before, you’d probably notice when a player joins a room there can be a few frames of lag, especially on lower-powered devices like the Quest 2. That’s no longer the case with full body avatars.Another nice thing about the MeshData feature is that it allowed us to compress the data per vertex so we could manipulate how much memory that bone weights and UVs took up. With this optimization, we saved about 30% of our total memory.
Here's a code snapshot showing how we changed our vertex data layout to save memory by using half and byte data types instead of floats and integers, cutting our unvarying data by 60%.
Memory Optimization Innovation #3: Just-in-Time Level of Detail
For our third and final innovation, we developed a technique we call just-in-time level of detail. The classic floating bean system builds all three level of detail meshes whenever a new avatar appears and holds on to them in memory. We then only show one of these meshes at a time based on how far that player is from your camera.
Showing the different level of details that an avatar may have.
Because of the work we did to use the MeshData type above, we could explore building a dynamic avatar that builds the avatar when the level of detail should change without experiencing those frame lags. So now we only pay the memory for what we display. This saved us another 40% of memory on average.This trades off some compute time for memory savings, but we found across all of our platforms that thanks to Unity’s MeshData and multithreaded Jobs, it does not impact FPS!So in total, we were able to reduce the cost of the full body avatar by around 60~70%, more than the 50% we needed. This is why we say that the full body avatar is more optimized than the floating bean.
Memory problem solved! What about the frame rate?
Thanks to Unity’s animation system, we found that all of our platforms could pretty smoothly handle the complexity of the extra limbs. Where we did add some considerable computational expense is in the full body inverse kinematic (IK) solver we use.Full body IK helps with the believable movements of the body, arms, and legs on screens and in VR while only tracking or animating your head and hands. With enough players in a room, lower-end platforms started to show bad FPS. So instead of updating the full body IK solver for every player on every frame, we built a system to dynamically update player animations up to a certain amount per frame, prioritizing those avatars closest to the player or those that haven't been updated in a few frames.This priority-sorted technique ensures that you always get a good FPS, but at the cost of some players’ movement stuttering a bit, especially when they are far away from your camera. This felt like the right trade-off for us.