"First, let me explain about 4D facial scan.
4D facial scan is a technology that performs 3D facial scans over time to scan moving facial expressions.
It's also called dynamic 3D facial scan or volumetric scan for facial.

4D facial scanning captures time-varying 3D geometry. It captures both spatial facial structure and temporal dynamics of facial performance.
This technology is essential for productions requiring high-quality digital humans such as movie, game, and photo realistic digital double productions.
The video shows examples of 4D facial scan sequences. You can see how facial expressions change over time."


"Now, let me explain our motivation. In 4D facial scanning, we often need to edit the 4D mesh to fix problems. For example, wrong eye blinks, strange mouth shapes, or general mesh artifacts.
To edit 4D facial meshes, the standard approach uses head stabilization. This means we align all frames to a reference pose for consistent editing.
The video shows the difference. On the left is the original 4D scan sequence. On the right is the head stabilized sequence. You can see how the natural head motion is removed."


You can see, this process removes head movement.
However, here's the problem: natural head movements like nods, emphasis, and expressive head tilts are lost during stabilization. Actor's natural head movements are essential for conveying emotion and intent.


"So, our objectives are to preserve original head transformations while enabling precise local facial expression editing. We also need to maintain temporal consistency near the editing frame.
The technical challenges are: How do we apply consistent edits across frames with different head poses? How do we adapt artist-defined fix shapes to various poses? And how do we ensure natural temporal continuity after editing?"

"Let me show you our system overview. Our method has three main stages.


First, 4D data acquisition and preprocessing stage. In this stage, we will explain how we built our custom 4D data acquisition system, how we compressed high-volume facial 4D scan data, and how we prepared facial 4D scan data to production-ready quality.


Second, facial fix-shape editing. In this stage, we will explain how artists perform transform-invariant facial mesh editing to fix 4D scan facial artifacts.


Third, temporal smoothing. In this stage, we will explain how we spread the edits to nearby frames to create smooth transitions across the sequence.


"Now, we will introduce our 4D facial scan data acquisition system. We use 8 industrial machine vision cameras, each with 6 megapixels. We capture images at 24 frames per second. All cameras and lighting are synchronized with millisecond precision.

The data volume is quite large. Each camera generates 9MB image per frame. So we get 1.72 gigabytes per second. For a 120-frame sequence, that's about 200 gigabytes.
The left image shows our custom scanning rig. The right images show synchronized multi-view capture images from different angles."


"For post-processing, we use Google Draco mesh compression to reduce computation load. With Google Draco, we can create meshes with identical quality using only 1% of the original file size. The left images show the original mesh and Google Draco compressed mesh.

Also, we perform mesh registration using template mesh to get canonical mesh sequences with consistent topology. This process is essential for texture work and facial rigging in production. The right images show our template mesh and the mesh registration results."

"These slides explain how we generate a full head 4D facial mesh using only facial area 4D mesh data. Machine vision cameras are expensive. As the number of cameras increases, we need to process much larger amounts of data. So building a system to scan the entire head has limitations.
Therefore, we scanned only the facial area with 8 machine vision cameras. Then we seamlessly connected this to the back head mesh. To do this, we blended the connection areas so the back head mesh moves naturally according to the facial scan data movement. The images show this process."

"This is our 4D scan data result from the frontal view. The left image shows the scanned video, the center image shows the original 4D scan data, and the right image shows the result organized with canonical template mesh.


Here is our 4D scan data result from the side view.


Next stage is facial fix-shape editing stage. In this stage, artists can correct various errors in 4D scan data, such as eye-close problems. Artists can edit data on a consistent frontal face model regardless of head movement. Then they can apply these edits to the 4D mesh.
To do this, we specify 6 to 8 facial landmarks. The red dots show the facial landmarks we use for alignment. These include eye corners, nose tip, and other stable facial features.
We compute rigid transform Tf between the 4D facial sequence and the artist's fix-shape. Through this, the artist's edits on the frontal face are automatically applied to 4D scan data with head movement.


"The final stage is temporal smoothing. In this stage, we perform temporal smoothing so the frames edited by the artist connect smoothly with the frames before and after. To do this, we perform weighted linear interpolation on a certain number of frames before and after the edited frame. We experimentally confirmed that for 24fps sequences, when the window size k is 5 frames before and after, we get visually acceptable results.

Here you can see the original sequence on top and the edited sequence on the bottom. The red box shows the target frame where we applied the edit. The edit is smoothly propagated to neighboring frames to avoid any sudden changes."

"Here's our result. On the left is the original mesh with the problem. On the right is our edited result. As you can see, we successfully corrected the facial expression while preserving the head pose and natural appearance."


"Let me discuss our limitations. First, manual fix shape creation is time-consuming and requires artistic expertise. This limits scalability. Second, our evaluation is qualitative only. We don't have quantitative metrics or user studies.
For future directions, we plan to automate fix shape creation using example-based synthesis from curated facial prior libraries. We also want to add quantitative evaluation criteria and extend to more diverse editing scenarios.
In conclusion, we present transform-invariant facial expression editing for 4D mesh sequences. Our method preserves original head transforms while enabling precise expression modification. It maintains temporal coherence and pose consistency.
The practical impact is significant. We can fix localized expression errors without disrupting natural head motion. This is valuable for facial retargeting, VFX, and performance-driven animation."