Decoupling shape from texture rescues 3D fine-tuning

Fine-tuning 3D generative models usually breaks down at the boundary between geometry and surface materials. In a guide for adapting TRELLIS.2, Hugging Face engineers Suvaditya Mukherjee, Merve Noyan, and Pedro Cuenca isolate why end-to-end training fails on small asset collections. When they tested the base model on 188 footwear scans from the Google Scanned Objects dataset, it produced plausible shoe bodies while consistently botching the fine geometry of the soles.

The standard instinct is to train the full pipeline, but common 3D datasets make that counterproductive. Most real-world scans supply diffuse color textures with baked-in lighting rather than clean material channels for roughness and metallicity. Updating the model's texture flow diffusion transformer on that data causes catastrophic forgetting, teaching the model to bake shadows directly into object surfaces. The team kept the material stage frozen and placed low-rank adapters only on the two structural stages: a rank 8 adapter on the dense sparse-structure transformer that creates the coarse occupancy grid, and a rank 16 adapter on the sparse shape transformer that resolves surface voxels.

That configuration trains only 9.3 million parameters across both transformers, letting a tiny domain dataset correct structural defects without corrupting lighting priors. For robotics and simulation teams generating specialized assets, decoupling geometric repair from material rendering is the only reliable path forward. When your downstream pipeline relies on physically based rendering, keeping material properties clean matters as much as fixing the mesh.