SIGGRAPH Asia 2026

X-Splat: Unified Gaussian Splatting for
Joint RGB and X-ray 3D Reconstruction

Wangze Xu1, Chu Zhou2, Mae Yamaguchi1, Linrui Dai1, I-Chao Shen1, Takashi Ijiri3, Imari Sato1,2

1The University of Tokyo   2National Institute of Informatics   3Shibaura Institute of Technology

One model, one camera, both modalities. Three scenes play the same shot sequence in lockstep: RGB, then X-ray revealed through a moving window, a sweep from RGB to X-ray, a sweep on to the overlay, and finally the Gaussian primitives themselves.

Abstract

X-Splat reconstructs an object's outer appearance and its internal structure in a single set of 3D Gaussians. RGB photographs and X-ray projections see the same object through completely different physics — surface reflectance versus density line integrals — and are captured in unrelated coordinate frames. X-Splat ties them together with a geometry-based cross-modal alignment and a surface set that lets appearance live only where it can be observed, so the joint model stays as compact as a single-modality one.

Jointly reconstructing RGB and X-ray provides new insights by connecting external appearance with internal structure, enabling a more complete understanding of real-world objects than single modality. Yet existing methods train separate models per modality and suffer from redundant geometry, alignment artifacts, and loose coupling that limits cross-modal insight discovery. We propose a unified representation in which all Gaussian primitives share a single set of geometric parameters, with only surface-proximal primitives additionally carrying appearance attributes. We perform cross-modal registration through clean multi-view stereo and X-ray geometric proxies instead of aligning two sets of optimized Gaussians, improving alignment stability. We then jointly optimize the unified representation via supervision from both modalities, complemented by a surface-depth consistency loss derived from X-ray geometry to guide RGB reconstruction. Experiments on the InsideOut dataset and our Flower & Insect dataset show that our framework achieves more robust cross-modal alignment, higher RGB rendering quality without compromising X-ray reconstruction, and a more compact representation than maintaining two independent geometries.

Drag to see inside

Both sides are the same Gaussians at the same viewpoint, rendered twice — once as reflected light, once as absorbed X-rays. Drag the handle, or use the arrow keys.

RGB X-ray

Method

X-Splat teaser: unified Gaussians splatted to RGB and X-ray, compared against InsideOut, 3DGS and R-squared Gaussian
One set of Gaussian primitives carries both reflective (RGB) and absorptive (X-ray) attributes, capturing external appearance and internal structure at once. Single-modality specialists (3DGS, R2-Gaussian) cover only one axis; the prior unified method InsideOut† covers both, with lower RGB fidelity in our reproduction. Each Ours/InsideOut† panel shows three vertical strips at the same viewpoint: pure RGB, overlay, pure X-ray. († our own re-implementation of Lee et al., ICCV 2025 — the official code and preprocessing are not public, and that re-implementation is not part of the code release.)
Framework overview: X-ray projections to FDK volume to coarse X-ray 3DGS, initialising a unified Gaussian set optimised under three losses
Framework overview. A coarse X-ray reconstruction anchors cross-modal alignment and initialises the unified Gaussian set, in which each primitive carries shared geometry {μ, R, S} and an X-ray attribute ρ, with RGB appearance {α, c} restricted to a surface subset. The unified set is jointly optimized under three losses: X-ray photometric, RGB photometric, and a CT-guided depth loss.

Training runs as one command over six stages: data preparation (COLMAP poses, masks, cone-beam projections, an MVS point cloud), 3,000 iterations of X-ray-only pre-training that yield the density field and CT surface, a PCA+ICP realignment that brings the RGB frame into the X-ray one, then 12,000 iterations of joint optimization over the shared Gaussians with the surface set, CT-guided depth supervision and virtual geometry views — followed by rendering and evaluation. Each stage will be its own entry point in the code release.

Code and data

Training, preprocessing and evaluation code, the insect capture and our trained models will be released at github.com/zezeaaa/X-Splat, which will also carry the download links and the preparation route for each dataset.

Three datasets appear in the paper and we can only redistribute one of them:

BibTeX

@inproceedings{xu2026xsplat,
  title     = {X-Splat: Unified Gaussian Splatting for Joint RGB and X-ray 3D Reconstruction},
  author    = {Xu, Wangze and Zhou, Chu and Yamaguchi, Mae and Dai, Linrui and
               Shen, I-Chao and Ijiri, Takashi and Sato, Imari},
  booktitle = {SIGGRAPH Asia 2026 Conference Papers},
  year      = {2026}
}

Please also cite the work this builds on and the datasets you use — R2-Gaussian, 3D Gaussian Splatting, InsideOut (Lee et al. 2025), and the RIKEN flower CT library.