VRS-NeRF: Accelerating Neural Radiance Field Rendering with Variable Rate Shading

Rolff, Tim; Steinicke, Frank; Li, Ke; Schmidt, Susanne; Frintrop, Simone

doi:10.1109/ISMAR59233.2023.00039

Contribution to a conference proceedings/Contribution to a book

PUBDB-2023-07544

VRS-NeRF: Accelerating Neural Radiance Field Rendering with Variable Rate Shading

Rolff, T. (Corresponding author) ; Schmidt, S. ; Li, K.DESY* ; Steinicke, F. ; Frintrop, S.

2023
IEEE

2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) : [Proceedings] - IEEE, 2023. - ISBN 979-8-3503-2838-7 - doi:10.1109/ISMAR59233.2023.00039
2023 IEEE International Symposium on Mixed and Augmented Reality, ISMAR, Sydney, Australia, 16 Oct 2023 - 20 Oct 2023 IEEE 243 - 252 (2023) [10.1109/ISMAR59233.2023.00039]

This record in other databases:

Please use a persistent id in citations: doi:10.1109/ISMAR59233.2023.00039

Abstract: Recent advancements in Neural Radiance Fields (NeRF) provide enormous potential for a wide range of Mixed Reality (MR) applications. However, the applicability of NeRF to real-time MR systems is still largely limited by the rendering performance of NeRF. In this paper, we present a novel approach for Variable Rate Shading for Neural Radiance Fields (VRS-NeRF). In contrast to previous techniques, our approach does not require training multiple neural networks or re-training of already existing ones, but instead utilizes the raytracing properties of NeRF. This is achieved by merging rays depending on a variable shading rate, which reduces the overall number of queries to the neural network. We demonstrate the generalizability of our approach by implementing three alternative functions for the determination of the shading rate. The first method uses the gaze of users to effectively implement a foveated rendering technique in NeRF. For the other two techniques, we utilize shading rates based on edges and saliency. Based on a psychophysical experiment and multiple image-based metrics, we suggest a set of parameters for each technique, yielding an optimal tradeoff between rendering performance gain and perceived visual quality.

Contributing Institute(s):