SlideShare una empresa de Scribd logo
1 de 24
Descargar para leer sin conexión
Neural Radiance Fields
Neural Fields in Computer Vision
Field
스칼라장 : 온도, 압력, 오디오 등 벡터장 : 유체속도장, 자기장 등 텐서장 : 영상 등
Neural Field
3D Synthesize : COLMAP - Structure-from-Motion and Multi-View Stereo
https://colmap.github.io/
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://neuralfields.cs.brown.edu/cvpr22
https://keras.io/examples/vision/nerf/
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf
https://github.com/NakuraMino/Minimal-NeRF
1. MLP 기반의 NeRF
* input의 Spherical Vector(2D)를 Cartesian(3D)
으로 변경
* input의 viewing direction : sigma, pi
(Cartesian)가 3*2*L_embed로 확장 ( positional
encoding )
* output은 RGBa
(a는 volume density)
=> !!중요한점, 1 view는 1 pixel에 대응
8 Depth FC (with 256 dim - Relu)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py
MLP 기반의 NeRF (Dataset from Blender)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py
https://docs.blender.org/api/blender_python_api_2_71_release/
https://github.com/bmild/nerf/issues/78
https://www.3dgep.com/understanding-the-view-matrix/#:~:text=The%20world%20transformation%20matrix%20is,%2Dspace%20to%20view%2Dspace.
MLP 기반의 NeRF (Dataset from Blender)
camera_angle_x : Camera lens horizontal field of view <- 카메라의 Focal 값
(모두 동일값)
rotation : radians (degree) <- 카메라 제자리 회전 값… (모두 동일값)
transform_matrix (Worldspace transformation matrix, matrix_world (blender))
<- Camera 월 좌표계 및 상대 Rotation 정보
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender)
transform_matrix (Look-At 행렬) - Camera 2 World
focal = .5 * W / np.tan(.5 * camera_angle_x)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender) -
rays_o는 rays_original (position) : 각 HxWx3 pixel의 original ray point를 의미 == 모든 점이
camera x,y,z로 똑같음 (각 픽셀마다의 camera position)
rays_d는 direction: dirs는 HxW내에서의 상대 좌표. c2w를 곱해줌으로서 camera viewing
direction(+depth)을 계산 (각 픽셀마다의 camera view direction)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
http://sgvr.kaist.ac.kr/~sungeui/GCG_S22/Student_Presentations/CS580_PaperPresentation_dgkim.pdf
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function
MLP 기반의 NeRF (Dataset from Blender) -
z_vals <- Random Accumulate Point 지정
pts <- z_vals위치의 point들을 생성
pts input에 대한 network output 값의 합산이 rgb가
될 수 있도록 함
(weights 값 합산은 near 1)
=> raw[:3]는 rgb_map과 mse, raw[4]는 직접 mse가
없고 raw[:3]와 곱해져 간접 업데이트
(본 코드에는 raw[4] - a는 업데이트 없음 fix z_vals)
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://m.blog.naver.com/jetarrow82/221252045857
1. MLP 기반의 NeRF - Viewing Direction의 중요성
실제 물체는 non-Lambertian effects 특성을 가짐
Lambertian Surface는 ViewPoint에 상관없이
똑같은 빛의 양을 반사함
NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis
(ECCV 2020)
https://www.matthewtancik.com/nerf
1. MLP 기반의 NeRF - Positional Encoding
NeRF Result
https://phog.github.io/snerg/viewer/index.html?dir=https://storage.googleapis.com/snerg/750/lego
<= 같이 보자!
NeRF: Synthesizer로서의 가치
Small Size Model Rendering 도메인이 한정되어 있지 않음
Texture
https://learnopengl.com/Advanced-OpenGL/Cubemaps
https://lifeisforu.tistory.com/375
CubeMap - 보통 ray
영향을 크게 받지 않는
원경에 사용
Spherical Harmonics( SH, 구면
조화 ) - irradiance map(난반사
정도)을 저장하는 효율적 방식
Plenoxels : ReRF without Neural Networks
NeRF key point is differentiable Volume Renderer
MLP Volume Rendering을 Voxel Grid와 Interpolation의 조합으로 치환
Beyond Synthesize
- iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020)
Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS)
https://yenchenlin.me/inerf/
Trained Nerf를 통해
Generation하고 Pose Estimation을
Iterative하게 Gradient Based
Optimization을 하여 Estimated
Pose를 취득
Beyond Synthesize
- iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020)
Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS)
https://yenchenlin.me/inerf/
Beyond Synthesize
- Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA)
https://yenchenlin.me/nerf-supervision/
Keypoint detection과 유사.
Keypoint는 sparse하게 pose
정보를 추출한다면, Dense
Descriptor는 Dense하게 Pose
정보를 추출
Beyond Synthesize
- Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA)
https://yenchenlin.me/nerf-supervision/
Beyond Synthesize
- NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES (2022 Pre)
- 셀링포인트는 다양한
각도에서 영상을 촬영하면
NeRF 샘플에서
Unsupervised로
Segmentation을 만들어줌
=> 한계점
- K-Means/Pre-trained
Network에 의존적 (BG가
명확하게 분리되고, Object의
Semantic 정보가 명확해야 할
것으로 보임)
NeRFs…
https://kakaobrain.github.io/NeRF-Factory/ <= Nerf 성능 비교
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs <= 넓은 공간 모델링
Instant Neural Graphics Primitive
<= NeRF, SDF(signed distance functions), Gigapixel image등 (Computer graphics primitives)을 처리할 수
있는 네트워크
https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4
Dex-NeRF : Using a Neural Radiance Field to Grasp Transparent Objects
- Transparent, Light-sensitive한 물체를 잘 다룰 수 있지 않을까?
https://sites.google.com/view/dex-nerf
vision-only robot navigation in a neural radiance world - Vision Based Planning
https://mikh3x4.github.io/nerf-navigation/
Relighting the Real World With Neural Rendering -
https://www.unite.ai/adobe-neural-rendering-relighting-2021/
NeRF-Studio
https://docs.nerf.studio/en/latest/index.html

Más contenido relacionado

La actualidad más candente

Lec14 multiview stereo
Lec14 multiview stereoLec14 multiview stereo
Lec14 multiview stereoBaliThorat1
 
Image segmentation with deep learning
Image segmentation with deep learningImage segmentation with deep learning
Image segmentation with deep learningAntonio Rueda-Toicen
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)Fellowship at Vodafone FutureLab
 
GANs and Applications
GANs and ApplicationsGANs and Applications
GANs and ApplicationsHoang Nguyen
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning健程 杨
 
CNN and its applications by ketaki
CNN and its applications by ketakiCNN and its applications by ketaki
CNN and its applications by ketakiKetaki Patwari
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Jason Tsai
 
Image classification using cnn
Image classification using cnnImage classification using cnn
Image classification using cnnDebarko De
 
Introduction to Generative Adversarial Networks (GANs)
Introduction to Generative Adversarial Networks (GANs)Introduction to Generative Adversarial Networks (GANs)
Introduction to Generative Adversarial Networks (GANs)Appsilon Data Science
 
Simultaneous Smoothing and Sharpening of Color Images
Simultaneous Smoothing and Sharpening of Color ImagesSimultaneous Smoothing and Sharpening of Color Images
Simultaneous Smoothing and Sharpening of Color ImagesCristina Pérez Benito
 
Diffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesisDiffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesisBeerenSahu
 
Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNNNoura Hussein
 
Generative Adversarial Networks (GAN)
Generative Adversarial Networks (GAN)Generative Adversarial Networks (GAN)
Generative Adversarial Networks (GAN)Manohar Mukku
 
Color image processing Presentation
Color image processing PresentationColor image processing Presentation
Color image processing PresentationRevanth Chimmani
 
Depth estimation do we need to throw old things away
Depth estimation do we need to throw old things awayDepth estimation do we need to throw old things away
Depth estimation do we need to throw old things awayNAVER Engineering
 
Computer Vision: Shape from Specularities and Motion
Computer Vision: Shape from Specularities and MotionComputer Vision: Shape from Specularities and Motion
Computer Vision: Shape from Specularities and MotionDamian T. Gordon
 
Chapter 9 morphological image processing
Chapter 9   morphological image processingChapter 9   morphological image processing
Chapter 9 morphological image processingAhmed Daoud
 

La actualidad más candente (20)

Lec14 multiview stereo
Lec14 multiview stereoLec14 multiview stereo
Lec14 multiview stereo
 
Image segmentation with deep learning
Image segmentation with deep learningImage segmentation with deep learning
Image segmentation with deep learning
 
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
AlexNet(ImageNet Classification with Deep Convolutional Neural Networks)
 
GANs and Applications
GANs and ApplicationsGANs and Applications
GANs and Applications
 
Attention in Deep Learning
Attention in Deep LearningAttention in Deep Learning
Attention in Deep Learning
 
CNN and its applications by ketaki
CNN and its applications by ketakiCNN and its applications by ketaki
CNN and its applications by ketaki
 
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
Attention Models (D3L6 2017 UPC Deep Learning for Computer Vision)
 
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
Introduction to Spiking Neural Networks: From a Computational Neuroscience pe...
 
Image classification using cnn
Image classification using cnnImage classification using cnn
Image classification using cnn
 
Introduction to Generative Adversarial Networks (GANs)
Introduction to Generative Adversarial Networks (GANs)Introduction to Generative Adversarial Networks (GANs)
Introduction to Generative Adversarial Networks (GANs)
 
Simultaneous Smoothing and Sharpening of Color Images
Simultaneous Smoothing and Sharpening of Color ImagesSimultaneous Smoothing and Sharpening of Color Images
Simultaneous Smoothing and Sharpening of Color Images
 
Diffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesisDiffusion models beat gans on image synthesis
Diffusion models beat gans on image synthesis
 
Deep Learning for Video: Action Recognition (UPC 2018)
Deep Learning for Video: Action Recognition (UPC 2018)Deep Learning for Video: Action Recognition (UPC 2018)
Deep Learning for Video: Action Recognition (UPC 2018)
 
Image classification using CNN
Image classification using CNNImage classification using CNN
Image classification using CNN
 
Generative Adversarial Networks (GAN)
Generative Adversarial Networks (GAN)Generative Adversarial Networks (GAN)
Generative Adversarial Networks (GAN)
 
Color image processing Presentation
Color image processing PresentationColor image processing Presentation
Color image processing Presentation
 
Depth estimation do we need to throw old things away
Depth estimation do we need to throw old things awayDepth estimation do we need to throw old things away
Depth estimation do we need to throw old things away
 
Computer Vision: Shape from Specularities and Motion
Computer Vision: Shape from Specularities and MotionComputer Vision: Shape from Specularities and Motion
Computer Vision: Shape from Specularities and Motion
 
AlexNet
AlexNetAlexNet
AlexNet
 
Chapter 9 morphological image processing
Chapter 9   morphological image processingChapter 9   morphological image processing
Chapter 9 morphological image processing
 

Similar a Neural Radiance Field

Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...
Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...
Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...inside-BigData.com
 
Deferred Pixel Shading on the PLAYSTATION®3
Deferred Pixel Shading on the PLAYSTATION®3Deferred Pixel Shading on the PLAYSTATION®3
Deferred Pixel Shading on the PLAYSTATION®3Slide_N
 
Parallel wisard object tracker a rambased tracking system
Parallel wisard object tracker a rambased tracking systemParallel wisard object tracker a rambased tracking system
Parallel wisard object tracker a rambased tracking systemcseij
 
Improve real-time hair rendering.
Improve real-time hair rendering.Improve real-time hair rendering.
Improve real-time hair rendering.Valentin Janiaut
 
PCA-SIFT: A More Distinctive Representation for Local Image Descriptors
PCA-SIFT: A More Distinctive Representation for Local Image DescriptorsPCA-SIFT: A More Distinctive Representation for Local Image Descriptors
PCA-SIFT: A More Distinctive Representation for Local Image Descriptorswolf
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationGiorgio Carbone
 
Depth Fusion from RGB and Depth Sensors IV
Depth Fusion from RGB and Depth Sensors  IVDepth Fusion from RGB and Depth Sensors  IV
Depth Fusion from RGB and Depth Sensors IVYu Huang
 
An Assessment of Image Matching Algorithms in Depth Estimation
An Assessment of Image Matching Algorithms in Depth EstimationAn Assessment of Image Matching Algorithms in Depth Estimation
An Assessment of Image Matching Algorithms in Depth EstimationCSCJournals
 
Review On Different Feature Extraction Algorithms
Review On Different Feature Extraction AlgorithmsReview On Different Feature Extraction Algorithms
Review On Different Feature Extraction AlgorithmsIRJET Journal
 
Pengantar Structure from Motion Photogrammetry
Pengantar Structure from Motion PhotogrammetryPengantar Structure from Motion Photogrammetry
Pengantar Structure from Motion PhotogrammetryDany Laksono
 
Presentation_ Stable View Synthesis.pptx
Presentation_ Stable View Synthesis.pptxPresentation_ Stable View Synthesis.pptx
Presentation_ Stable View Synthesis.pptxdaptivuse
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsDataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsPetteriTeikariPhD
 
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video  Processing (NeRF...[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video  Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...Deep Learning JP
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic SegmentationSemantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation岳華 杜
 
CUDA by Example : Constant Memory and Events : Notes
CUDA by Example : Constant Memory and Events : NotesCUDA by Example : Constant Memory and Events : Notes
CUDA by Example : Constant Memory and Events : NotesSubhajit Sahu
 

Similar a Neural Radiance Field (20)

TransNeRF
TransNeRFTransNeRF
TransNeRF
 
Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...
Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...
Scratch to Supercomputers: Bottoms-up Build of Large-scale Computational Lens...
 
Nadia2013 research
Nadia2013 researchNadia2013 research
Nadia2013 research
 
Deferred Pixel Shading on the PLAYSTATION®3
Deferred Pixel Shading on the PLAYSTATION®3Deferred Pixel Shading on the PLAYSTATION®3
Deferred Pixel Shading on the PLAYSTATION®3
 
Parallel wisard object tracker a rambased tracking system
Parallel wisard object tracker a rambased tracking systemParallel wisard object tracker a rambased tracking system
Parallel wisard object tracker a rambased tracking system
 
Improve real-time hair rendering.
Improve real-time hair rendering.Improve real-time hair rendering.
Improve real-time hair rendering.
 
PCA-SIFT: A More Distinctive Representation for Local Image Descriptors
PCA-SIFT: A More Distinctive Representation for Local Image DescriptorsPCA-SIFT: A More Distinctive Representation for Local Image Descriptors
PCA-SIFT: A More Distinctive Representation for Local Image Descriptors
 
Master's Thesis - Data Science - Presentation
Master's Thesis - Data Science - PresentationMaster's Thesis - Data Science - Presentation
Master's Thesis - Data Science - Presentation
 
DICTA 2017 poster
DICTA 2017 posterDICTA 2017 poster
DICTA 2017 poster
 
Depth Fusion from RGB and Depth Sensors IV
Depth Fusion from RGB and Depth Sensors  IVDepth Fusion from RGB and Depth Sensors  IV
Depth Fusion from RGB and Depth Sensors IV
 
An Assessment of Image Matching Algorithms in Depth Estimation
An Assessment of Image Matching Algorithms in Depth EstimationAn Assessment of Image Matching Algorithms in Depth Estimation
An Assessment of Image Matching Algorithms in Depth Estimation
 
Review On Different Feature Extraction Algorithms
Review On Different Feature Extraction AlgorithmsReview On Different Feature Extraction Algorithms
Review On Different Feature Extraction Algorithms
 
BDL_project_report
BDL_project_reportBDL_project_report
BDL_project_report
 
Pengantar Structure from Motion Photogrammetry
Pengantar Structure from Motion PhotogrammetryPengantar Structure from Motion Photogrammetry
Pengantar Structure from Motion Photogrammetry
 
Presentation_ Stable View Synthesis.pptx
Presentation_ Stable View Synthesis.pptxPresentation_ Stable View Synthesis.pptx
Presentation_ Stable View Synthesis.pptx
 
Dataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problemsDataset creation for Deep Learning-based Geometric Computer Vision problems
Dataset creation for Deep Learning-based Geometric Computer Vision problems
 
HS Demo
HS DemoHS Demo
HS Demo
 
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video  Processing (NeRF...[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video  Processing (NeRF...
[DL輪読会]Neural Radiance Flow for 4D View Synthesis and Video Processing (NeRF...
 
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic SegmentationSemantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
Semantic Segmentation - Fully Convolutional Networks for Semantic Segmentation
 
CUDA by Example : Constant Memory and Events : Notes
CUDA by Example : Constant Memory and Events : NotesCUDA by Example : Constant Memory and Events : Notes
CUDA by Example : Constant Memory and Events : Notes
 

Más de Dong Heon Cho

Forward-Forward Algorithm
Forward-Forward AlgorithmForward-Forward Algorithm
Forward-Forward AlgorithmDong Heon Cho
 
2020 > Self supervised learning
2020 > Self supervised learning2020 > Self supervised learning
2020 > Self supervised learningDong Heon Cho
 
All about that pooling
All about that poolingAll about that pooling
All about that poolingDong Heon Cho
 
Background elimination review
Background elimination reviewBackground elimination review
Background elimination reviewDong Heon Cho
 
Transparent Latent GAN
Transparent Latent GANTransparent Latent GAN
Transparent Latent GANDong Heon Cho
 
Multi object Deep reinforcement learning
Multi object Deep reinforcement learningMulti object Deep reinforcement learning
Multi object Deep reinforcement learningDong Heon Cho
 
Multi agent reinforcement learning for sequential social dilemmas
Multi agent reinforcement learning for sequential social dilemmasMulti agent reinforcement learning for sequential social dilemmas
Multi agent reinforcement learning for sequential social dilemmasDong Heon Cho
 
Hybrid reward architecture
Hybrid reward architectureHybrid reward architecture
Hybrid reward architectureDong Heon Cho
 
Use Jupyter notebook guide in 5 minutes
Use Jupyter notebook guide in 5 minutesUse Jupyter notebook guide in 5 minutes
Use Jupyter notebook guide in 5 minutesDong Heon Cho
 
AlexNet and so on...
AlexNet and so on...AlexNet and so on...
AlexNet and so on...Dong Heon Cho
 
Deep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDeep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDong Heon Cho
 
How can we train with few data
How can we train with few dataHow can we train with few data
How can we train with few dataDong Heon Cho
 
Domain adaptation gan
Domain adaptation ganDomain adaptation gan
Domain adaptation ganDong Heon Cho
 
Dense sparse-dense training for dnn and Other Models
Dense sparse-dense training for dnn and Other ModelsDense sparse-dense training for dnn and Other Models
Dense sparse-dense training for dnn and Other ModelsDong Heon Cho
 

Más de Dong Heon Cho (20)

Forward-Forward Algorithm
Forward-Forward AlgorithmForward-Forward Algorithm
Forward-Forward Algorithm
 
What is Texture.pdf
What is Texture.pdfWhat is Texture.pdf
What is Texture.pdf
 
BADGE
BADGEBADGE
BADGE
 
2020 > Self supervised learning
2020 > Self supervised learning2020 > Self supervised learning
2020 > Self supervised learning
 
All about that pooling
All about that poolingAll about that pooling
All about that pooling
 
Background elimination review
Background elimination reviewBackground elimination review
Background elimination review
 
Transparent Latent GAN
Transparent Latent GANTransparent Latent GAN
Transparent Latent GAN
 
Image matting atoc
Image matting atocImage matting atoc
Image matting atoc
 
Multi object Deep reinforcement learning
Multi object Deep reinforcement learningMulti object Deep reinforcement learning
Multi object Deep reinforcement learning
 
Multi agent reinforcement learning for sequential social dilemmas
Multi agent reinforcement learning for sequential social dilemmasMulti agent reinforcement learning for sequential social dilemmas
Multi agent reinforcement learning for sequential social dilemmas
 
Multi agent System
Multi agent SystemMulti agent System
Multi agent System
 
Hybrid reward architecture
Hybrid reward architectureHybrid reward architecture
Hybrid reward architecture
 
Use Jupyter notebook guide in 5 minutes
Use Jupyter notebook guide in 5 minutesUse Jupyter notebook guide in 5 minutes
Use Jupyter notebook guide in 5 minutes
 
AlexNet and so on...
AlexNet and so on...AlexNet and so on...
AlexNet and so on...
 
Deep Learning AtoC with Image Perspective
Deep Learning AtoC with Image PerspectiveDeep Learning AtoC with Image Perspective
Deep Learning AtoC with Image Perspective
 
LOL win prediction
LOL win predictionLOL win prediction
LOL win prediction
 
How can we train with few data
How can we train with few dataHow can we train with few data
How can we train with few data
 
Domain adaptation gan
Domain adaptation ganDomain adaptation gan
Domain adaptation gan
 
Dense sparse-dense training for dnn and Other Models
Dense sparse-dense training for dnn and Other ModelsDense sparse-dense training for dnn and Other Models
Dense sparse-dense training for dnn and Other Models
 
Squeeeze models
Squeeeze modelsSqueeeze models
Squeeeze models
 

Último

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...PrithaVashisht1
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionajayrajaganeshkayala
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxVenkatasubramani13
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxDwiAyuSitiHartinah
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityAggregage
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024Becky Burwell
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerPavel Šabatka
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructuresonikadigital1
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?sonikadigital1
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Guido X Jansen
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptaigil2
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Vladislav Solodkiy
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.JasonViviers2
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introductionsanjaymuralee1
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best PracticesDataArchiva
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)Data & Analytics Magazin
 

Último (16)

Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...Elements of language learning - an analysis of how different elements of lang...
Elements of language learning - an analysis of how different elements of lang...
 
CI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual interventionCI, CD -Tools to integrate without manual intervention
CI, CD -Tools to integrate without manual intervention
 
Mapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptxMapping the pubmed data under different suptopics using NLP.pptx
Mapping the pubmed data under different suptopics using NLP.pptx
 
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptxTINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
TINJUAN PEMROSESAN TRANSAKSI DAN ERP.pptx
 
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for ClarityStrategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
Strategic CX: A Deep Dive into Voice of the Customer Insights for Clarity
 
SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024SFBA Splunk Usergroup meeting March 13, 2024
SFBA Splunk Usergroup meeting March 13, 2024
 
The Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayerThe Universal GTM - how we design GTM and dataLayer
The Universal GTM - how we design GTM and dataLayer
 
ChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics InfrastructureChistaDATA Real-Time DATA Analytics Infrastructure
ChistaDATA Real-Time DATA Analytics Infrastructure
 
How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?How is Real-Time Analytics Different from Traditional OLAP?
How is Real-Time Analytics Different from Traditional OLAP?
 
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
Persuasive E-commerce, Our Biased Brain @ Bikkeldag 2024
 
MEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .pptMEASURES OF DISPERSION I BSc Botany .ppt
MEASURES OF DISPERSION I BSc Botany .ppt
 
Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023Cash Is Still King: ATM market research '2023
Cash Is Still King: ATM market research '2023
 
YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.YourView Panel Book.pptx YourView Panel Book.
YourView Panel Book.pptx YourView Panel Book.
 
Virtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product IntroductionVirtuosoft SmartSync Product Introduction
Virtuosoft SmartSync Product Introduction
 
5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices5 Ds to Define Data Archiving Best Practices
5 Ds to Define Data Archiving Best Practices
 
AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)AI for Sustainable Development Goals (SDGs)
AI for Sustainable Development Goals (SDGs)
 

Neural Radiance Field

  • 1. Neural Radiance Fields Neural Fields in Computer Vision
  • 2. Field 스칼라장 : 온도, 압력, 오디오 등 벡터장 : 유체속도장, 자기장 등 텐서장 : 영상 등
  • 4. 3D Synthesize : COLMAP - Structure-from-Motion and Multi-View Stereo https://colmap.github.io/
  • 5. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://neuralfields.cs.brown.edu/cvpr22 https://keras.io/examples/vision/nerf/
  • 6. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf https://github.com/NakuraMino/Minimal-NeRF 1. MLP 기반의 NeRF * input의 Spherical Vector(2D)를 Cartesian(3D) 으로 변경 * input의 viewing direction : sigma, pi (Cartesian)가 3*2*L_embed로 확장 ( positional encoding ) * output은 RGBa (a는 volume density) => !!중요한점, 1 view는 1 pixel에 대응 8 Depth FC (with 256 dim - Relu)
  • 7. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py MLP 기반의 NeRF (Dataset from Blender)
  • 8. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://github.com/bmild/nerf/blob/20a91e764a28816ee2234fcadb73bd59a613a44c/load_blender.py https://docs.blender.org/api/blender_python_api_2_71_release/ https://github.com/bmild/nerf/issues/78 https://www.3dgep.com/understanding-the-view-matrix/#:~:text=The%20world%20transformation%20matrix%20is,%2Dspace%20to%20view%2Dspace. MLP 기반의 NeRF (Dataset from Blender) camera_angle_x : Camera lens horizontal field of view <- 카메라의 Focal 값 (모두 동일값) rotation : radians (degree) <- 카메라 제자리 회전 값… (모두 동일값) transform_matrix (Worldspace transformation matrix, matrix_world (blender)) <- Camera 월 좌표계 및 상대 Rotation 정보
  • 9. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) transform_matrix (Look-At 행렬) - Camera 2 World focal = .5 * W / np.tan(.5 * camera_angle_x)
  • 10. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - rays_o는 rays_original (position) : 각 HxWx3 pixel의 original ray point를 의미 == 모든 점이 camera x,y,z로 똑같음 (각 픽셀마다의 camera position) rays_d는 direction: dirs는 HxW내에서의 상대 좌표. c2w를 곱해줌으로서 camera viewing direction(+depth)을 계산 (각 픽셀마다의 camera view direction)
  • 11. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) http://sgvr.kaist.ac.kr/~sungeui/GCG_S22/Student_Presentations/CS580_PaperPresentation_dgkim.pdf
  • 12. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.scratchapixel.com/lessons/mathematics-physics-for-computer-graphics/lookat-function MLP 기반의 NeRF (Dataset from Blender) - z_vals <- Random Accumulate Point 지정 pts <- z_vals위치의 point들을 생성 pts input에 대한 network output 값의 합산이 rgb가 될 수 있도록 함 (weights 값 합산은 near 1) => raw[:3]는 rgb_map과 mse, raw[4]는 직접 mse가 없고 raw[:3]와 곱해져 간접 업데이트 (본 코드에는 raw[4] - a는 업데이트 없음 fix z_vals)
  • 13. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://m.blog.naver.com/jetarrow82/221252045857 1. MLP 기반의 NeRF - Viewing Direction의 중요성 실제 물체는 non-Lambertian effects 특성을 가짐 Lambertian Surface는 ViewPoint에 상관없이 똑같은 빛의 양을 반사함
  • 14. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (ECCV 2020) https://www.matthewtancik.com/nerf 1. MLP 기반의 NeRF - Positional Encoding
  • 16. NeRF: Synthesizer로서의 가치 Small Size Model Rendering 도메인이 한정되어 있지 않음
  • 17. Texture https://learnopengl.com/Advanced-OpenGL/Cubemaps https://lifeisforu.tistory.com/375 CubeMap - 보통 ray 영향을 크게 받지 않는 원경에 사용 Spherical Harmonics( SH, 구면 조화 ) - irradiance map(난반사 정도)을 저장하는 효율적 방식
  • 18. Plenoxels : ReRF without Neural Networks NeRF key point is differentiable Volume Renderer MLP Volume Rendering을 Voxel Grid와 Interpolation의 조합으로 치환
  • 19. Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/ Trained Nerf를 통해 Generation하고 Pose Estimation을 Iterative하게 Gradient Based Optimization을 하여 Estimated Pose를 취득
  • 20. Beyond Synthesize - iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020) Neural Fields in Visual Computing and Beyond (2022 EUROGRPAHICS) https://yenchenlin.me/inerf/
  • 21. Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/ Keypoint detection과 유사. Keypoint는 sparse하게 pose 정보를 추출한다면, Dense Descriptor는 Dense하게 Pose 정보를 추출
  • 22. Beyond Synthesize - Learning Dense Object Descriptors from Neural Radiance Fields (2022 ICRA) https://yenchenlin.me/nerf-supervision/
  • 23. Beyond Synthesize - NERF-SOS: ANY-VIEW SELF-SUPERVISED OBJECT SEGMENTATION ON COMPLEX SCENES (2022 Pre) - 셀링포인트는 다양한 각도에서 영상을 촬영하면 NeRF 샘플에서 Unsupervised로 Segmentation을 만들어줌 => 한계점 - K-Means/Pre-trained Network에 의존적 (BG가 명확하게 분리되고, Object의 Semantic 정보가 명확해야 할 것으로 보임)
  • 24. NeRFs… https://kakaobrain.github.io/NeRF-Factory/ <= Nerf 성능 비교 Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs <= 넓은 공간 모델링 Instant Neural Graphics Primitive <= NeRF, SDF(signed distance functions), Gigapixel image등 (Computer graphics primitives)을 처리할 수 있는 네트워크 https://nvlabs.github.io/instant-ngp/assets/mueller2022instant.mp4 Dex-NeRF : Using a Neural Radiance Field to Grasp Transparent Objects - Transparent, Light-sensitive한 물체를 잘 다룰 수 있지 않을까? https://sites.google.com/view/dex-nerf vision-only robot navigation in a neural radiance world - Vision Based Planning https://mikh3x4.github.io/nerf-navigation/ Relighting the Real World With Neural Rendering - https://www.unite.ai/adobe-neural-rendering-relighting-2021/ NeRF-Studio https://docs.nerf.studio/en/latest/index.html