1. The document discusses accelerating collapsed variational Bayesian inference for latent Dirichlet allocation (CVB) using Nvidia CUDA compatible GPU devices. 2. It describes parallelizing CVB for LDA by assigning different topics to different GPU threads. This achieves near-linear speedup compared to a single-threaded CPU implementation. 3. Experiments on text and image datasets demonstrate that the GPU implementation provides faster inference over the CPU version, though data transfer latency and memory limits remain challenges for large-scale problems.

1. Accelerating C ollapsed V ariational B ayesian Inference for L atent D irichlet A llocation with Nvidia CUDA Compatible Devices Tomonari MASADA 正田 備也 Nagasaki University [email_address]

3. LDA (latent Dirichlet allocation )

5. Dir ( β ) topic = word multinomial Tomonari MASADA (IEA-AIE 2009) symmetric Dirichlet prior v 1 v 2 v 3 v 4 t 1 φ 11 φ 12 φ 13 φ 14 v 1 v 2 v 3 v 4 t 2 φ 21 φ 22 φ 23 φ 24 v 1 v 2 v 3 v 4 t 3 φ 31 φ 32 φ 33 φ 34

6. document Dir ( α ) per-doc topic multinomial Tomonari MASADA (IEA-AIE 2009) symmetric Dirichlet prior θ j 1 θ j 2 θ j 3

7. v 3 v 1 v 3 v 2 v 2 θ j 1 θ j 2 θ j 3 Tomonari MASADA (IEA-AIE 2009) v 1 v 2 v 3 v 4 t 3 φ 31 φ 32 φ 33 φ 34 v 1 v 2 v 3 v 4 t 2 φ 21 φ 22 φ 23 φ 24 v 1 v 2 v 3 v 4 t 1 φ 11 φ 12 φ 13 φ 14

10. ・・・ Tomonari MASADA (IEA-AIE 2009) vote γ 111 γ 112 γ 113 party γ 12 1 γ 122 γ 123 prime γ 131 γ 132 γ 133 ・・・ stock γ 241 γ 242 γ 243 ratio γ 251 γ 252 γ 253 prime γ 231 γ 232 γ 233 ・・・ party γ 321 γ 322 γ 323 celeb γ 361 γ 362 γ 363 prime γ 331 γ 332 γ 333 ・・・

15. O( JK ) size O( KW ) size O( K ) size O( MK ) size Tomonari MASADA (IEA-AIE 2009) j : doc id w : word id k : topic id E[ n jk ] Var[ n jk ] E[ n kw ] Var[ n kw ] E[ n k ] Var[ n k ] γ jwk

17. Parallelization of CVB for LDA “ as many threads as topics”

20. Reduction for normalization O (log K ) Tomonari MASADA (IEA-AIE 2009)

21. Related Work

23. Accelerating CVB for LDA by GPGPU

24. Nvidia CUDA (C ompute U nified D evice A rchitecture ) Grid Device Memory documents topics Shared Memory Registers Thread Registers Thread Block Shared Memory Registers Thread Registers Thread Block

25. Device memory access latency Grid Device Memory 16KB Shared Memory Registers Thread Registers Thread Block Shared Memory Registers Thread Registers Thread Block

26. Data transfer latency Host Memory Transfer one large block instead of many smaller ones! Grid Device Memory Shared Memory Registers Thread Registers Thread Block Shared Memory Registers Thread Registers Thread Block

27. O( JK ) size O( KW ) size O( K ) size parameters of approximated posterior Tomonari MASADA (IEA-AIE 2009) j : doc id w : word id k : topic id E[ n jk ] Var[ n jk ] E[ n kw ] Var[ n kw ] E[ n k ] Var[ n k ] γ jwk

29. write conflicts O( KW ) size Tomonari MASADA (IEA-AIE 2009) γ j’wk E[ n kw ] Var[ n kw ] γ jwk

30. Experiments

32. 16 topics 64 iterations on CPU 64 iterations on GPU

33. 64 iterations on CPU 64 iterations on GPU 32 topics

34. 64 topics 64 iterations on CPU 64 iterations on GPU

37. Tomonari MASADA (IEA-AIE 2009)

39. Summary

42. Thank you for your attention! 非常感謝 !!!