This paper presents Query-by-Dancing, which is a dance music retrieval system that enables a user to retrieve music by dance motion. When dancers search for music to play when dancing, they sometimes refer to music in online dance videos in which motions similar to their own dance are used. Previous music retrieval systems, however, cannot support retrieval specialized for dancing because they do not accept dance motions as the query. We therefore developed our Query-by-Dancing system that uses a video of a dancing user as the input query to search a database of dance videos. This query video is recorded using an ordinary RGB camera that does not obtain depth information, such as one attached to a smartphone. The pose and motions in the query are then analyzed and used to retrieve a set of dance videos with similar pose and motions. The system then enables the user to browse the musical pieces attached to those retrieved videos so that the user can find a piece that is appropriate for his/her dancing. An interesting problem here is that simply finding the most similar videos results in getting results not similar in terms of genres of dance motions. We solved this by using a novel measure similar to tf-idf to weight the importance of dance motions when retrieving videos. We conducted the comparative experiments on four dance genres and confirmed that the system gained an average of three points or more evaluation scored for three dance genres, Waack, Pop, Break, and that our proposed method was able to deal with different dance genres.

1. Query-by-Dancing:
A Dance Music Retrieval System
Based on Body-Motion Similarity
Shuhei Tsuchida, Satoru Fukayama, Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST)
10.01.2019 MMM2019

2. It is important for dancers to find appropriate musical pieces.
Dancers often look for musical pieces
Stage performance Practice
DJ, freestyling Music video
source:
https://youtu.be/lRY-rdCQ7eI
source:
https://youtu.be/pcTBr64Elew

3. How does a dancer usually find musical pieces?
Keywords
• Hiphop dance
• Break dance
• etc.
Musical pieces
Existing Retrieval System
Dancer
Dance Style

4. How does a dancer usually find musical pieces?
Keywords
• Hiphop dance
• Break dance
• etc.
Musical pieces
Existing Retrieval System
Dancer
Dance Style

5. How does a dancer usually find musical pieces?
Keywords
• Hiphop dance
• Break dance
• etc.
Musical pieces
Existing Retrieval System
Dancer
Dance Style

6. How does a dancer usually find musical pieces?
Keywords
• Hiphop dance
• Break dance
• etc.
Musical pieces
Existing Retrieval System
Dancer
Dance Style
Too vague

7. How does a dancer usually find musical pieces?
Keywords
• Favorite dancer
• Favorite group
• Dance events
• etc.
Existing Retrieval System
Dancer
Dance Style
Musical piece
Compare

8. How does a dancer usually find musical pieces?
Keywords
• Favorite dancer
• Favorite group
• Dance events
• etc.
Existing Retrieval System
Dancer
Dance Style
Compare
Musical piece

9. How does a dancer usually find musical pieces?
Keywords
• Favorite dancer
• Favorite group
• Dance events
• etc.
Existing Retrieval System
Dancer
Dance Style
Musical piece
Compare

10. How does a dancer usually find musical pieces?
Keywords
• Favorite dancer
• Favorite group
• Dance events
• etc.
Existing Retrieval System
Dancer
Dance Style
Musical piece
Compare
Tedious and time-consuming

11. How does a dancer usually find musical pieces?
Query-by-Dancing
Dancer
Dance motion
Musical pieces

12. How does a dancer usually find musical pieces?
Query-by-Dancing
Dancer
Dance motion
Musical pieces

13. How does a dancer usually find musical pieces?
Query-by-Dancing
Dancer
Dance motion
Musical pieces

14. System overview

15. Query-by-Dancing: retrieval results

16. Query-by-Dancing: retrieval results
Input video

17. Query-by-Dancing: retrieval results
Retrieval results

18. Query-by-Dancing: retrieval results (Break)

19. Query-by-Dancing: retrieval results (Break)

20. Query-by-Dancing: retrieval results
Similarity

21. Query-by-Dancing: retrieval results
0,1,2, … … 97, 98, 99

22. Query-by-Dancing: retrieval results
Top 10

23. Query-by-Dancing: retrieval results
Retrieval rank, Dance genre

24. Query-by-Dancing: retrieval results (Pop)

25. Query-by-Dancing: retrieval results (Pop)

26. Query-by-Dancing: retrieval results (Waack)

27. Query-by-Dancing: retrieval results (Waack)

28. System flow
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ
2
Pre-processing
Calculate similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces

29. System flow
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ
2
Pre-processing
Calculate similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces

30. Pre-processing: select a dancer
The system first estimates the person's skeleton information
in all video frames using the OpenPose library.
Z. Cao et al.: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields (2017).

31. Pre-processing: select a dancer
The system first estimates the person's skeleton information
in all video frames using the OpenPose library.
Z. Cao et al.: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields (2017).
A dancer was defined as a person who is close to
the center of the screen and appears big.

32. Pre-processing: calculate joint angles
• Pose
• Motion

33. Pre-processing: calculate joint angels
• Pose
• Motion

34. Pre-processing: calculate joint angels
• Pose
• Motion

35. System flow
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ
2
Pre-processing
Calculate similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces

36. Calculate dance motion similarity: Euclidean distance
t

37. Calculate dance motion similarity: Euclidean distance
t

38. Calculate dance motion similarity: Euclidean distance
t

39. Calculate dance motion similarity: Euclidean distance
t

40. Calculate dance motion similarity: Euclidean distance
The system calculates the Euclidean distances
between all frames of an input video and all frames of a video in the video database.
d(vin(n), v(i)(m))
1 ≤ n ≤ Nin 1 ≤ m ≤ N(i) 1 ≤ i ≤ I

41. Calculate dance motion similarity: Euclidean distance
The system calculates the Euclidean distances
between all frames of an input video and all frames of a video in the video database.
d(vin(n), v(i)(m))
1 ≤ n ≤ Nin 1 ≤ m ≤ N(i) 1 ≤ i ≤ I
• If the system uses these calculation results
(Euclidean distance) as a similarity distance,
→ The system doesn't work well.
Reason: Most of dance videos include
several common movements.
• The system uses weighting methods to consider
these common movements.

42. Small Weight
Unimportant dance motion
Many common motions
in the database
Calculate dance motion similarity: weight
Large Weight
Important dance motion
Rare motions
in the database
The system uses a method similar to tf-idf often used
in natural language processing.

43. Small Weight
Unimportant dance motion
Many common motions
in the database
Calculate dance motion similarity: weight
Large Weight
Important dance motion
Rare motions
in the database
The system uses a method similar to tf-idf often used
in natural language processing.
Database
Video 1
Video 2
Video 3
Video 4
Video 5
… …

44. Small Weight
Unimportant dance motion
Many common motions
in the database
Calculate dance motion similarity: weight
Large Weight
Important dance motion
Rare motions
in the database
The system uses a method similar to tf-idf often used
in natural language processing.
Database
Video 1
Video 2
Video 3
Video 4
Video 5
… …

45. Small Weight
Unimportant dance motion
Many common motions
in the database
Calculate dance motion similarity: weight
Large Weight
Important dance motion
Rare motions
in the database
The system uses a method similar to tf-idf often used
in natural language processing.

46. System overview
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2
Pre-
processing
Calculate
similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces

47. System overview
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2
Pre-
processing
Calculate
similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces
Weighted-
Euclidean distance
0 IVideos in database

48. System overview
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2
Pre-
processing
Calculate
similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces
Weighted-
Euclidean distance
0 IVideos in database

49. System overview
vθ , vΔθ , vΔ θ
d, Wα
U α
2
(i)
Input video Dance video database
Video1 Video2 Video3
C
Select a dancer
Calculate and integrate joint angles vθ, vΔθ, and vΔ θ2
Pre-
processing
Calculate
similarity
Calculate weight W
Calculate weighted Euclidean distances of body-motion U(i)
Output musical pieces
Weighted-
Euclidean distance
0 IVideos in database
The system finds the videos with the top k values among weighted-
Euclidean distances and presents the candidate musical pieces.

50. Evaluation experiment
Input video
Output musical pieces

51. Evaluation experiment
Input video
Output musical pieces
Strongly disagree 1 - 2 - 3 - 4 - 5 Strongly agree

52. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

53. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

54. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

55. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

56. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

57. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

58. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre

59. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Output musical pieces
Input Dance Genre
The participants can get the musical pieces that they dance easily.

60. Results
*
: p<.05
*
*
*
Averaged score
Averagedevaluationscore
Input Dance Genre
Output musical pieces

61. • We interviewed participants who gave a low score to pop
to determine the reason for this.
Discussion
• The dance motions used as the query included
"vibration" and "wave" dance techniques
that match specific sounds.

62. • We interviewed participants who gave a low score to pop
to determine the reason for this.
Discussion
• The dance motions used as the query included
"vibration" and "wave" dance techniques
that match specific sounds.

63. • We interviewed participants who gave a low score to pop
to determine the reason for this.
Discussion
• The dance motions used as the query included
"vibration" and "wave" dance techniques
that match specific sounds.

64. • The dance motions used as the query included
"vibration" and "wave" dance techniques
that match specific sounds.
→ (Future work) An interface that supports
more advanced motion matching.
• We interviewed participants who gave a low score to pop
to determine the reason for this.
Discussion

65. Summary
Query-by-Dancing
• enables a user to retrieve a musical piece
using dance motions.
• can get better musical pieces by weighting
the importance of dance motions.
Future work
• We plan to add a wider range of dance genres
to the database of dance videos.