SlideShare una empresa de Scribd logo
1 de 90
Descargar para leer sin conexión
SEQUENTIAL GALLERY
FOR INTERACTIVE VISUAL
DESIGN OPTIMIZATION
Yuki Koyama, Issei Sato, and Masataka Goto
© 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
Sequential Gallery
for Interactive Visual Design Optimization
Yuki Koyama1, Issei Sato2, and Masataka Goto1
1. National Institute of Advanced Industrial Science and Technology (AIST)
2. The University of Tokyo
Initial Optimized
Sequential Gallery
… …
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Overview
Background and Target Problem
3
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Background: Parametric Visual Design is Everywhere
4
… etc.
Procedural designPhoto color enhancement Generative modeling
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Background: Parametric Visual Design is Everywhere
5
Problem: Need many trials and errors
• E.g., try a different parameter combination, see the result, judge whether it is
better or not, and then decide which combination to try next …

• This is a high-dimensional search task and can be tedious and time-
consuming
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Background: Parametric Visual Design is Everywhere
Motivation: What computational support is possible?
Technical challenge:
• Parameters need to be determined based on the user’s preference

• It is difficult to fully automate the parameter tweaking process
6
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Overview
Proposed System: Sequential Gallery
7
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 8
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 9
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 10
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
2D search subtask #1
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 11
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
2D search subtask #1
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 12
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
2D search subtask #1
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
2D search subtask #2
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 13
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
2D search subtask #2
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
2D search subtask #3
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 14
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
2D search subtask #3
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
2D search subtask #4
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 15
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
2D search subtask #4
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 16
…
Target:
n design parameters
(e.g., photo enhance)
…
Output:
An optimal
parameter set
Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Overview
Example Usage
17
Photo Color Enhancement (12D)
Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
Original photograph Enhanced photograph

(after 4 iterations)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Overview
Summary of Contributions
21
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Summary of Contributions
• Novel algorithm: Sequential plane search
• A variant of preferential Bayesian optimization
(PBO), enabling user-in-the-loop optimization

• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

• Interactive system: Sequential Gallery
• Use the sequential-plane-search algorithm in
combination with a zoomable grid interface

• Enable the user to effectively explore the design
space and perform the optimization
22
... ...
Sequential plane search
Sequential Gallery
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Summary of Contributions
• Novel algorithm: Sequential plane search
• A variant of preferential Bayesian optimization
(PBO), enabling user-in-the-loop optimization

• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

• Interactive system: Sequential Gallery
• Use the sequential-plane-search algorithm in
combination with a zoomable grid interface

• Enable the user to effectively explore the design
space and perform the optimization
23
... ...
Sequential plane search
Sequential Gallery
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Summary of Contributions
• Novel algorithm: Sequential plane search
• A variant of preferential Bayesian optimization
(PBO), enabling user-in-the-loop optimization

• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

• Interactive system: Sequential Gallery
• Use the sequential-plane-search algorithm in
combination with a zoomable grid interface

• Enable the user to effectively explore the design
space and perform the optimization
24
... ...
Sequential plane search
Sequential Gallery
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Problem Definition
From Mathematical Viewpoint
25
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Problem Definition from Mathematical Viewpoint
26
• Suppose that we have sliders to adjust

• Let be the search space and
be a set of parameter values

• Let be a perceptual
preference function (= goodness function)
which returns a goodness value

• We want to solve an optimization problem:
n
𝒳 = [0,1]n
x ∈ 𝒳 n
g : 𝒳 → ℝ
x* = argmax
x∈𝒳
g(x)
[Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds.
Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Problem Definition from Mathematical Viewpoint
27
• Suppose that we have sliders to adjust

• Let be the search space and
be a set of parameter values

• Let be a perceptual
preference function (= goodness function)
which returns a goodness value

• We want to solve an optimization problem:
n
𝒳 = [0,1]n
x ∈ 𝒳 n
g : 𝒳 → ℝ
x* = argmax
x∈𝒳
g(x)
[Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds.
Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interacting with Goodness Function
28
Absolute assessment
should not be used: 

The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]

Relative assessment
should be used: 

The user can answer
which option is better
among two (or more)
options
Absolute
assessment
Relative
assessmentA B
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interacting with Goodness Function
29
Absolute assessment
should not be used: 

The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]

Relative assessment
should be used: 

The user can answer
which option is better
among two (or more)
options
Absolute
assessment
Relative
assessmentA B
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interacting with Goodness Function
30
Absolute assessment
should not be used: 

The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]

Relative assessment
should be used: 

The user can answer
which option is better
among two (or more)
options
Absolute
assessment
Relative
assessmentA B
Preferential feedback
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interacting with Goodness Function
31
Absolute assessment
should not be used: 

The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]

Relative assessment
should be used: 

The user can answer
which option is better
among two (or more)
options
Absolute
assessment
Relative
assessmentA B
Preferential Bayesian optimization (PBO)

can run using relative assessment (i.e., preferential feedback)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Preferential Bayesian Optimization (PBO)
Previous Techniques and Our New Technique
32
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Basics:“Standard” Bayesian Optimization (BO)
• Is a global “black-box” optimization algorithm

• Can find optimal solutions with only a small
number of function evaluations
• Thus, useful to handle expensive-to-evaluate
objective functions

• Example applications: hyperparameter tuning for
machine learning models [Akiba+, KDD 2019]
33
See [Shahriari+, Proc. IEEE 2016] for details
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Preferential BO (PBO)
• PBO is an extension of BO, which runs with relative assessment (or
preferential feedback), rather than absolute assessment of function values

• PBO can find optimal solutions with only a small number of preferential
feedbacks
• Note: human is expensive-to-query
34
Relative
assessmentA B
xchosen
= argmax
x∈{xA
,xB
}
g(x) xchosen
= argmax
x∈𝒮
g(x) xchosen
= argmax
x∈𝒫
g(x)
[Brochu+, NIPS 2007]
“Pairwise Comparison” Query
[Koyama+, SIGGRAPH 2017]
“Line Search” Query
(Sequential Line Search)
[Ours]
“Plane Search” Query
(Sequential Plane Search)
Query
type:
User
task:
Notes: Is the first PBO algorithm
Needs fewer iterations than
[Brochu+ NIPS 2007]
Needs even fewer iterations,
and have a good compatibility
with grid interfaces
xchosen
= argmax
x∈{xA
,xB
}
g(x) xchosen
= argmax
x∈𝒮
g(x) xchosen
= argmax
x∈𝒫
g(x)
[Brochu+, NIPS 2007]
“Pairwise Comparison” Query
[Koyama+, SIGGRAPH 2017]
“Line Search” Query
(Sequential Line Search)
[Ours]
“Plane Search” Query
(Sequential Plane Search)
Query
type:
User
task:
Notes: Is the first PBO algorithm
Needs fewer iterations than
[Brochu+ NIPS 2007]
Needs even fewer iterations,
and have a good compatibility
with grid interfaces
xchosen
= argmax
x∈{xA
,xB
}
g(x) xchosen
= argmax
x∈𝒮
g(x) xchosen
= argmax
x∈𝒫
g(x)
[Brochu+, NIPS 2007]
“Pairwise Comparison” Query
[Koyama+, SIGGRAPH 2017]
“Line Search” Query
(Sequential Line Search)
[Ours]
“Plane Search” Query
(Sequential Plane Search)
Query
type:
User
task:
Notes: Is the first PBO algorithm
Needs fewer iterations than
[Brochu+ NIPS 2007]
Needs even fewer iterations,
and has a good compatibility
with grid interfaces
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Sequential Gallery Workflow
38
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
User’s feedback
Next search plane
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
User’s feedback
Next search plane
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
… 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
…… 2D search subtask 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
…… 2D search subtask 2D search subtask 2D search subtask
2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
53
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
54
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
55
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
56
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
57
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
58
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
59
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Interface: Zoomable Grid Interface for 2D Search Subtasks
60
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Benefits of Zoomable Grid Interface
61
• Allows the user to easily grasp the available
options in the 2D subspace by just seeing the grid
view

• WYSIWYG (What-You-See-Is-What-You-Get); do
not need to be aware of raw parameters

• Compatibility with the sequential-plane-search task
(i.e., 2D search)

• Minimum quantization errors (thanks to zooming
operations)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Sequential Plane Search
How to Determine the Next Plane in Each Iteration
62
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Criterion to Select 2D Search Planes
We want to select the most effective 2D subspace (=
search plane) for the next query

• Effectiveness here means the degree to which it
is worth observing in the next iteration to find the
optimal solution

Question: How can we define the effectiveness?
63
?
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Acquisition Function in (Standard) BO
• Standard BO uses acquisition functions

• An acquisition function evaluates how
effective a point is as the next query

• The maximizer of the acquisition function is
selected as the next point

• We extend this point-wise acquisition function
to evaluate how effective a search plane is
as the next query (next slide)
64
Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can
automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions)
strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Acquisition Function in (Standard) BO
• Standard BO uses acquisition functions

• An acquisition function evaluates how
effective a point is as the next query

• The maximizer of the acquisition function is
selected as the next point

• We extend this point-wise acquisition function
to evaluate how effective a search plane is
as the next query (next slide)
65
Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can
automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions)
strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Acquisition Function for Determining the Next Plane
• Let be the accumulated preferential data so far obtained

• Point-wise acquisition function (used in standard BO):

[Note 1]

• Plane-wise acquisition function (proposed):



• We choose the next plane by solving a maximization problem:

[Note 2]
𝒟
apoint
(x; 𝒟)
aplane
(𝒫; 𝒟) =
∫ 𝒫
apoint
(x; 𝒟)dS
𝒫next
𝒫next
= argmax
𝒫∈𝒳
aplane
(𝒫; 𝒟)
66
[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to
estimate the goodness function landscape from the available preferential data.

[Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
apoint
𝒫 x
𝒳
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Acquisition Function for Determining the Next Plane
• Let be the accumulated preferential data so far obtained

• Point-wise acquisition function (used in standard BO):

[Note 1]

• Plane-wise acquisition function (proposed):



• We choose the next plane by solving a maximization problem:

[Note 2]
𝒟
apoint
(x; 𝒟)
aplane
(𝒫; 𝒟) =
∫ 𝒫
apoint
(x; 𝒟)dS
𝒫next
𝒫next
= argmax
𝒫∈𝒳
aplane
(𝒫; 𝒟)
67
[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to
estimate the goodness function landscape from the available preferential data.

[Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
apoint
𝒫 x
𝒳
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Acquisition Function for Determining the Next Plane
• Let be the accumulated preferential data so far obtained

• Point-wise acquisition function (used in standard BO):

[Note 1]

• Plane-wise acquisition function (proposed):



• We choose the next plane by solving a maximization problem:

[Note 2]
𝒟
apoint
(x; 𝒟)
aplane
(𝒫; 𝒟) =
∫ 𝒫
apoint
(x; 𝒟)dS
𝒫next
𝒫next
= argmax
𝒫∈𝒳
aplane
(𝒫; 𝒟)
68
[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to
estimate the goodness function landscape from the available preferential data.

[Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
apoint
𝒫 x
𝒳
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Applications:
Possible Scenarios and Demonstrations
69
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Potential Applications
70
Photo color enhancement Generative modeling Procedural texturing
… and many other parametric design scenarios
Photo Color Enhancement (12D)
Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
Original photograph Enhanced photograph

(after 5 iterations)
Body Shaping (10D)
Using the SMPL model [Loper+15] (the first 10 principal components)
[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization
“He was of medium height, solidly built,
wide in the shoulders, thick in the neck,
with a jovial heavy-jawed red face […]”
Dashiell Hammett. 1930. The Maltese Falcon.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Evaluation [1/2]:
Optimization Performance Comparison
77
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Performance Comparison Using Synthetic Functions
78
• Goals:
• Evaluate the efficiency of our sequential plane search
compared to the previous work [Koyama+17]

• Confirm that BO (i.e., the use of the acquisition
function) contributes to the optimization performance

• How:
• Use synthetic objective functions to simulate human
responses (shown on the right)

• Algorithms to be compared: (next slide)
Gaussian (5D/15D)
Rosenbrock (10D/20D)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Algorithms to be Compared
79
Baseline 1:

SLS
Baseline 2:

SPS (Random) SPS (Ours)
Sequential line search
[Koyama+17]
Sequential plane
search, but the plane
is randomly chosen
(instead of using BO)
Sequential plane
search (using BO)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Result: Performance Comparison [1/2]
80
Synthetic objective function: Gaussian function
Performance: SLS < SPS (random) < SPS (ours)
(Lowerisbetter)
5D 15D
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Result: Performance Comparison [2/2]
81
Synthetic objective function: Rosenbrock function
(Lowerisbetter)
10D 20D
Performance: SLS < SPS (random) < SPS (ours)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Evaluation [2/2]:
Informal User Study
82
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Informal User Study
• Participants: Six students/researchers (five novices and one expert)

• Task: Enhance photographs using our system (12D)

• Results:

• All the participants could successfully perform optimization and find
satisfactory results in 5.36 iterations in average (SD = 2.69)

• Participants appreciated the grid view since they could get inspiration for
possible color variations easily and quickly
83
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Discussions
Limitations and Future Work
84
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Discussions: High Dimensionality
• BO (and thus PBO, too) is inherently not good
at handling very high-dimensional problems
[Wang+16]

• We assumed the dimensionality is at most 20
• To overcome the dimensionality issue,
application-specific extensions should be made

• E.g., for generative modeling (e.g., GANs),
[Chiu+, SIGGRAPH 2020] is a possible option
to combine with Sequential Gallery
85
C.f., A method specific to
generative modeling 

[Chiu+, SIGGRAPH 2020]
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Discussion: Others
• Initial plane selection: we randomly choose initial planes, but other
strategies are possible (e.g., use Design Gallery [Marks+97] for the
first iteration and then start the sequential plane search iterations)

• Time-changing preference: we assume that the user’s preference
does not change during the iteration, but this is not true in some
cases; supporting time-changing preference is an interesting future
work

• Grid resolution and zooming levels: currently, we manually select
the grid resolution and the number of levels, but this may be adjusted
dynamically by analyzing the just-noticeable difference (JND)

• Prior knowledge: when some prior data is available, we could build
a rough approximation of the goodness function and then use it as a
prior of the Bayesian inference; this would make the optimization
even more efficient
86
Refer to the paper
for details
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Summary
87
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
Summary
• Sequential Gallery is an interactive system for user-in-the-loop visual
design optimization

• Its efficiency is enabled by sequential plane search, which is a new variant
of preferential Bayesian optimization and is able to find the solution with
only a minimal number of iterations
88
Initial Optimized
Sequential Gallery
… …
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)
References
• [Akiba+, KDD 2019] Takuya Akiba, Shotaro Sano, Toshihiko
Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna:
A Next-generation Hyperparameter Optimization Framework.
In Proc. KDD ’19. 2623–2631. DOI:https://doi.org/
10.1145/3292500.3330701

• [Brochu+, NIPS 2007] Eric Brochu, Nando de Freitas, and
Abhijeet Ghosh. 2007. Active Preference Learning with
Discrete Choice Data. In Proc. NIPS '07. 409–416. http://
papers.nips.cc/paper/3219-active-preference-learning-with-
discrete-choice-data.pdf

• [Brochu+, SCA 2010] Eric Brochu, Tyson Brochu, and
Nando de Freitas. 2010. A Bayesian Interactive Optimization
Approach to Procedural Animation Design. In Proc. SCA ’10.
103–112. https://doi.org/10.2312/SCA/SCA10/103-112 

• [Chiu+, SIGGRAPH 2020] Chia-Hsing Chiu, Yuki
Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao
Yue. 2020. Human-in-the-Loop Differential Subspace Search
in High-Dimensional Latent Space. ACM Trans. Graph. 39, 4,
pp.85:1–85:15 (July 2020). DOI:https://doi.org/
10.1145/3386569.3392409

• [Koyama+, SIGGRAPH 2017] Yuki Koyama, Issei Sato,
Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential line
search for efficient visual design optimization by crowds.
ACM Trans. Graph. 36, 4, Article 48 (July 2017), 11 pages.
DOI:https://doi.org/10.1145/3072959.3073598

• [Koyama+, Computational Interaction (2018)] Yuki
Koyama and Takeo Igarashi. 2018. Computational Design
with Crowds. In Computational Interaction (Eds. Antti
Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew
Howes), Oxford University Press, pp.153—184. https://
arxiv.org/abs/2002.08657

• [Shahriari+, Proc. IEEE (2016)] Bobak Shahriari, Kevin
Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas.
2016. Taking the Human Out of the Loop: A Review of
Bayesian Optimization. Proc. IEEE 104,1(January 2016),148–
175. https://doi.org/10.1109/JPROC.2015.2494218

• [Wang+, JAIR (2016)] Ziyu Wang, Frank Hutter, Masrour
Zoghi, David Matheson, and Nando de Feitas. 2016.
Bayesian Optimization in a Billion Dimensions via Random
Embeddings. J. Artif. Intell. Res. 55 (February 2016), 361–
387. https://doi.org/10.1613/jair.4806
89
Sequential Gallery
for Interactive Visual Design Optimization
Yuki Koyama1, Issei Sato2, and Masataka Goto1
1. National Institute of Advanced Industrial Science and Technology (AIST)
2. The University of Tokyo
Initial Optimized
Sequential Gallery
… …

Más contenido relacionado

La actualidad más candente

研究について思うところ | What i think about research (in Japanese)
研究について思うところ | What i think about research (in Japanese)研究について思うところ | What i think about research (in Japanese)
研究について思うところ | What i think about research (in Japanese)Yuta Itoh
 
深層学習の数理
深層学習の数理深層学習の数理
深層学習の数理Taiji Suzuki
 
金融時系列のための深層t過程回帰モデル
金融時系列のための深層t過程回帰モデル金融時系列のための深層t過程回帰モデル
金融時系列のための深層t過程回帰モデルKei Nakagawa
 
TensorFlowで逆強化学習
TensorFlowで逆強化学習TensorFlowで逆強化学習
TensorFlowで逆強化学習Mitsuhisa Ohta
 
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)Shinya Takamaeda-Y
 
金融情報における時系列分析
金融情報における時系列分析金融情報における時系列分析
金融情報における時系列分析Fujio Toriumi
 
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...Deep Learning JP
 
研究法(Claimとは)
研究法(Claimとは)研究法(Claimとは)
研究法(Claimとは)Jun Rekimoto
 
機械学習の未解決課題
機械学習の未解決課題機械学習の未解決課題
機械学習の未解決課題Hiroyuki Masuda
 
(DL輪読)Variational Dropout Sparsifies Deep Neural Networks
(DL輪読)Variational Dropout Sparsifies Deep Neural Networks(DL輪読)Variational Dropout Sparsifies Deep Neural Networks
(DL輪読)Variational Dropout Sparsifies Deep Neural NetworksMasahiro Suzuki
 
差分プライバシーとは何か? (定義 & 解釈編)
差分プライバシーとは何か? (定義 & 解釈編)差分プライバシーとは何か? (定義 & 解釈編)
差分プライバシーとは何か? (定義 & 解釈編)Kentaro Minami
 
深層学習による非滑らかな関数の推定
深層学習による非滑らかな関数の推定深層学習による非滑らかな関数の推定
深層学習による非滑らかな関数の推定Masaaki Imaizumi
 
Pythonによる黒魔術入門
Pythonによる黒魔術入門Pythonによる黒魔術入門
Pythonによる黒魔術入門大樹 小倉
 
深層学習とベイズ統計
深層学習とベイズ統計深層学習とベイズ統計
深層学習とベイズ統計Yuta Kashino
 
【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Modelscvpaper. challenge
 
自動微分変分ベイズ法の紹介
自動微分変分ベイズ法の紹介自動微分変分ベイズ法の紹介
自動微分変分ベイズ法の紹介Taku Yoshioka
 
私は如何にして心配するのを止めてPyTorchを愛するようになったか
私は如何にして心配するのを止めてPyTorchを愛するようになったか私は如何にして心配するのを止めてPyTorchを愛するようになったか
私は如何にして心配するのを止めてPyTorchを愛するようになったかYuta Kashino
 
最適化超入門
最適化超入門最適化超入門
最適化超入門Takami Sato
 
#FTMA15 第二回課題 鬼コースサーベイ
#FTMA15 第二回課題 鬼コースサーベイ#FTMA15 第二回課題 鬼コースサーベイ
#FTMA15 第二回課題 鬼コースサーベイYoichi Ochiai
 
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)Shinya Takamaeda-Y
 

La actualidad más candente (20)

研究について思うところ | What i think about research (in Japanese)
研究について思うところ | What i think about research (in Japanese)研究について思うところ | What i think about research (in Japanese)
研究について思うところ | What i think about research (in Japanese)
 
深層学習の数理
深層学習の数理深層学習の数理
深層学習の数理
 
金融時系列のための深層t過程回帰モデル
金融時系列のための深層t過程回帰モデル金融時系列のための深層t過程回帰モデル
金融時系列のための深層t過程回帰モデル
 
TensorFlowで逆強化学習
TensorFlowで逆強化学習TensorFlowで逆強化学習
TensorFlowで逆強化学習
 
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)
Veriloggen.Stream: データフローからハードウェアを作る(2018年3月3日 高位合成友の会 第5回 @東京工業大学)
 
金融情報における時系列分析
金融情報における時系列分析金融情報における時系列分析
金融情報における時系列分析
 
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
[DL輪読会]Vision Transformer with Deformable Attention (Deformable Attention Tra...
 
研究法(Claimとは)
研究法(Claimとは)研究法(Claimとは)
研究法(Claimとは)
 
機械学習の未解決課題
機械学習の未解決課題機械学習の未解決課題
機械学習の未解決課題
 
(DL輪読)Variational Dropout Sparsifies Deep Neural Networks
(DL輪読)Variational Dropout Sparsifies Deep Neural Networks(DL輪読)Variational Dropout Sparsifies Deep Neural Networks
(DL輪読)Variational Dropout Sparsifies Deep Neural Networks
 
差分プライバシーとは何か? (定義 & 解釈編)
差分プライバシーとは何か? (定義 & 解釈編)差分プライバシーとは何か? (定義 & 解釈編)
差分プライバシーとは何か? (定義 & 解釈編)
 
深層学習による非滑らかな関数の推定
深層学習による非滑らかな関数の推定深層学習による非滑らかな関数の推定
深層学習による非滑らかな関数の推定
 
Pythonによる黒魔術入門
Pythonによる黒魔術入門Pythonによる黒魔術入門
Pythonによる黒魔術入門
 
深層学習とベイズ統計
深層学習とベイズ統計深層学習とベイズ統計
深層学習とベイズ統計
 
【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models【メタサーベイ】基盤モデル / Foundation Models
【メタサーベイ】基盤モデル / Foundation Models
 
自動微分変分ベイズ法の紹介
自動微分変分ベイズ法の紹介自動微分変分ベイズ法の紹介
自動微分変分ベイズ法の紹介
 
私は如何にして心配するのを止めてPyTorchを愛するようになったか
私は如何にして心配するのを止めてPyTorchを愛するようになったか私は如何にして心配するのを止めてPyTorchを愛するようになったか
私は如何にして心配するのを止めてPyTorchを愛するようになったか
 
最適化超入門
最適化超入門最適化超入門
最適化超入門
 
#FTMA15 第二回課題 鬼コースサーベイ
#FTMA15 第二回課題 鬼コースサーベイ#FTMA15 第二回課題 鬼コースサーベイ
#FTMA15 第二回課題 鬼コースサーベイ
 
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)
Veriloggen: Pythonによるハードウェアメタプログラミング(第3回 高位合成友の会 @ドワンゴ)
 

Similar a [SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization

Copy of Copy of Untitled presentation (1).pdf
Copy of Copy of Untitled presentation (1).pdfCopy of Copy of Untitled presentation (1).pdf
Copy of Copy of Untitled presentation (1).pdfjosephdonnelly2024
 
An analytical framework for formulating metrics for evaluating multi-dimensio...
An analytical framework for formulating metrics for evaluating multi-dimensio...An analytical framework for formulating metrics for evaluating multi-dimensio...
An analytical framework for formulating metrics for evaluating multi-dimensio...Rei Takami
 
Engineering + Programming portfolio
Engineering + Programming portfolioEngineering + Programming portfolio
Engineering + Programming portfolioJosephDonnelly14
 
Breaking the Boundaries of Human-in-the-Loop Optimization
Breaking the Boundaries of Human-in-the-Loop OptimizationBreaking the Boundaries of Human-in-the-Loop Optimization
Breaking the Boundaries of Human-in-the-Loop OptimizationYiChiLiao2
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfAPSanyal1
 
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeMatching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeCUbRIK Project
 
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeMatching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeLuca Galli
 
Questions On The Equation For Regression
Questions On The Equation For RegressionQuestions On The Equation For Regression
Questions On The Equation For RegressionTiffany Sandoval
 
Cartoonization of images using machine Learning
Cartoonization of images using machine LearningCartoonization of images using machine Learning
Cartoonization of images using machine LearningIRJET Journal
 
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURE
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURESTUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURE
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTUREAnganam
 
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...Aalto University
 
Interactive Editing of Signed Distance Fields
Interactive Editing of Signed Distance FieldsInteractive Editing of Signed Distance Fields
Interactive Editing of Signed Distance FieldsMatthias Trapp
 
Computer Graphics Notes
Computer Graphics NotesComputer Graphics Notes
Computer Graphics NotesGurpreet singh
 
Shannon Park Redevelopment Plan Remodeling with Esri CityEngine
Shannon Park Redevelopment Plan Remodeling with Esri CityEngineShannon Park Redevelopment Plan Remodeling with Esri CityEngine
Shannon Park Redevelopment Plan Remodeling with Esri CityEngineCOGS Presentations
 
B. SC CSIT Computer Graphics Lab By Tekendra Nath Yogi
B. SC CSIT Computer Graphics Lab By Tekendra Nath YogiB. SC CSIT Computer Graphics Lab By Tekendra Nath Yogi
B. SC CSIT Computer Graphics Lab By Tekendra Nath YogiTekendra Nath Yogi
 
Metric Recovery from Unweighted k-NN Graphs
Metric Recovery from Unweighted k-NN GraphsMetric Recovery from Unweighted k-NN Graphs
Metric Recovery from Unweighted k-NN Graphsjoisino
 
Day 1 presentation terminology
Day 1 presentation   terminologyDay 1 presentation   terminology
Day 1 presentation terminologykelv_w
 

Similar a [SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization (20)

Copy of Copy of Untitled presentation (1).pdf
Copy of Copy of Untitled presentation (1).pdfCopy of Copy of Untitled presentation (1).pdf
Copy of Copy of Untitled presentation (1).pdf
 
An analytical framework for formulating metrics for evaluating multi-dimensio...
An analytical framework for formulating metrics for evaluating multi-dimensio...An analytical framework for formulating metrics for evaluating multi-dimensio...
An analytical framework for formulating metrics for evaluating multi-dimensio...
 
Engineering + Programming portfolio
Engineering + Programming portfolioEngineering + Programming portfolio
Engineering + Programming portfolio
 
Breaking the Boundaries of Human-in-the-Loop Optimization
Breaking the Boundaries of Human-in-the-Loop OptimizationBreaking the Boundaries of Human-in-the-Loop Optimization
Breaking the Boundaries of Human-in-the-Loop Optimization
 
Shading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdfShading analysis using Ecotect software.pdf
Shading analysis using Ecotect software.pdf
 
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeMatching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
 
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a PurposeMatching Game Mechanics and Human Computation Tasks in Games with a Purpose
Matching Game Mechanics and Human Computation Tasks in Games with a Purpose
 
Questions On The Equation For Regression
Questions On The Equation For RegressionQuestions On The Equation For Regression
Questions On The Equation For Regression
 
Cartoonization of images using machine Learning
Cartoonization of images using machine LearningCartoonization of images using machine Learning
Cartoonization of images using machine Learning
 
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURE
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURESTUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURE
STUDY ON SCOPE OF PARAMETRIC SOFTWARES IN ARCHITECTURE
 
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...
User Interfaces that Design Themselves: Talk given at Data-Driven Design Day ...
 
Interactive Editing of Signed Distance Fields
Interactive Editing of Signed Distance FieldsInteractive Editing of Signed Distance Fields
Interactive Editing of Signed Distance Fields
 
Computer Graphics Notes
Computer Graphics NotesComputer Graphics Notes
Computer Graphics Notes
 
Shannon Park Redevelopment Plan Remodeling with Esri CityEngine
Shannon Park Redevelopment Plan Remodeling with Esri CityEngineShannon Park Redevelopment Plan Remodeling with Esri CityEngine
Shannon Park Redevelopment Plan Remodeling with Esri CityEngine
 
B. SC CSIT Computer Graphics Lab By Tekendra Nath Yogi
B. SC CSIT Computer Graphics Lab By Tekendra Nath YogiB. SC CSIT Computer Graphics Lab By Tekendra Nath Yogi
B. SC CSIT Computer Graphics Lab By Tekendra Nath Yogi
 
Metric Recovery from Unweighted k-NN Graphs
Metric Recovery from Unweighted k-NN GraphsMetric Recovery from Unweighted k-NN Graphs
Metric Recovery from Unweighted k-NN Graphs
 
Ix2515851588
Ix2515851588Ix2515851588
Ix2515851588
 
Ix2515851588
Ix2515851588Ix2515851588
Ix2515851588
 
Task 2
Task 2Task 2
Task 2
 
Day 1 presentation terminology
Day 1 presentation   terminologyDay 1 presentation   terminology
Day 1 presentation terminology
 

Más de Yuki Koyama

[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...Yuki Koyama
 
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Yuki Koyama
 
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Yuki Koyama
 
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...Yuki Koyama
 
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...Yuki Koyama
 
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザインYuki Koyama
 
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半Yuki Koyama
 
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...Yuki Koyama
 
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)Yuki Koyama
 
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)Yuki Koyama
 
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)Yuki Koyama
 
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...Yuki Koyama
 
Real-Time Example-Based Elastic Deformation (SCA '12)
Real-Time Example-Based Elastic Deformation (SCA '12)Real-Time Example-Based Elastic Deformation (SCA '12)
Real-Time Example-Based Elastic Deformation (SCA '12)Yuki Koyama
 

Más de Yuki Koyama (13)

[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
[UIST 2022] BO as Assistant: Using Bayesian Optimization for Asynchronously G...
 
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
Precomputed Optimal One-Hop Motion Transition for Responsive Character Animat...
 
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
Decomposing Images into Layers with Advanced Color Blending [Pacific Graphics...
 
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
[CHI 2018] OptiMo: Optimization-Guided Motion Editing for Keyframe Character ...
 
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
[SIGGRAPH 2017] Sequential Line Search for Efficient Visual Design Optimizati...
 
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
[IPSJ-ONE 2017] コンピュテーションがつなぐ人とデザイン
 
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
[CEDEC2016] 大規模学習を用いたCGの最先端研究の紹介 - 前半
 
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...
[CHI 2016] SelPh: Progressive Learning and Support of Manual Photo Color Enha...
 
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
視覚デザイン探索のためのクラウドソーシングを活用したパラメタ空間解析(WISS 2014)
 
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
Crowd-Powered Parameter Analysis for Visual Design Exploration (UIST 2014)
 
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
こだわり物理エンジン (Innovative Technologies 2013 @ DC EXPO)
 
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
View-Dependent Control of Elastic Rod Simulation for 3D Character Animation (...
 
Real-Time Example-Based Elastic Deformation (SCA '12)
Real-Time Example-Based Elastic Deformation (SCA '12)Real-Time Example-Based Elastic Deformation (SCA '12)
Real-Time Example-Based Elastic Deformation (SCA '12)
 

Último

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Commit University
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...Daniel Zivkovic
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7DianaGray10
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdfPedro Manuel
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-pyJamie (Taka) Wang
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Juan Carlos Gonzalez
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDELiveplex
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfAijun Zhang
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1DianaGray10
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostMatt Ray
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6DianaGray10
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsSafe Software
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Alexander Turgeon
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Websitedgelyza
 

Último (20)

Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)Crea il tuo assistente AI con lo Stregatto (open source python framework)
Crea il tuo assistente AI con lo Stregatto (open source python framework)
 
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
All in AI: LLM Landscape & RAG in 2024 with Mark Ryan (Google) & Jerry Liu (L...
 
UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7UiPath Studio Web workshop series - Day 7
UiPath Studio Web workshop series - Day 7
 
Nanopower In Semiconductor Industry.pdf
Nanopower  In Semiconductor Industry.pdfNanopower  In Semiconductor Industry.pdf
Nanopower In Semiconductor Industry.pdf
 
20230202 - Introduction to tis-py
20230202 - Introduction to tis-py20230202 - Introduction to tis-py
20230202 - Introduction to tis-py
 
Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?Governance in SharePoint Premium:What's in the box?
Governance in SharePoint Premium:What's in the box?
 
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDEADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
ADOPTING WEB 3 FOR YOUR BUSINESS: A STEP-BY-STEP GUIDE
 
Machine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdfMachine Learning Model Validation (Aijun Zhang 2024).pdf
Machine Learning Model Validation (Aijun Zhang 2024).pdf
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1Secure your environment with UiPath and CyberArk technologies - Session 1
Secure your environment with UiPath and CyberArk technologies - Session 1
 
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCostKubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
KubeConEU24-Monitoring Kubernetes and Cloud Spend with OpenCost
 
UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6UiPath Studio Web workshop series - Day 6
UiPath Studio Web workshop series - Day 6
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
201610817 - edge part1
201610817 - edge part1201610817 - edge part1
201610817 - edge part1
 
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration WorkflowsIgniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024Valere | Digital Solutions & AI Transformation Portfolio | 2024
Valere | Digital Solutions & AI Transformation Portfolio | 2024
 
COMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a WebsiteCOMPUTER 10 Lesson 8 - Building a Website
COMPUTER 10 Lesson 8 - Building a Website
 

[SIGGRAPH 2020] Sequential Gallery for Interactive Visual Design Optimization

  • 1. SEQUENTIAL GALLERY FOR INTERACTIVE VISUAL DESIGN OPTIMIZATION Yuki Koyama, Issei Sato, and Masataka Goto © 2 0 2 0 S I G G R A P H . A L L R I G H T S R E S E R V E D .
  • 2. Sequential Gallery for Interactive Visual Design Optimization Yuki Koyama1, Issei Sato2, and Masataka Goto1 1. National Institute of Advanced Industrial Science and Technology (AIST) 2. The University of Tokyo Initial Optimized Sequential Gallery … …
  • 3. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Overview Background and Target Problem 3
  • 4. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Background: Parametric Visual Design is Everywhere 4 … etc. Procedural designPhoto color enhancement Generative modeling
  • 5. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Background: Parametric Visual Design is Everywhere 5 Problem: Need many trials and errors • E.g., try a different parameter combination, see the result, judge whether it is better or not, and then decide which combination to try next … • This is a high-dimensional search task and can be tedious and time- consuming
  • 6. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Background: Parametric Visual Design is Everywhere Motivation: What computational support is possible? Technical challenge: • Parameters need to be determined based on the user’s preference • It is difficult to fully automate the parameter tweaking process 6
  • 7. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Overview Proposed System: Sequential Gallery 7
  • 8. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 8 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set
  • 9. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 9 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface
  • 10. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 10 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface 2D search subtask #1
  • 11. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 11 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set 2D search subtask #1 Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface
  • 12. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 12 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set 2D search subtask #1 Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface 2D search subtask #2
  • 13. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 13 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set 2D search subtask #2 Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface 2D search subtask #3
  • 14. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 14 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set 2D search subtask #3 Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface 2D search subtask #4
  • 15. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 15 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set 2D search subtask #4 Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface
  • 16. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 16 … Target: n design parameters (e.g., photo enhance) … Output: An optimal parameter set Sequential Gallery: An interactive optimization framework where the user sequentially performs 2D search subtasks via a grid interface
  • 17. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Overview Example Usage 17
  • 18. Photo Color Enhancement (12D) Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)
  • 20. Original photograph Enhanced photograph (after 4 iterations)
  • 21. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Overview Summary of Contributions 21
  • 22. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Summary of Contributions • Novel algorithm: Sequential plane search • A variant of preferential Bayesian optimization (PBO), enabling user-in-the-loop optimization • Can find optimal solutions with fewer iterations than the previous algorithm [Koyama+17] • Interactive system: Sequential Gallery • Use the sequential-plane-search algorithm in combination with a zoomable grid interface • Enable the user to effectively explore the design space and perform the optimization 22 ... ... Sequential plane search Sequential Gallery
  • 23. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Summary of Contributions • Novel algorithm: Sequential plane search • A variant of preferential Bayesian optimization (PBO), enabling user-in-the-loop optimization • Can find optimal solutions with fewer iterations than the previous algorithm [Koyama+17] • Interactive system: Sequential Gallery • Use the sequential-plane-search algorithm in combination with a zoomable grid interface • Enable the user to effectively explore the design space and perform the optimization 23 ... ... Sequential plane search Sequential Gallery
  • 24. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Summary of Contributions • Novel algorithm: Sequential plane search • A variant of preferential Bayesian optimization (PBO), enabling user-in-the-loop optimization • Can find optimal solutions with fewer iterations than the previous algorithm [Koyama+17] • Interactive system: Sequential Gallery • Use the sequential-plane-search algorithm in combination with a zoomable grid interface • Enable the user to effectively explore the design space and perform the optimization 24 ... ... Sequential plane search Sequential Gallery
  • 25. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Problem Definition From Mathematical Viewpoint 25
  • 26. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Problem Definition from Mathematical Viewpoint 26 • Suppose that we have sliders to adjust • Let be the search space and be a set of parameter values • Let be a perceptual preference function (= goodness function) which returns a goodness value • We want to solve an optimization problem: n 𝒳 = [0,1]n x ∈ 𝒳 n g : 𝒳 → ℝ x* = argmax x∈𝒳 g(x) [Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds. Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
  • 27. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Problem Definition from Mathematical Viewpoint 27 • Suppose that we have sliders to adjust • Let be the search space and be a set of parameter values • Let be a perceptual preference function (= goodness function) which returns a goodness value • We want to solve an optimization problem: n 𝒳 = [0,1]n x ∈ 𝒳 n g : 𝒳 → ℝ x* = argmax x∈𝒳 g(x) [Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds. Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
  • 28. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interacting with Goodness Function 28 Absolute assessment should not be used: 
 The user cannot directly answer the function value reliably [Brochu+10; Koyama+18] Relative assessment should be used: 
 The user can answer which option is better among two (or more) options Absolute assessment Relative assessmentA B
  • 29. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interacting with Goodness Function 29 Absolute assessment should not be used: 
 The user cannot directly answer the function value reliably [Brochu+10; Koyama+18] Relative assessment should be used: 
 The user can answer which option is better among two (or more) options Absolute assessment Relative assessmentA B
  • 30. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interacting with Goodness Function 30 Absolute assessment should not be used: 
 The user cannot directly answer the function value reliably [Brochu+10; Koyama+18] Relative assessment should be used: 
 The user can answer which option is better among two (or more) options Absolute assessment Relative assessmentA B Preferential feedback
  • 31. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interacting with Goodness Function 31 Absolute assessment should not be used: 
 The user cannot directly answer the function value reliably [Brochu+10; Koyama+18] Relative assessment should be used: 
 The user can answer which option is better among two (or more) options Absolute assessment Relative assessmentA B Preferential Bayesian optimization (PBO)
 can run using relative assessment (i.e., preferential feedback)
  • 32. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Preferential Bayesian Optimization (PBO) Previous Techniques and Our New Technique 32
  • 33. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Basics:“Standard” Bayesian Optimization (BO) • Is a global “black-box” optimization algorithm • Can find optimal solutions with only a small number of function evaluations • Thus, useful to handle expensive-to-evaluate objective functions • Example applications: hyperparameter tuning for machine learning models [Akiba+, KDD 2019] 33 See [Shahriari+, Proc. IEEE 2016] for details
  • 34. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Preferential BO (PBO) • PBO is an extension of BO, which runs with relative assessment (or preferential feedback), rather than absolute assessment of function values • PBO can find optimal solutions with only a small number of preferential feedbacks • Note: human is expensive-to-query 34 Relative assessmentA B
  • 35. xchosen = argmax x∈{xA ,xB } g(x) xchosen = argmax x∈𝒮 g(x) xchosen = argmax x∈𝒫 g(x) [Brochu+, NIPS 2007] “Pairwise Comparison” Query [Koyama+, SIGGRAPH 2017] “Line Search” Query (Sequential Line Search) [Ours] “Plane Search” Query (Sequential Plane Search) Query type: User task: Notes: Is the first PBO algorithm Needs fewer iterations than [Brochu+ NIPS 2007] Needs even fewer iterations, and have a good compatibility with grid interfaces
  • 36. xchosen = argmax x∈{xA ,xB } g(x) xchosen = argmax x∈𝒮 g(x) xchosen = argmax x∈𝒫 g(x) [Brochu+, NIPS 2007] “Pairwise Comparison” Query [Koyama+, SIGGRAPH 2017] “Line Search” Query (Sequential Line Search) [Ours] “Plane Search” Query (Sequential Plane Search) Query type: User task: Notes: Is the first PBO algorithm Needs fewer iterations than [Brochu+ NIPS 2007] Needs even fewer iterations, and have a good compatibility with grid interfaces
  • 37. xchosen = argmax x∈{xA ,xB } g(x) xchosen = argmax x∈𝒮 g(x) xchosen = argmax x∈𝒫 g(x) [Brochu+, NIPS 2007] “Pairwise Comparison” Query [Koyama+, SIGGRAPH 2017] “Line Search” Query (Sequential Line Search) [Ours] “Plane Search” Query (Sequential Plane Search) Query type: User task: Notes: Is the first PBO algorithm Needs fewer iterations than [Brochu+ NIPS 2007] Needs even fewer iterations, and has a good compatibility with grid interfaces
  • 38. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Sequential Gallery Workflow 38
  • 39. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 40. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 41. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 42. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 43. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm User’s feedback Next search plane
  • 44. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 45. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 46. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 47. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 48. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm User’s feedback Next search plane
  • 49. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 50. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) … 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 51. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) …… 2D search subtask 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 52. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) …… 2D search subtask 2D search subtask 2D search subtask 2-dimensional search subspaces (= search planes) determined by the sequential plane search algorithm
  • 53. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 53
  • 54. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 54
  • 55. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 55
  • 56. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 56
  • 57. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 57
  • 58. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 58
  • 59. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 59
  • 60. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Interface: Zoomable Grid Interface for 2D Search Subtasks 60
  • 61. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Benefits of Zoomable Grid Interface 61 • Allows the user to easily grasp the available options in the 2D subspace by just seeing the grid view • WYSIWYG (What-You-See-Is-What-You-Get); do not need to be aware of raw parameters • Compatibility with the sequential-plane-search task (i.e., 2D search) • Minimum quantization errors (thanks to zooming operations)
  • 62. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Sequential Plane Search How to Determine the Next Plane in Each Iteration 62
  • 63. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Criterion to Select 2D Search Planes We want to select the most effective 2D subspace (= search plane) for the next query • Effectiveness here means the degree to which it is worth observing in the next iteration to find the optimal solution Question: How can we define the effectiveness? 63 ?
  • 64. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Acquisition Function in (Standard) BO • Standard BO uses acquisition functions • An acquisition function evaluates how effective a point is as the next query • The maximizer of the acquisition function is selected as the next point • We extend this point-wise acquisition function to evaluate how effective a search plane is as the next query (next slide) 64 Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions) strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
  • 65. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Acquisition Function in (Standard) BO • Standard BO uses acquisition functions • An acquisition function evaluates how effective a point is as the next query • The maximizer of the acquisition function is selected as the next point • We extend this point-wise acquisition function to evaluate how effective a search plane is as the next query (next slide) 65 Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions) strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
  • 66. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Acquisition Function for Determining the Next Plane • Let be the accumulated preferential data so far obtained • Point-wise acquisition function (used in standard BO): [Note 1] • Plane-wise acquisition function (proposed): • We choose the next plane by solving a maximization problem: [Note 2] 𝒟 apoint (x; 𝒟) aplane (𝒫; 𝒟) = ∫ 𝒫 apoint (x; 𝒟)dS 𝒫next 𝒫next = argmax 𝒫∈𝒳 aplane (𝒫; 𝒟) 66 [Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to estimate the goodness function landscape from the available preferential data. [Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint. apoint 𝒫 x 𝒳
  • 67. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Acquisition Function for Determining the Next Plane • Let be the accumulated preferential data so far obtained • Point-wise acquisition function (used in standard BO): [Note 1] • Plane-wise acquisition function (proposed): • We choose the next plane by solving a maximization problem: [Note 2] 𝒟 apoint (x; 𝒟) aplane (𝒫; 𝒟) = ∫ 𝒫 apoint (x; 𝒟)dS 𝒫next 𝒫next = argmax 𝒫∈𝒳 aplane (𝒫; 𝒟) 67 [Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to estimate the goodness function landscape from the available preferential data. [Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint. apoint 𝒫 x 𝒳
  • 68. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Acquisition Function for Determining the Next Plane • Let be the accumulated preferential data so far obtained • Point-wise acquisition function (used in standard BO): [Note 1] • Plane-wise acquisition function (proposed): • We choose the next plane by solving a maximization problem: [Note 2] 𝒟 apoint (x; 𝒟) aplane (𝒫; 𝒟) = ∫ 𝒫 apoint (x; 𝒟)dS 𝒫next 𝒫next = argmax 𝒫∈𝒳 aplane (𝒫; 𝒟) 68 [Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of in PBO; in short, it is based on the use of Gaussian process regression to estimate the goodness function landscape from the available preferential data. [Note 2]: This maximization problem is differentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint. apoint 𝒫 x 𝒳
  • 69. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Applications: Possible Scenarios and Demonstrations 69
  • 70. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Potential Applications 70 Photo color enhancement Generative modeling Procedural texturing … and many other parametric design scenarios
  • 71. Photo Color Enhancement (12D) Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)
  • 73. Original photograph Enhanced photograph (after 5 iterations)
  • 74. Body Shaping (10D) Using the SMPL model [Loper+15] (the first 10 principal components)
  • 76. “He was of medium height, solidly built, wide in the shoulders, thick in the neck, with a jovial heavy-jawed red face […]” Dashiell Hammett. 1930. The Maltese Falcon.
  • 77. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Evaluation [1/2]: Optimization Performance Comparison 77
  • 78. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Performance Comparison Using Synthetic Functions 78 • Goals: • Evaluate the efficiency of our sequential plane search compared to the previous work [Koyama+17] • Confirm that BO (i.e., the use of the acquisition function) contributes to the optimization performance • How: • Use synthetic objective functions to simulate human responses (shown on the right) • Algorithms to be compared: (next slide) Gaussian (5D/15D) Rosenbrock (10D/20D)
  • 79. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Algorithms to be Compared 79 Baseline 1: SLS Baseline 2: SPS (Random) SPS (Ours) Sequential line search [Koyama+17] Sequential plane search, but the plane is randomly chosen (instead of using BO) Sequential plane search (using BO)
  • 80. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Result: Performance Comparison [1/2] 80 Synthetic objective function: Gaussian function Performance: SLS < SPS (random) < SPS (ours) (Lowerisbetter) 5D 15D
  • 81. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Result: Performance Comparison [2/2] 81 Synthetic objective function: Rosenbrock function (Lowerisbetter) 10D 20D Performance: SLS < SPS (random) < SPS (ours)
  • 82. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Evaluation [2/2]: Informal User Study 82
  • 83. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Informal User Study • Participants: Six students/researchers (five novices and one expert) • Task: Enhance photographs using our system (12D) • Results: • All the participants could successfully perform optimization and find satisfactory results in 5.36 iterations in average (SD = 2.69) • Participants appreciated the grid view since they could get inspiration for possible color variations easily and quickly 83
  • 84. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Discussions Limitations and Future Work 84
  • 85. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Discussions: High Dimensionality • BO (and thus PBO, too) is inherently not good at handling very high-dimensional problems [Wang+16] • We assumed the dimensionality is at most 20 • To overcome the dimensionality issue, application-specific extensions should be made • E.g., for generative modeling (e.g., GANs), [Chiu+, SIGGRAPH 2020] is a possible option to combine with Sequential Gallery 85 C.f., A method specific to generative modeling 
 [Chiu+, SIGGRAPH 2020]
  • 86. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Discussion: Others • Initial plane selection: we randomly choose initial planes, but other strategies are possible (e.g., use Design Gallery [Marks+97] for the first iteration and then start the sequential plane search iterations) • Time-changing preference: we assume that the user’s preference does not change during the iteration, but this is not true in some cases; supporting time-changing preference is an interesting future work • Grid resolution and zooming levels: currently, we manually select the grid resolution and the number of levels, but this may be adjusted dynamically by analyzing the just-noticeable difference (JND) • Prior knowledge: when some prior data is available, we could build a rough approximation of the goodness function and then use it as a prior of the Bayesian inference; this would make the optimization even more efficient 86 Refer to the paper for details
  • 87. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Summary 87
  • 88. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) Summary • Sequential Gallery is an interactive system for user-in-the-loop visual design optimization • Its efficiency is enabled by sequential plane search, which is a new variant of preferential Bayesian optimization and is able to find the solution with only a minimal number of iterations 88 Initial Optimized Sequential Gallery … …
  • 89. Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020) References • [Akiba+, KDD 2019] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proc. KDD ’19. 2623–2631. DOI:https://doi.org/ 10.1145/3292500.3330701 • [Brochu+, NIPS 2007] Eric Brochu, Nando de Freitas, and Abhijeet Ghosh. 2007. Active Preference Learning with Discrete Choice Data. In Proc. NIPS '07. 409–416. http:// papers.nips.cc/paper/3219-active-preference-learning-with- discrete-choice-data.pdf • [Brochu+, SCA 2010] Eric Brochu, Tyson Brochu, and Nando de Freitas. 2010. A Bayesian Interactive Optimization Approach to Procedural Animation Design. In Proc. SCA ’10. 103–112. https://doi.org/10.2312/SCA/SCA10/103-112 • [Chiu+, SIGGRAPH 2020] Chia-Hsing Chiu, Yuki Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao Yue. 2020. Human-in-the-Loop Differential Subspace Search in High-Dimensional Latent Space. ACM Trans. Graph. 39, 4, pp.85:1–85:15 (July 2020). DOI:https://doi.org/ 10.1145/3386569.3392409 • [Koyama+, SIGGRAPH 2017] Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential line search for efficient visual design optimization by crowds. ACM Trans. Graph. 36, 4, Article 48 (July 2017), 11 pages. DOI:https://doi.org/10.1145/3072959.3073598 • [Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds. Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184. https:// arxiv.org/abs/2002.08657 • [Shahriari+, Proc. IEEE (2016)] Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE 104,1(January 2016),148– 175. https://doi.org/10.1109/JPROC.2015.2494218 • [Wang+, JAIR (2016)] Ziyu Wang, Frank Hutter, Masrour Zoghi, David Matheson, and Nando de Feitas. 2016. Bayesian Optimization in a Billion Dimensions via Random Embeddings. J. Artif. Intell. Res. 55 (February 2016), 361– 387. https://doi.org/10.1613/jair.4806 89
  • 90. Sequential Gallery for Interactive Visual Design Optimization Yuki Koyama1, Issei Sato2, and Masataka Goto1 1. National Institute of Advanced Industrial Science and Technology (AIST) 2. The University of Tokyo Initial Optimized Sequential Gallery … …