eTryOn
& an introduction to XR in Fashion
Jim Downing
CEO, Metail
The eTryOn
Project
IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
XR4Fashion Workshop
4th October, 2021
This work is licensed under a
Creative Commons Attribution-ShareAlike 4.0 International License.

Contents
• Intro
• Applications of XR in fashion
• XR advantages
• eTryOn

Virtual try-on 2012-2018

Browzwear
3D garment
simulation
High res
2D ray
trace
render
Draft
3D
scan
3D rigged
scanatar
Pose
detection
from
photo
+
Metail posed
3D
Browzwear
scanatars
EcoShot - offline AR for composite model images

First Shopping & Fashion
Fellow, Snap GHOST
Innovation Lab
Metail x Valentin’s
Studio
eTryOn project
(EC funded)
AR try-on 2020-
Snap codes at the end of the presentation...

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse
Future
Widespread
Established
Nascent

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
In-game
clothing
Physical
Metaverse
Future
Widespread
Established
Nascent

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Virtual
sampling
In-game
clothing
Offline AR try-
on (composite
images / video)
Physical
Metaverse
Future
Widespread
Established
Nascent

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Virtual
sampling
Virtual
showrooming
& B2B sales
Virtual
fashion
Virtual try-
on
Digital
models
In-game
clothing
Offline AR try-
on (composite
images / video)
Physical
Metaverse
AR in-store
AR / VR
product
view
Future
Widespread
Established
Nascent

A chart of XR in fashion
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Virtual
collaborative
design
Virtual
sampling
Virtual
showrooming
& B2B sales
Virtual
fashion
Virtual try-
on
Realtime AR
try-on
(magic
mirror)
Digital
models
In-game
clothing
Offline AR try-
on (composite
images / video)
Physical
Metaverse
AR in-store
AR / VR
product
view
Future
Widespread
Established
Nascent

Differing priorities
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse

Differing priorities
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse
Accuracy
Realism

Differing priorities
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse
Accuracy
Realism
Specialist tools /
platforms
Existing
platforms

Differing priorities
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse
Accuracy
Realism Interactivity
Specialist tools /
platforms
Existing
platforms
Immersion
Inertia from legacy
Ux

Differing priorities
Apparel supply chain:
Brands, manufacturers,
wholesale
Consumer-facing: Social, marketing, retail
Physical
Metaverse
Accuracy
Realism Interactivity
Specialist tools /
platforms
Existing
platforms
Immersion
Inertia from legacy
Ux

Benefit 1: Dematerialisation of
product development
● Fashion’s two big root problems: -
○ Consume-dispose behaviour (“fast fashion”)
○ Making too much stuff with too much risk that there won’t
be enough demand for it (“The inventory problem”)
● Dematerialisation helps because
○ Fewer physical samples, less air freight
○ Faster → easier to respond to issues
○ Faster → develop later, don’t develop things the market
won’t demand
○ Only make designs that will sell → Less overproduction

Benefit 2: Richer engagement with
brand & design
• Advertising oligopoly means high cost of customer acquisition
• → companies looking for cost savings
• Need to engage with consumers where they spend their
attention
• Gen Y / millennials: Social (FB, insta, youtube)
• Gen Z: Social / gaming (TikTok, Snapchat, twitch)
• Deeper engagement in marketing
• → more likely to buy things we’ll love and wear.

Benefit 3: Virtual Fashion as a new
modality
• Gen Z less hung up on physical = authentic
• Fashion for identity and self expression, without the
consumption
• → A sustainable alternative to Fast Fashion?
• E.g. The Fabricant, DressX, DIGITALAX

The eTryOn Project
Background
Consortium
Project timeline
3 applications

The eTryOn Project
Funded under the European Commission Horizon 2020 programme
New HFI
experiences
Creative
Social
Shopping
Modernize the way people
create, consume and experience
fashion items (clothes) by
offering novel Human-Fashion-
Interaction (HFI) applications
https://etryon-h2020.eu/

Consortium
Centre for Research &
Technology Hellas
(CERTH), Greece
Belgium
UK
UK
Switzerland
3D Avatar generation
from images on mobile
● 3D Avatars into 3D CAD and XR
platforms
● 3D garments into XR platforms
● Garment model optimisation and ML
development
● Building XR Apps
● Consumer preference prediction
● Pilots and project oversight
● 3D CAD garments & development
● Planning & running user pilots
● Prediction of consumer preference &
marketing insights

eTryOn Timeline
Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3
2020 2021 2022
User
requirements
Initial
architecture
design
Initial APIs
and
middleware
specification
Application
Pilots start
Pilots finish
Technical
report
Technical
report
Technical
report
Technical
report
https://etryon-h2020.eu/

1 - Designer App
Designers and
Fashion Perspective
Future HFI
Limited
information
on the
fitting
Garment
production
iterations -
> high cost
Fitting to
grayscale,
predefined
& static
avatars
Immersive
VR offering
realism and
movement
of avatars
Current HFI

● Avatar photorealism &
quality
● Animation of fit model
avatars
● Garment animation quality
vs interactivity
● Users have very high bar
for accuracy
● Match convenience of
existing tools
○ Sketches
○ Illustrator
○ Samples
● Predicting market
response
Key Challenges

2 - (Social) DressMeUp App
Influencers and
Fashion issues
Future HFI
Clothes
bought
for a one-
off sharing
-> high
cost
Cost &
materials
limit the
garment
designs
Access to
physical
garments
is required
Digital garment
collections
regardless of
material cost for
one-off sharing
Current HFI

● Avatar accuracy and
rigging
● Super-accurate pose
detection
● Matching light &
environment
● Composition through
image processing & ML
User Photo
3D CAD garment,
simulated on user’s
shape & pose
Composite image
Recommend
garments based on
interest
Key Challenges

3 - Magic Mirror App
Shoppers and
Fashion issues
Future HFI
Finding the
right fashion
item in terms
of size-fit,
colour, style
and quality is
challenging
High return
rates -> high
operational
costs
Online shopping
of garments
similar to buying
books (manual
fitting)
AR app that
simulates the
physical
experience of
trying on
clothes in-store
Current HFI

Social AR: Snapchat Lens
● Avatar accuracy
● Skinned garments lack realism
○ No draping
● Can be difficult to get skinning
weights looking good
○ (new external mesh makes
this easier).
Remaining technical
challenges

● Unity-based mobile app
● Recommendation-drive product
discovery
● More accurate avatar shape
(precalculated)
● Pose detection through ARKit
(better?)
● Better garment sim using
ObiCloth
● Feedback on fit of garment
using realtime heat maps
● Model resolution and
performance
● Optimising garment sim to
accurately reflect garment
● ML prediction of fit (and later
drape)
Magic Mirror App
c.f. Snapchat lens Technical Challenges

XR in Fashion: Challenges &
Opportunities
• Massive social media & gaming
industry investment
• Increasing mobile compute
power & sensors
• Commercial success of AR
advertising in other sectors
(e.g. furniture, make up)
• Adoption of 3D CAD not
widespread & slow adoption
• Cost & effort in creating 3D
content for XR
• Fidelity of material
representation
• Computational cost of garment
draping
• No clear driver from user need
Opportunities Challenges

Questions?
jim@metail.co.uk
www.metail.com
https://www.linkedin.com/in/jimdowninguk/
IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
XR4Fashion Workshop
4th October, 2021