1. Adapting ISO 20462 Softcopy
Quality Ruler Method for On-line
Image Quality Studies
Peter D. Burns, Jonathan B. Phillips1 and Don Williams2
Burns Digital Imaging, 1NVIDIA Corp. and
2
Image Science Associates
Reference:
P. D. Burns, J. B. Phillips and D Williams, Adapting ISO 20462 Softcopy Quality Ruler
Method for On-line Image Quality Studies, Proc. SPIE Vol. 8653, 86530E-1, 2013
IS&T and SPIE Electronic Imaging Symposium, Feb. 2013

2. Outline
1. ISO 20462 Image quality ruler method
2. Extended to softcopy image quality
3. Adaptation for Internet-based image quality
evaluations
• Viewing method
• Image selection and preparation
• Design for judgments
4. Verification study
• Presentation of softcopy ruler (images)
• Previous texture-loss experiment
5. Conclusions and recommendations
• Comparison with ISO softcopy ruler results
IS&T and SPIE Electronic Imaging 2013 2

3. Introduction
ISO 20462 Image quality ruler method
• Set of calibrated reference hardcopy images
• Primary Standard Quality Scale, SQS1, in overall JND (just noticeable
difference) units
• Ruler is a set of prints based on a single scene, varying from ideal to
lower image quality
Extended to softcopy image quality
• Digital Reference stimuli, updated ISO 20462
• Calibrated display and viewing conditions, verified with four labs.
• Secondary Standard Quality Scale, SQS2
Internet-based image quality evaluations
• Our objective: Develop and test method that uses commercial online
presentation
• efficient methods for gathering evaluation data (e.g. crowdsourcing)
that can be related to the ISO method
• when commonly used (uncontrolled) viewing conditions, hardware,
software and lighting are needed
IS&T and SPIE Electronic Imaging 2013 3

4. ISO softcopy ruler method
Ruler image is adjusted until it matches test image on right
• GUI software used slider
Updated
by slider
movement
IS&T and SPIE Electronic Imaging 2013 4

5. Previous work: ISO softcopy ruler
application for texture blur
Measurement of image quality degradation due to loss of texture,
caused by noise-cleaning image processing
• Ruler varied in image sharpness
• test scenes varied in texture loss
Memorial Art Gallery, Flowers, Girl and Mountain.
Scenes used in their original form (1088 x 816, or 1253 x 834 pixels),

6. Previous work: ISO softcopy ruler
application for texture blur
Noise-cleaned test images compared to ruler images varying in
sharpness to obtain SQS2 JNDs for each test image
Ruler, Standard Quality Scale 30 Ruler, Standard Quality Scale 15
noise-cleaned sample treatment
texture loss level 6
IS&T and SPIE Electronic Imaging 2013 6

7. Adaptation of ISO Method for On-line Evaluations
Presentation and Image Selection, Preparation
1. Presentation
• Internet-browser based
• Commercial online presentation, Survey Monkey
– Likely maximum display size and resolution
• Presentation of images without cropping or scaling
– Image file size limitations
• Static presentation of ruler, rather than dynamic slider-controlled
– Button selection with anchors (two sets)
2. Image selection and preparation
• Preserve image microstructure
– Crop selected ruler and test images
• Use previous texture-loss study and images
– Ruler based on range of image sharpness
– Texture loss due to noise-cleaning image processing
IS&T and SPIE Electronic Imaging 2013 7

8. Adaptation of ISO Method for On-line Evaluations
1. Presentation
• Button selection with two anchor reference images
• Each test image was rated twice, compared with two sets of
references [0 (low), 15] and [15, 30 (high)] SQS2 values
IS&T and SPIE Electronic Imaging 2013 8

9. Adaptation of ISO Method for On-line Evaluations
2. Image selection and preparation
• Selection of scenes based on JND sensitivity found in previous work
• Texture loss due to noise-cleaning image processing
(1088 x 816, or 1253 x 834 pixels),
IS&T and SPIE Electronic Imaging 2013 9

10. Adaptation of ISO Method for On-line Evaluations
2. Image selection and preparation
• Commercial service: Survey Monkey
• URL vs. upload
• Display field aimed at display of 1600 x 900 pixel
Original scenes in previous work (1088 x 816, or 1253 x 834 pixels)
Cropped scenes in current experiment (525 x 330 pixels per scene)
IS&T and SPIE Electronic Imaging 2013 10

11. Study: Image Quality Due to Noise-Cleaning
1. Same scenes and treatments that were used in previous verification
study
2. Initial subjective results with 21 overall quality buttons
• Observers with and without involvement in the imaging industry. (N=21)
3. Second study with 9 overall quality buttons for the observers to
select
• Similar observer population, though not the same observers (N=11)
• Experienced observers in image quality evaluation (N=20)
Rate the quality of the test image based on the position between the left and right
reference images
IS&T and SPIE Electronic Imaging 2013 11

12. ISO Softcopy Results for the Four Scenes
• From ISO method study rsults
• SQS2 JNDs for the 4 scenes and noise-cleaning treatments levels for
calibrated reference points from previous study.
• Error bars are standard errors, N=17
IS&T and SPIE Electronic Imaging 2013 12

13. Results 1: Adapted Online Method
Inferred SQS2 JNDs for the 4 scenes and treatments levels for 21
buttons of overall quality
• 21 observers
IS&T and SPIE Electronic Imaging 2013 13

14. Results 2: Adapted Online Method
Inferred SQS2 JNDs for the 4 scenes and treatments levels for 9
buttons of overall quality
• 11 observers
IS&T and SPIE Electronic Imaging 2013 14

15. Results 3: Adapted Online Method
Comparison of our inferred SQS2 JND results with those of the
reference experiment (previous work) with calibrated SQS2 JNDs.
Averaged over observers and scenes.
IS&T and SPIE Electronic Imaging 2013 15

16. Summary
1. Softcopy ruler method has been adapted for unconstrained on-line
image quality studies
• Static ruler-anchor display vs. slider-ruler selection
• Button selection of image quality rating
• Cropped scenes
2. Results show similar scene-differences in IQ sensitivity
3. 9- and 21-button results were similar, implying a quantizing of ruler
scale response, as intended
4. Pooled results data for all four scenes were consistent with results
from the original softcopy ruler study
5. Uncontrolled, on-line results showed more sensitivity to the overall
image quality degradation (rather than less, as anticipated)
• Possibly due to cropping most salient image areas, or
uncontrolled viewing distance
6. Conclusion: Internet-based image quality evaluations can be related
to ISO 20462 standard if common reference images are used.
IS&T and SPIE Electronic Imaging 2013 16

17. SUBJECTIVE IMAGE
QUALITY EVALUATION
(Not presented at EI 2013)
Image quality ruler

18. ISO 20462 Part 3 – The Quality Ruler
Method
“Photography—Psychophysical experimental methods to
estimate image quality—Part 3: Quality ruler method”
Standardization of anchored pair method based on JND units
Substitute length units with an image set having a series of sharpness
levels
Quality ruler approach allows for repeatable and definable results
High quality (sharp) Low quality (blurry)
Compare test image to images on ruler and pick best match on ruler

19. ISO 20462 Part 3 – The Quality Ruler
Method
Standard Quality
Scale (SQS)
32 24 16 8 0
Excellent Very Good Fair Poor Not Worth
Good Keeping
Standard Reference
Stimuli (SRS) or Digital
Reference Stimuli (DRS)
JND = 75:25 split in paired comparison;
JNDs = advertisable differences;
JNDs = interval of image quality categories

20. Softcopy Quality Ruler
Softcopy quality ruler method
based on ISO 20462 Part 3
• 30" Apple Cinema Display monitor
• Controlled viewing distance with
chin/head rest*
– 25", 34", or 43"
• Simultaneous viewing of ruler and
test images
• Quality ruler reference stimuli span
30 just noticeable difference values
in sharpness
* e.g., University of Houston College of
Optometry (UHCO) HeadSpot

21. Softcopy Ruler Method Development
Raw Demosaic Linear sRGB Camera MTF
image WB + CCM image Response
Monitor displays PSF image captured Monitor MTF
Single bright pixel and analyzed Response
System MTF =
Camera MTF * Monitor MTF * Downsampling MTF ( 1MP)

22. 21 Quality Rulers
• 21 scene choices
• 30 ruler images per scene
• SQS from 2 to 31 in increments of approximately 1 JND
• Matlab code for running quality ruler
Soon to be officially located at:
http://www.imaing.org/ist/resources/standards/standards.cfm
Secondary location:
http://www.aptina.com/ImArch/

23. Softcopy Quality Ruler Screen
Screen Capture
In situ

24. Softcopy Quality Ruler Method Workflow
One-time setup Running the tests
Obtain ruler Set up viewing Obtain Obtain test Perform Analyze data
images environment software tools images psychovisual and report
studies results
Fixed sets of Specifications Software Specifications Specifications Tools for
digital anchor on viewing applications to on obtaining on subject analyzing the
images, i.e. distance and display images test images at selection, vision results and
- 21 scenes ambient and collect any time of the screening, report
- 30 images per lighting. viewer year and number of template
scene Create ICC responses. around the subjects, etc.
profile for the globe in a
monitor. timely manner