50120130405018

International Journal of Computer Engineering and Technology (IJCET), ISSN 0976-6367(Print),
INTERNATIONAL JOURNAL OF COMPUTER ENGINEERING &
ISSN 0976 - 6375(Online), Volume 4, Issue 5, September - October (2013), © IAEME

TECHNOLOGY (IJCET)

ISSN 0976 – 6367(Print)
ISSN 0976 – 6375(Online)
Volume 4, Issue 5, September – October (2013), pp. 155-164
© IAEME: www.iaeme.com/ijcet.asp
Journal Impact Factor (2013): 6.1302 (Calculated by GISI)
www.jifactor.com

IJCET
©IAEME

SUBJECTIVE QUALITY ASSESSMENT OF NEW MEDICAL IMAGE
DATABASE
Asst. Prof. Jameelah H. Suad1, Wurood A. Jbara2
1
2

Computer Science, College of Science, AL-Mustansiriyah University, Iraq
Computer Science, College of Science, AL-Mustansiriyah University, Iraq

ABSTRACT
A new medical image database is created in order to perform the subjective evaluation to
obtain the Mean Opinion Score (MOS). Our database contains 100 test medical images (20 reference
images and 5 types of distortions for each reference image). The MOS value for this database has
been obtained as a result of 15 doctors. Observers doctors are specialists in electronic medical
diagnosis carried out about 1650 individual human quality judgments. The collected MOS can be
used for test the effectiveness performance of different visual quality metrics as well as for the
design of new metrics. Also, the designed medical image database provides a lot of samples that can
be used in the tested the efficiency of the numerical observer model to evaluate the medical image
quality in the same way that consistent with human perception significantly.
Keywords: Subjective evaluation, Mean opinion score, Visual quality, Numerical observer.
I. INTRODUCTION
The evaluation of image quality is important for many image processing systems, such as
those for acquisition, compression, restoration, enhancement, reproduction etc. For instance, the goal
of image compression is to reduce the amount of data required to store an image while at the same
time it is ensured the results are of good quality enough; in image enhancement systems, final images
should be of better visual quality than the originals; and taking into account current communication
networks, their images are transported by channels that introduce errors, thus they should be
evaluated to ensure can be worked with the final images they have transported [1].
The goal of image quality assessment (IQA) research is to design algorithms for objective
evaluation of quality in a way that is consistent with subjective human evaluation. By “consistent”
the algorithm’s assessments of quality should be in close agreement with human judgements,

155


regardless of the type of distortion corrupting the image, the content of the image, or strength of the
distortion [2].
The evaluation of image quality may be divided into two classes, subjective and objective
methods. Intuitively one can say that the best judge of quality is the human himself. That is why
subjective methods are said to be the most precise measures of perceptual quality and to date
subjective experiments are the only widely recognized method of judging perceived quality [3]. In
these experiments humans are involved who have to vote for the quality of an image in a controlled
test environment. This can be done by simply providing a distorted image of which the quality has to
be evaluated by the subject. Another way is to additionally provide a reference image which the
subject can use to determine the relative quality of the distorted image[4].
Mostly commonly, the mean opinion scores (MOS) is used where the individual participant’s
scores are averaged to level out individual factors. Popular subjective evaluation methods useful for
image quality evaluation are Double Stimulus Continuous Quality Scale (DSCQS),Double Stimulus
Impairment Scale (DSIS), Single Stimulus Method (SSM) and Two Alternative Forced Choice
(TAFC). However, the subjective methods are normally difficult to implement, expensive, timeconsuming, and impractical in many cases [5].
Another class is objective method, these are automatic algorithms for quality assessment that
could analyses images and report their quality without human involvement. Such methods could
eliminate the need for expensive subjective studies. Objective image quality assessment research
aims to design quality measures that can automatically predict perceived image quality. The most
widely used metrics are mean squared error (MSE), Signal-to-Noise Ratio (SNR), and Peak Signalto-Noise Ratio (PSNR), which exhibit weak performances that has not been in agreement with
perceived quality assessment based on subjective test [6].
In this paper, we present a new medical image database, and then invite a group of specialist
doctors in the e-diagnosis to assess the quality of medical images in the database that was created.
The paper is structured as follows. Section 2 presents the benefits of designed medical image
database. The design of new medical image database present in Section 3. Section 4 shows our
experimental results. And finally Section 5 provides conclusions.
II. BENEFITS OF NEW MEDICAL IMAGE DATABASE
Recently the scientific community has done great efforts to develop and test image and video
quality assessment methods incorporating perceptual measures. Many of the quality metrics
proposed were based on properties of human vision system (HVS) [7]. However, till now there are
no such quality metrics that are able to take into account all peculiarities of HVS. There are several
reasons for that. First, HVS is not well understood yet. Second, it is not clear how to model all
possible image distortion types and levels. Third, people use different image databases to carry out
testing of existing and new quality metrics. Fourth, experiments with appropriate number of
volunteers are needed for assessment of image visual quality and reliable testing of the image visual
quality metrics [8].
Conditions to carry out experiments, methodology to process the results of these experiments,
what is a necessary number of participants, etc., are other questions yet to be answered [7].
Moreover, assessing the quality of medical images is a great challenge because of the lack of
available databases dedicated to medical images in order to use in the design of quality model to
assess the quality of medical images. So we have been forced to created a medical image database
that allows to alleviate some of shortcomings mentioned above and to make a comparison of
different quality metrics integrally for particular groups (subsets) of distortion types. Also, test the
new numerical observer model to evaluate the medical image quality.

156


This database provides, among others, the following opportunities:
1) It allows testing new quality metrics (if their implementation codes are available);
2) The database provides already MOS for medical images;
3) It is possible to consider applicability of quality metrics for particular applications by grouping
experiment data for a given type of distortion(s) and analyzing correlation coefficients for the
tested quality metrics;
4) The database allows determining types of distortions for which a given quality metric performs
poorly, thus showing its drawbacks and, probably, indicating what should be taken into account
for improvement of metric’s performance.
III. DESIGN OF NEW MEDICAL IMAGE DATABASE
The main goal of medical image database is to provide collection of reference medical
images and their distorted images in order to conduct the subjective evaluation procedure intended
for quality assessment of medical image and then use the results in design new numerical observer
model to evaluate the medical image quality. Also, the obtained results can be used in the future to
test and verification of the performance efficiency for the numerical observer model. The main steps
of subjective evaluation are shown in Fig.1:

Medical Reference image
Images
DB

Different Types of
Distortions and
Noise
Distorted
images

Subjective Evaluation by
Averaging Opinions of many
Observers(Doctors) on Image
Visual Quality

Compute the MOS Values

Array of MOS values
Fig.1: Steps of subjective evaluation
3.1 Medical Image Database
The quality of any image database strictly depends on the reference images that are used. The
main strategy is to select the medical images that represent a wide variety of cases of disease. That is,
the images in the database should present different textural characteristics, various percentage of
homogeneous regions, edges, and details. Therefore, we borrowed a large number of reference
medical images from Baghdad Central Hospital. These images are captured using the magnetic
157


resonance technique. The Fig.2.shown the 20 reference medical images which are used in this
database. These images has different cases and objects with various characteristics.

Fig.2: Reference medical images of our database
The size of each reference image is 512×512 pixels. The motivation of including these
images into our database was to provide adequate testing for numerical model intended to assess the
medical image quality.
3.2Distortions and Noise
We aim to have the medical image database contain a wide variety of distorted images that
are generated by using different types of distortions and noise which are based on various fields. In
this database we consider five types of distortions and noise that are important for the most
intensively used and studied image processing applications. Table(1) presents the distortions and
noise which are modeled in our image database.

158


No.
Type 1
Type 2
Type 3
Type 4
Type 5

Table (1): Distortion and noise types.
Correspondence to practical
Type of distortion
situation
Additive Gaussian
Image acquisition, Image
noise
transmission
Blurring
Filtering
JPEG compression
Image compression
Image acquisition, Image
Salt and pepper
transmission
Sharpness
Image Enhancement

As mentioned in Table 1 there are five different types of distortions are:
1. Additive noise is often present in images [9] and it is commonly modeled as a white Gaussian
noise. This type of distortion is included in most of studies of quality metric effectiveness.
This type of distortion is, probably, one of few cases when metrics MSE and PSNR present a
good match with the HVS.
2. blurring is also considered in the proposed database since it is an important type of distortions
often met in practical applications and frequently included in studies dealing with visual
quality metrics. This type produced due to filtering process [2].
3. Image compression is an application widely used in medical image, because medical image
usually compressed before being transferred to the other side during e-diagnosis process.
Therefore, images distorted with lossy compression (JPEG) have been included into our
database. The tasks of evaluating distortions for lossy image compression techniques are of
great interest. Besides, we have included into our database the images compressed by JPEG
[10].
4. Impulse noise (Salt and pepper ), we have used a typically used model of uniformly
distributed impulse noise arises, in particular, due to coding/decoding errors in data
transmission [11].The presence of images affected by impulse noise in the database is
necessary. This might assist to adequately evaluate effectiveness of proposed quality model.
5. Finally, we have added into our database the sharpness images. Sharpness is arguably the
most important medical image quality factor because it determines the amount of detail an
imaging system can reproduce. Sharpness is defined by the boundaries between zones of
different pattern [11].
3.3 Subjective Evaluation
In this section, the main goal to use subjective evaluation procedure in order to assess the
quality of medical images in our database, and then obtain the MOS scores that are produced by
Doctor observers.
The subjective evaluation procedure is a widely used for the assessment of image or video
quality, but it has several obvious disadvantages. It is very tedious, time consume, expensive and
impossible to be executed automatically. The subjective evaluation methods are divided into three
primary categories: the first is Single Stimulus Impairment Scale (SSIS); the second is the Double
Stimulus Impairment Scale (DSIS) and finally the Double Stimulus Continuous Quality Scale
(DSCQS). All of these methods were based on ITU-R Recommendation (International
Telecommunications Union) [12].
The DSIS method is better suited for assessing clearly visible impairments, such as
distortions caused by compression or transmission errors. Therefore, we use a DSIS procedure to
assess the quality of the medical images within our database.
159


The DSIS operates on the five level impairment grading. The reference image is always
shown with the distorted one. Assessment of the images quality refers to the distortion level, not the
absolute image quality. This scale is commonly known as the 5-dgree scale, where 5 equals the
imperceptible level of impairment and 1 equals the very annoying level as explained in Table (2).
Table (2):The DSIS Scale
Quality
5 Excellent
4 Good
3 Fair
2 Poor
1 Bad

Five-degree scale
Impairment
5 Imperceptible
4 Perceptible, but not annoying
3 Slightly annoying
2 Annoying
1 Very annoying

A. Setting of Subjective Test
At the starting of subjective test, we must tack in consideration the ITU-T recommendations
in order to doing the subjective test in suitable environment [13].Here, we summarize the
recommendations of ITU regarding the medical image evaluation:
1. Subjects Number: As stated in ITU recommendations, the number of subjects required to
perform the subjective quality experiment can vary from (5 to 35). The typically number of
subjects is about 15. They should have normal vision, and should preferably be expert in
diagnosis of medical image.
2. Information for Observers: Before carrying out the experiments some information should be
given to the observers. These information consist of type of evaluation approach, the types of
distortion, the ranking scale, duration of test, ..etc. This information should be explained and
given to the observers in document form. The observer should be trained with the training trials
to familiarize them with the mission they will carry out.
3. Environment: Specifications have been established for the room environment, ambient lighting
conditions and viewing distance.
4. Images Display: Randomness in images display is preferred.
Our subjective test was performed in suitable medical environment that meets all the
standards of subjective evaluation at medical laboratory of Baghdad Central Hospital with fifteen of
specialist Doctors in e-diagnosis.
B. Conduct DSIS Protocol
The DSIS protocol is used to produce the MOS values for all distorted medical images in
created database. Thus, a group of Doctor observers are invited to judge the quality of the distorted
medical image. The DSIS needs that number of (typically 15) observers with good experiences in ediagnosis for medical image and the observers must take training part in order to visual adaptation.
In the training part, observers spend about 10-15 minutes in order to adapt with the specialized
lighting conditions. Later, they are by showed the reference and distorted medical images.
In order to that the observer do not suffer from tiredness during the test period, we have
designed a graphical user interface in order to make the evaluation process easy and require no
computer skills from observers. The graphical user interface was designed on a PC running Windows
7, using the Visual Basic 6.0 with Microsoft Access. The graphical user interface during the
subjective testis shown in Fig.3.

160


At the starting of the experiment, the observers enters his/her name and then the multiple
pairs of reference and distorted medical images are showed with the question "what is the
impairment average of reference and distorted medical image".

Fig.3: The graphical user interface for DSIS test.
C. MOS Computation
After completing the subjective test, the observers provided their opinions in the linguistic
terms such as (excellent, good...etc.). These terms must be collected and converted to a numerical
style in order to be analyzed and give accurate estimation for the quality of distorted medical images.
~
The MOS ukj is computed for each distorted medical image as shown below:

1
~
u kj =
N

N

∑u

... (1)

ikj

i =1

Where N represents the observer’s number, uikj is the score of the observer i of the distorted image k
for the distortion type j.
IV. RESULTS AND DISCUSSION
The subjective experiments using DSIS protocol have been conducted in order to obtain the
MOS scores from the observers (Doctors)for all distorted medical images in our database.
After the observer gives his/her opinion, the resultant value is stored in the data file of
database. Table (3) shows the data file database containing values that produced by observers for two
reference medical image and their distorted versions.
Table (3): The data file
Images
1_bluer
1_Gauss
1_JPEG com
1_Sharp
1_S&P
2_bluer
2_Gauss
2_JPEG com
2_Sharp
2_S&P

Distortion
Bluer
Gaussian
JPEG comp.
Sharpness
Salt & pepper
Bluer
Gaussian
JPEG comp.
Sharpness
Salt & pepper

Dr Ali Dr Mayada M Dr Ayad Dr Mayada J Dr Aymen Dr Ayam Dr Alia W. Dr Wasan Dr Salim Dr Ahmed Dr Sura Dr Mohamed Dr Marwan Dr Alaa K. Dr Rsual
3
3
4
4
4
3
3
3
2
3
4
4
3
3
3
1
1
2
2
1
2
1
1
1
1
2
1
1
1
1
3
4
5
5
4
4
3
4
3
4
5
4
4
4
3
4
4
5
5
5
5
4
5
4
4
5
5
4
5
4
1
1
2
1
2
1
1
2
1
2
2
1
1
1
1
2
3
3
4
3
3
3
3
3
3
4
3
3
3
3
1
1
1
1
2
1
2
2
1
1
1
1
1
1
1
4
4
4
4
5
3
3
3
4
3
5
5
4
5
4
5
5
5
5
5
4
5
5
4
4
5
5
5
5
5
1
2
2
2
2
1
1
1
1
2
2
2
1
2
1

161


The Fig. 3 explained the summary of results for all distorted medical images with five
different distortions and noise.
5

4
Salt & pepper
Sharpness

3

JPEG comp.
Gaussian
2

Bluer

1
20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1

Fig.4: Summary of results
After a subjective evaluation is completed, the MOS scores will be computed. Then, the
computed MOS scores for two examples are stored in the new table which is showed in Table (4).
Table (4):MOS scores

Images
1_bluer
1_Gauss
1_JPEG com
1_Sharp
1_S&P
2_bluer
2_Gauss
2_JPEG com
2_Sharp
2_S&P

Distortion
MOS
Bluer
3.2666667
Gaussian
1.2666667
JPEG comp.
3.9333333
Sharpness
4.5333333
Salt & pepper 1.3333333
Bluer
3.0666667
Gaussian
1.2
JPEG comp.
4
Sharpness
4.8
Salt & pepper 1.5333333

MOS scores for all distorted medical image are completed, we analyze the effect of each
distortion type on each reference medical image. Then, the overall effect of each distortion type is
obtained. The Table (5) presented the of each distortion type on each reference medical image with
the overall average.

162


Table (5): The effect of each distortion type for all medical image
im 1
im 2
im 3
im 4
im 5
im 6
im 7
im 8
im 9
im 10
im 11
im 12
im 13
im 14
im 15
im 16
im 17
im 18
im 19
im 20
Ratio=

Bluer
3.26667
3.06667
3.13333
3
2.93333
2.8
3.33333
3
3.06667
2.46667
2.6
2.73333
2.26667
2.73333
2.26667
2.6
2.73333
3.2
2.2
2.2
2.78

Gaussian
1.266667
1.2
1.133333
1.4
1.466667
1.6
1.5333
1.266667
1.6
1.266667
1.33333
1.266667
1.2
1.133333
1.466667
1.466667
1.533333
1.6
1.466667
1.333333
1.376665

JPEG comp. Sharpness
3.93333333 4.5333333
4
4.8666667
4.13333333 4.4666667
4.13333333 4.7333333
4.2
4.5333333
3.93333333
4.4
4.06666667
4.2
3.86666667
4.333
3.66666667 4.2666667
3.86666667 3.9333333
3.46666667
4.2
3.53333333 4.2666667
3.53333333 4.4666667
3.6
4.4
4
4.73
3.73333333
4.477
3.66666667
4.4
4.06666667
4.466
3.6
4.2666667
3.33333333
4.2
3.81666667 4.4069667

Salt & pepper
1.333333333
1.533333333
1.466666667
1.6
1.533333333
2
2.066666667
1.733333333
1.533333333
1.4
1.333333333
1.8
1.466666667
1.733333333
1.533333333
1.866666667
1.8
1.933333333
1.533333333
1.4
1.63

As noted results of effect in Table 5, clear the Sharpness have the highest quality ratio. The
second best quality is Compression followed by the Blurring, and Salt& pepper respectively. The
Gaussian noise have poorest quality.
V. CONCLUSIONS
In this paper, we have created the new medical image database which is contains many of a
distorted images with different types of distortion and perform subjective evaluation to obtain the
MOS values for each distorted image. This database represents an encouraging step towards the
design and testing of dedicated numerical models to assess the quality of medical images. Also, this
database provide wide variety of distorted medical images in order to analyze the suitability of many
known image visual quality metrics for the measure of medical image quality. For future work, we
plan to extended the database with provide the different types of distortion and noise.
VI. REFERENCES
[1] Mireia C. C., Channelized hotelling observer optimization for medical image quality
assessment in lesion detection tasks, Technical Report for the degree of Superior
Telecommunication Engineering, Illinois Institute of Technology, Chicago-USA, July 2011.
[2] Sheikh H.R., Sabir M.F., Bovik A.C., A statistical evaluation of recent full reference image
quality assessment algorithms, IEEE Transactions on Image Processing, vol.15, 2006, pp.
3440 – 3451.
[3] B. Alexandre, P. Le Callet, C. Patrizio, and C. Romain, Quality assessment of stereoscopic
images, EURASIP Journal on Image and Video Processing, 2009.

163


[4] M. Carnec, P. Le Callet, and D. Barba, Objective quality assessment of color images based on
a generic perceptual reduced reference, Signal Processing: Image Communication, vol. 23(4),
2008, pp. 239–256.
[5] Sheikh, H.R.; Bovik A.C., Information theoretic approaches to image quality assessment, In:
Bovik, A.C. Handbook of Image and Video Processing. Elsevier, 2005.
[6] Damon M. Chandler, Seven challenges in image quality assessment: past, present, and future
research, Journal of ISRN Signal Processing, vol. 2013, Article ID 905685, 2013, pp. 53-68.
[7] N. Ponomarenko, V. Lukin, A. Zelensky, K. Egiazarian, M. Carli, F. Battisti, TID2008 - A
database for evaluation of full-reference visual quality assessment metrics, Advances of
Modern Radio electronics, vol. 10, 2009, pp. 30-45.
[8] M. S. Landy and N. Graham, Visual perception of texture, in The Visual Neurosciences, MIT
Press, 2004, pp. 1106–1118.
[9] K. Barner and G. Arce, Nonlinear signal and image processing: theory, methods, and
applications, Electrical Engineering & Applied Signal Processing Series, CRC Press, ISBN10: 0849314275, 2003.
[10] P.Gastaldo, G.Parodi, J.Redi and R.Zunino, No-reference quality assessment of JPEG images
by using CBP neural networks, Artificial Neural Networks – ICANN, vol. 4669, 2007,
pp.564-572.
[11] K.N. Plataniotis and A.N. Venetsanopoulos, Color image processing and applications,
Springer Verlag, ISBN 3-540-66953-1, 2000.
[12] ITU-T Recommendation (P.910), Subjective video quality assessment methods for multimedia
applications, 2000.
[13] AmelaSadagic, Ben Teitelbaum, Dr. Jason Leigh, Prof. Magda El Zarki, Haining Liu,
Measuring the quality of video, Computer Science Department, University College London,
2002.
[14] R. Edbert Rajan and Dr.K.Prasadh, “Spatial and Hierarchical Feature Extraction Based on Sift
for Medical Images”, International Journal of Computer Engineering & Technology (IJCET),
Volume 3, Issue 2, 2012, pp. 308 - 322, ISSN Print: 0976 – 6367, ISSN Online: 0976 – 6375.
[15] Shipra Gupta and Chirag Sharma, “A Novel Technique in Spiht for Medical Image
Compression”, International Journal of Graphics and Multimedia (IJGM), Volume 4, Issue 1,
2013, pp. 1 - 8, ISSN Print: 0976 – 6448, ISSN Online: 0976 –6456.
[16] John Blesswin, Rema and Jenifer Joselin, “A Self Recovery Approach using Halftone Images
for Medical Imagery System”, International Journal of Computer Engineering & Technology
(IJCET), Volume 1, Issue 2, 2010, pp. 133 - 146, ISSN Print: 0976 – 6367, ISSN Online:
0976 – 6375.

164

50120130405018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (9)

Similar to 50120130405018

Similar to 50120130405018 (20)

More from IAEME Publication

More from IAEME Publication (20)

Recently uploaded

Recently uploaded (20)

50120130405018