The document discusses challenges in measuring the quality of experience for voice over Wi-Fi (VoWi-Fi) calls in both field and lab testing. It describes a field study conducted by Spirent and SRG to evaluate VoWi-Fi call quality and handovers under various conditions. Key performance indicators for VoWi-Fi quality are identified including MOS scores, packet loss, and call setup times. The advantages of laboratory testing over field testing are presented, along with example test cases and results from the lab evaluating VoWi-Fi quality under different scenarios. Additional considerations for testing VoWi-Fi performance are also outlined.
2. 2Spirent Communications PROPRIETARY AND CONFIDENTIAL
Agenda
Part 1: VoWi-Fi technology landscape
Drivers for Wi-Fi adoption
VoWi-Fi QoE Challenges
Part 2: SRG VoWi-Fi field-based study results
Wi-Fi study background and motivation
Study setup and measurement methodology
VoWi-Fi Field test results
Part 3: Evaluating VoWi-Fi performance in the lab
Advantages of lab-based testing
Evaluating Key Performance Indicators (KPIs) in the lab
Future deployment and test considerations
Audience Q&A
4. 4Spirent Communications PROPRIETARY AND CONFIDENTIAL
VoWi-Fi QoE Challenges
Offloading Algorithms
• When and why?
• When to go back?
Assessing Signal Strength and Integrity
• RSSI measurements before and
during offload
IP Layer
• Best effort delivery
• Access Point variations
Authorization
• ePDG authentication for
unsecured Wi-Fi access
5. 5Spirent Communications PROPRIETARY AND CONFIDENTIAL
KPIs used to measure VoWi-Fi Quality of Experience (QoE)
Voice Quality:
MOS Score – Uplink and Downlink
Packet (Audio) loss percentage
Conversational (Ear-to-mouth) delay
Call reliability:
Percentage of successful handovers in both directions under “expected” circumstances
Percentage of call drop under various ePDG adversarial scenarios (lab-based simulation)
Percentage of successful handovers when access point is loaded
Time taken for call setup (for Wi-Fi originated calls)
Time taken for LTE<-> Wi-Fi handover
6. 6Spirent Communications PROPRIETARY AND CONFIDENTIAL
Part 1: VoWi-Fi technology landscape
Drivers for Wi-Fi adoption
VoWi-Fi QoE Challenges
Part 2: SRG VoWi-Fi field-based study results
Wi-Fi study background and motivation
Study setup and measurement methodology
VoWi-Fi Field test results
Part 3: Evaluating VoWi-Fi performance in the lab
Advantages of lab-based testing
Evaluating Key Performance Indicators (KPIs) in the lab
Future deployment and test considerations
Audience Q&A
7. 7Spirent Communications PROPRIETARY AND CONFIDENTIAL
Background and rationale for the study
Spirent and SRG have collaborated on all-things-VoLTE for the last 15 months
VoWi-Fi is the rage
Although operators are rushing to embrace the technology – sometimes in
advance of VoLTE – there isn’t a clear understanding how the technology
performs
VoWi-Fi is a natural extension of VoLTE, given the interoperability that should
exist between the two voice solutions – also doesn’t hurt that HD Voice is
supported across all 3 RAN components of the operator’s network
8. 8Spirent Communications PROPRIETARY AND CONFIDENTIAL
Study setup and test methodology
Spirent Communications provided Nomad User Experience system
Accuver X-Cal M data collection tool and Accuver XCAP post-processing software
for logging all lower layer parameters
The testing relied on mobile-to-mobile calls – implies that the uplink MOS of one
device can affect the downlink MOS of the other
Testing included VoLTE-VoWi-Fi handovers and vice versa, as well as call quality
with different device configurations (3G only, Wi-Fi- only, etc.)
VoWi-Fi voice quality measured under realistic traffic loading at residential and
public Wi-Fi access points
9. 9Spirent Communications PROPRIETARY AND CONFIDENTIAL
Procedures:
Introduced additional background traffic onto same Wi-Fi access point while performing voice quality test
Observations:
Streaming HD Video (2x) had very little impact on voice quality when the downlink consisted of two video
streams
Field test results: residential Wi-Fi access point
How does the
performance look
when a device using
VoWi-Fi calling
on a residential Wi-Fi
access point is loaded
on the downlink
10. 10Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: residential Wi-Fi access point
How does the
performance look
when a device using
VoWi-Fi calling
on a residential Wi-Fi
access point is loaded
on the uplink
Procedures:
Transferred emails with ~10 MB files attached over same Wi-Fi AP being use to support voice calls
Observations:
Relatively modest amounts of uplink traffic can impact call quality, including no voice detection
As implemented, VoWi-Fi does not support QoS so all traffic is treated as best effort
11. 11Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: public Wi-Fi access point
How does the
performance look
when two
smartphones are
using VoWi-Fi
in a public Wi-Fi
access point
Procedures:
Stationary testing at airport with both smartphones using VoWi-Fi and the same SSID
– network seemed to be lightly loaded
Observations:
Bigger variances in the results and lower scores, despite lightly-loaded network
– VoWi-Fi on both ends increases impact
Quality of network has a major influence on call quality
12. 12Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: public Wi-Fi access point
How does the
performance look
when a device
using VoWiFi calls
another device on
VoWiFi on a public
Wi-Fi Access point
Procedures:
Comparable to airport test but done at San Diego Convention Center – 18,000 attendees for a conference
Observations:
Less variance in the results compared with the airport test
Quality of network has a major influence on call quality – in this case a good thing
13. 13Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: VoWi-Fi to VoLTE handovers
How does the
performance look
when a device using
VoWi-Fi hands over
from a residential
Wi-Fi Access Point
to VoLTE
Procedures:
Stationary device, which is placed near Wi-Fi AP, uses VoLTE; Second device was mobile – started on VoWi-Fi
Walked down the street pushing the “little red wagon” until the call switched to VoLTE
Returned back to Wi-Fi AP and waited for handover to Wi-Fi before walking again of Wi-Fi coverage
Observations:
No impact to MOS Score during handover; also hard to detect handover by listening to the call
Handovers back into VoWi-Fi take longer (no surprise) while outbound handovers aren’t predictable
14. 14Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: VoWi-Fi to VoLTE handovers
How does the network
and device activity
look when a device
switches between
VoLTE and VoWi-Fi
Procedures:
One device configured for 3G voice only; second device configured for VoLTE / VoWi-Fi
Originate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in hand
Continue until handover occurs and then walk back into coverage; repeat as necessary
Observations:
Everything consistent with expectations; lack of QoS with VoWi-Fi is evident
15. 15Spirent Communications PROPRIETARY AND CONFIDENTIAL
Field test results: VoWi-Fi to VoLTE handovers, II
How does the
network and device
activity look when a
device switches
between VoLTE and
VoWi-Fi
Procedures:
One device (S5) configured for 3G voice only; second device (S6) configured for VoLTE / VoWi-Fi
Originate call within Wi-Fi AP coverage and then proceed to walk out of coverage with both phones in hand
Continue until handover occurs and then walk back into coverage; repeat as necessary
Observations:
S5 used UMTS as intended but S6 didn’t use VoWi-Fi, instead using a combination of UMTS and 2G (NB-AMR)
Call drops when we returned to Wi-Fi coverage (repeated with same results)
16. 16Spirent Communications PROPRIETARY AND CONFIDENTIAL
Key observations
The benefits of VoWi-Fi are clear but it isn’t a panacea
VoWi-Fi is as good or as bad as the underlying Wi-Fi network
VoWi-Fi handovers with the LTE network generally worked but there were
some exceptions
Problems, which we attribute to network/IMS registration and device
interoperability, exist
17. 17Spirent Communications PROPRIETARY AND CONFIDENTIAL
Part 1: VoWi-Fi technology landscape
Drivers for Wi-Fi adoption
VoWi-Fi QoE Challenges
Part 2: SRG VoWi-Fi field-based study results
Wi-Fi study background and motivation
Study setup and measurement methodology
VoWi-Fi Field test results
Part 3: Evaluating VoWi-Fi performance in the lab
Advantages of lab-based testing
Evaluating Key Performance Indicators (KPIs) in the lab
Future deployment and test considerations
Audience Q&A
18. 18Spirent Communications PROPRIETARY AND CONFIDENTIAL
Advantages of evaluating VoWi-Fi in the lab
Audio quality issues related to low MOS scores, audio (packet) loss and codec
(AMR-NB/AMR-WB/EVS) negotiation failure
SIP registration issues
ePDG authentication failure
Handover at power levels higher/lower than that mandated by operator test plans
Poor QoE resulting from UE toggling between LTE – Wi-Fi networks
Does not require a wagon
19. 19Spirent Communications PROPRIETARY AND CONFIDENTIAL
Advantages of evaluating VoWi-Fi in the lab
VoLTE
VoWi-FieAP AKA
IPSec tunnel
VoWi-Fi
VoWi-Fi
20. 20Spirent Communications PROPRIETARY AND CONFIDENTIAL
Evaluating VoWi-FI KPIs in the lab
Test scenario Measurable KPIs QoE impact
Voicequality
Test voice quality when the Wi-Fi
power is in the edge condition of
making a decision on handover
from LTE-Wi-Fi or vice-versa and
is toggling between the two
MOS Score
Packet loss percentage
Network latency
Jitter
Lower MOS Score => lower
voice quality
High packet loss =>
Intermittent drops in speech
High network latency =>
Variable delays in speech
Test voice quality in the
presence of loading on the Wi-Fi
access point and tolerance of
UE to larger network delays
Test voice quality of two test
devices against a reference
“soft” client, both on the Uplink
and the Downlink
21. 21Spirent Communications PROPRIETARY AND CONFIDENTIAL
Evaluating VoWi-Fi KPIs in the lab
Test scenario Measurable KPIs QoE impact
Callreliability
Test the reliability of the call
through the handover process
and time for handover by control
of Wi-Fi power over multiple
iterations
Percentage of successful call
completion
Handover time from
LTE – Wi-Fi
Larger handover time =>
periods of voice discontinuity
Lower percentage of call
completion => Increased
dropped calls during handover
Improper response to ePDG
adversarial scenarios => call
drops during handover
Test the reliability of the call
when the UE is sent adversarial
responses by ePDG
Test the reliability of the call
through the handover process
and time for handover by control
of Wi-Fi power over multiple
iterations
22. 22Spirent Communications PROPRIETARY AND CONFIDENTIAL
Test case 1
Test voice quality
between two
devices Device A
and Device B and
measure MOS
score on both
Uplink and
Downlink
Result
Low MOS
score
measured
on Uplink of
Device A
Test Case 1: Voice Quality
Low MOS score on Downlink of Device B in device-to-device call
traced to low Uplink MOS score on Device A
23. 23Spirent Communications PROPRIETARY AND CONFIDENTIAL
Test case 2
ePDG
adversarial
testing scenario
Result
LTE-Wi-Fi
Handover
failure
UE fails to handover and drops call when ePDG issues error code
Test Case 2: ePDG responses
24. 24Spirent Communications PROPRIETARY AND CONFIDENTIAL
Test case 3
Test voice quality when the UE is
toggling between Wi-Fi and
LTE network
Result
Lower MOS Score
Test Case 3: Handover
Packet loss
Root cause identified: Packet loss as a result of device “toggling” between LTE-Wi-Fi
25. 25Spirent Communications PROPRIETARY AND CONFIDENTIAL
Test case 4
Call initiation on Wi-Fi
originated call
Result
Low MOS score
Test Case 4: Codec negotiation
UE1 supporting AMR-WB UE2 supporting only AMR-NB
Root cause identified: Disagreement of codec between UE1 and UE2
27. 27Spirent Communications PROPRIETARY AND CONFIDENTIAL
Bringing field study into the lab
SRG and Spirent partner on evaluating VoWi-Fi QoE in the lab
LTE EPC Configuration
IMS registration call flow
Wi-Fi authentication
ePDG Adversarial test condition
IMS test agent simulation
28. 28Spirent Communications PROPRIETARY AND CONFIDENTIAL
Additional VoWi-Fi QoE considerations
Service request type
Wi-Fi / LTE power level
Capacity/loading on Wi-Fi radio link
Measuring impact on voice quality while introducing fading and noise on radio
interface of Wi-Fi
Testing audio quality and call performance with different classes of access points
(Home, Enterprise, public Wi-Fi hotspots)
Testing performance in the presence of introducing IP impairments on the IP
backhaul
Testing the impact of performance in the presence of interference from other
adjacent bands such as LTE and/or adjacent Wi-Fi hotspots
ANDSF, Hotspot 2.0…
29. 29Spirent Communications PROPRIETARY AND CONFIDENTIAL
Part 1: VoWi-Fi technology landscape
Drivers for Wi-Fi adoption
VoWi-Fi QoE Challenges
Part 2: SRG VoWi-Fi field-based study results
Wi-Fi study background and motivation
Study setup and measurement methodology
VoWi-Fi Field test results
Part 3: Evaluating VoWi-Fi performance in the lab
Advantages of lab-based testing
Evaluating Key Performance Indicators (KPIs) in the lab
Future deployment and test considerations
Audience Q&A