Coverage and Introduction to UVM BMS College of Engineering June 2016

1. +
Coverage and
Introduction to UVM
BMS College of Engineering
June 2016
Shivoo Koteshwar
Director, MediaTek
shivoo.koteshwar@gmail.com
www.facebook.com/shivoo.koteshwar

2. Shivoo
+
What is Verification?
Verification is the process of verifying the transformation
steps in the design flow are executed correctly.
Algorithm
Architecture/
Spec RTL Gate GDSII ASIC
End
produc
t
Idea
Product
Acceptance
Test
Transformations
C-Model
Spec
Acceptance
Review
Simulation/
Code Review
Formal
Functional/
Timing
Verification
ATE
Sign-Off
Review
2

3. Shivoo
+
Verification Goal
n Ensure full conformance with specification:
n Must avoid false positives (untested functionalities)
???Pass
Fail
Good Bad(bug)
RTL code
√Tape out!
Debug
testbench
Debug
RTL code
Testbench
Simulation
result
False positive
results in
shipping a bad
design
How do we achieve this goal?
3
3

4. Shivoo
+ 4
Verification
Types
4

5. Shivoo
+
Simulators
n Simulators are the most common and familiar verification tools. They are named
simulators because their role is limited to approximating reality.
n A simulation is never the final goal of a project. The goal of all hardware design
projects is to create real physical designs that can be sold and generate profits.
n Simulators attempt to create an artificial universe that mimics the future real design.
This lets the designers interact with the design before it is manufactured and correct
flaws and problems earlier
n Simulators are only approximations of reality
n Many physical characteristics are simplified - or even ignored - to ease the simulation task.
For example, a digital simulator assumes that the only possible values for a signal are ‘0’,‘1’,
X, and Z. However, in the physical and analog world, the value of a signal is a continuous: an
infinite number of possible values. In a discrete simulator, events that happen
deterministically 5 ns apart may be asynchronous in the real world and may occur randomly
n Simulators are at the mercy of the descriptions being simulated
n The description is limited to a well-defined language with precise semantics. If that
description does not accurately reflect the reality it is trying to model, there is no way for you
to know that you are simulating something that is different from the design that will be
ultimately manufactured. Functional correctness and accuracy of models is a big problem as
errors cannot be proven not to exist.
5

6. Shivoo
+
Stimulus and Response
n Simulation requires stimulus
n Simulators are not static tools. A static verification tool performs its task on
the design without any additional information or action required by the user.
For example, linting tools are static tools. Simulators, on the other hand,
require that you provide a facsimile of the environment in which the design
will find itself.This facsimile is often called a testbench, stimulus.
n The testbench needs to provide a representation of the inputs observed by
the design, so the simulator can emulate the design’s responses based on its
description.
n The simulation outputs are validated externally, against design
intents.
n The other thing that you must not forget is that simulators have no knowledge
of your intentions. They cannot determine if a design being simulated is
correct. Correctness is a value judgment on the outcome of a simulation that
must be made by you, the designer.
n Once the design is submitted to an approximation of the inputs from its
environment, your primary responsibility is to examine the outputs produced
by the simulation of the design’s description and determine if that response
is appropriate.
6

7. Shivoo
+
Event Driven Simulation
n Simulators are never fast enough
n They are attempting to emulate a physical world where electricity travels at the speed of light
and transistors switch over one billion times in a second. Simulators are implemented using
general purpose computers that can execute, under ideal conditions, up to 100 million
instructions per second
n The speed advantage is unfairly and forever tipped in favor of the physical world
n Outputs change only when an input changes
n One way to optimize the performance of a simulator is to avoid simulating something that
does not need to be simulated.
n Figure shows a 2-input XOR gate. In the physical world, if the inputs do not change (a), even
though voltage is constantly applied, output does not change Only if one of the inputs change
(b) does the output change
n Change in values, called events, drive the simulation process
n The simulator could choose to continuously execute this model, producing the same output
value if the input values did not change.
n An opportunity to improve upon that simulator’s performance becomes obvious: do not
execute the model while the inputs are constants. Phrased another way: only execute a model
when an input changes. The simulation is therefore driven by changes in inputs. If you define
an input change as an event, you now have an event-driven simulator
7

8. Shivoo
+
Cycle Based Simulation
n Cycle-based simulations have no timing information
n This great improvement in simulation performance comes at a cost: all timing
and delay information is lost. Cycle-based simulators assume that the entire
design meets the set-up and hold requirements of all the flip-flops.
n When using a cycle-based simulator, timing is usually verified using a static
timing analyzer
n Cycle-based simulators can only handle synchronous circuits
n Cycle-based simulators further assume that the active clock edge is the only
significant event in changing the state of the design. All other inputs are
assumed to be perfectly synchronous with the active clock edge.Therefore,
cycle-based simulators can only simulate perfectly synchronous designs
n Anything containing asynchronous inputs, latches, or multiple-clock domains
cannot be simulated accurately.,The same restrictions apply to static timing
analysis.Thus, circuits that are suitable for cycle-based simulation to verify
the functionality, are suitable for static timing verification to verify the timing
8

9. Shivoo
+
Co-Simulators
n To handle the portions of a design that do not meet the requirements for cycle-based simulation,
most simulators are integrated with an event-driven simulator
n As shown, the synchronous portion of the design is simulated using the cycle-based algorithm,
while the remainder of the design is simulated using a conventional event-driven simulator
n Both simulators (event-driven and cycle-based) are running together, cooperating to simulate
the entire design
n Other popular co-simulation environments provide VHDL and Verilog,
HDL and C, or digital and analog co-simulation
9

10. Shivoo
+
Coverage
1. Code Coverage
2. Functional Coverage
3. Statement Coverage
4. Path Coverage
5. Expression Coverage
10

11. Shivoo
+
Code Coverage
n VHDL simulation tools can automatically calculate a metric called code coverage
(assuming you have licenses for this feature).
n Code coverage tracks what lines of code or expressions in the code have been
exercised.
n Code coverage cannot detect conditions that are not in the code
n Code coverage on a partially implemented design can reach 100%. It cannot detect
missing features and many boundary conditions (in particular those that span more
than one block)
n Code coverage is an optimistic metric. In combinational logic code in an HDL, a
process may be executed many times during a given clock cycle due to delta cycle
changes on input signals. This can result in several different branches of code being
executed. However, only the last branch of code executed before the clock edge
truly has been covered.
n Hence, code coverage cannot be used exclusively to indicate we are done testing.
11

12. Shivoo
+
Functional Coverage
n Functional coverage is code that observes execution of a test plan. As such, it is code you write to track
whether important values, sets of values, or sequences of values that correspond to design or interface
requirements, features, or boundary conditions have been exercised
n Specifically, 100% functional coverage indicates that all items in the test plan have been tested. Combine
this with 100% code coverage and it indicates that testing is done
n Functional coverage that examines the values within a single object is called either point coverage or
item coverage
n One relationship we might look at is different transfer sizes across a packet based bus. For example, the test plan may
require that transfer sizes with the following size or range of sizes be observed: 1, 2, 3, 4 to 127, 128 to 252, 253, 254, or
255.
n Functional coverage that examines the relationships between different objects is called cross coverage.
An example of this would be examining whether an ALU has done all of its supported operations with
every different input pair of registers
n VHDL’s Open Source VHDL Verification Methodology (OSVVM) provides a package, CoveragePkg, with a
protected type that facilitates capturing the data structure and writing functional coverage
n Functional Coverage provides additional supporting data that the design is tested. It’s a supplement to
Primitive testing directed, algorithmic, file based, or constrained random test methods
12

13. Shivoo
+
Statement Coverage
n Statement or block coverage measures how much of the total lines of code were
executed by the verification suite. A graphical user interface usually lets the user
browse the source code and quickly identify the statements that were not executed
n Add testbenches to execute all statements
n Two out of the eight executable statements - or 25 percent - were not executed. To bring the statement coverage
metric up to 100 percent, a desirable goal, it is necessary to understand what conditions are required to cause
the execution of the uncovered statements
n In this case, the parity must be set to either ODD or EVEN. Once the conditions have been determined, you must
understand why they never occurred in the first place. Is it a condition that can never occur? Is it a condition that
should have been verified by the by the existing verification suite? Or is it a condition that was forgotten?
n It is normal for some statements to not be executed ?
n If it is a condition that can never occur, the code in question is effectively dead: it will never be executed.
Removing that code is a definite option. However, a good defensive coder often includes code that is not
meant to be executed. Do not measure coverage for code not meant to be executed.
13

14. Shivoo
+
Path Coverage
n There is more than one way to execute a sequence of statements. Path coverage measures all
possible ways you can execute a sequence of statements.The code below has four possible
paths: the first if statement can either be true or false. So can the second.
n To verify all paths through this simple code section, it is necessary to execute it with all possible
state combinations for both if statements: false-false, false-true, true-false, and true-true
n The current verification suite, although it offers 100 percent statement coverage, only
offers 75 percent path coverage through this small code section
n Again, it is necessary to determine the conditions that cause the uncovered path to be
executed
n In this case, a testcase must set the parity to neither ODD nor EVEN and the number of
stop bits to two. Again, the important question one must ask is whether this is a condition
that will ever happen, or if it is a conditions that was overlooked
n Limit the length of statement sequences as Code coverage tools give up measuring path
coverage if their number is too large in a given code sequence
n Reaching 100 percent path coverage is very difficult
14

15. Shivoo
+
Expression Coverage
n If you look closely at the sample code, you notice that there are two
mutually independent conditions that can cause the first if statement to
branch the execution into its then clause: parity being set to either
ODD or EVEN. Expression coverage, as shown, measures the various
ways paths through the code are executed. Even if the statement
coverage is at 100 percent, the expression coverage is only at 50
percent
n Once more, it is necessary to understand why a controlling term of an
expression has not been exercised. In this case, no testbench sets the
parity to EVEN. Is it a condition that will never occur? Or was it another
oversight?
n Reaching 100 percent expression coverage is extremely difficult.
15

16. Shivoo
+
What Does 100 Percent Coverage
Mean?
n Completeness does not imply correctness:
n Code coverage indicates how thoroughly your entire verification suite exercises the source code. I
does not provide an indication, in any way, about the correctness of the verification suite
n Code coverage should be used to help identify corner cases that were not exercised by the
verification suite or implementation-dependent features that were introduced during the
implementation
n Code coverage is an additional indicator for the completeness of the verification job. It can help
increase your confidence that the verification job is complete, but it should not be your only
indicator.
n Code coverage lets you know if you are not done: Code coverage indicates if the verification
task is not complete through low coverage numbers. A high coverage number is by no means an
indication that the job is over
n Some tools can help you reach 100% coverage:There are testbench generation tools that
automatically generate stimulus to exercise the uncovered code sections of a design
n Code coverage tools can be used as profilers:When developing models for simulation only,
where performance is an important criteria, code coverage tools can be used for profiling.The
aim of profiling is the opposite of code coverage.The aim of profiling is to identify the lines of
codes that are executed most often.These lines of code become the primary candidates for
performance optimization efforts
16

17. Shivoo
+
Coverage-Driven Verification
Phases of random stimulus based verification
Time
Goal
Build verification
environment
Broad-Spectrum
Verification
Preliminary
Verification
Difficult to reach
Corner-case
Verification
Corner-case
Verification
%Coverage
customized
directed tests can
take care of this
random stimulus
17

18. Shivoo
+
Verification Technology
18

19. Shivoo
+
Testbench
Testbench Environment/Architecture
RTL
Monitor
TransactorSelf Check
Observes
data
from DUT
Identifies
transactions
Checks
correctness
Coverage
Driver
Generator
DUT
Transactor
Configure
Interfaces
Configures testbench and DUT
Drive DUT
Executes
transactions
Creates
random
transactions
Checks
completeness
Testcase
19

20. Shivoo
+
Testbench Considerations: Abstraction
Monitor
TransactorSelf Check
Coverage
Driver
Generator
DUT
Transactor
Configure
Signals
(interface)
Device Drivers
and Monitors
Transactors
Creates
Test Scenarios
Testcase
Create individual
Test Cases
20

21. Shivoo
+
NOTE
n Test can reach 100% functional coverage without reaching
100% code coverage
n This indicates the design contains untested code that is not
part of the test plan
n This can come from an incomplete test plan, extra
undocumented features in the design, or case statement
others branches that do not get exercised in normal
hardware operation
n Untested features need to either be tested or removed
n As a result, even with 100% functional coverage it is still a
good idea to use code coverage as a fail safe for the test
plan.
Test Done = Test Plan Executed and All Code Executed
21

22. Shivoo
+
Accelera
n Accellera Systems Initiative is an independent, not-for profit
organization dedicated to create, support, promote, and
advance system-level design, modeling, and verification
standards for use by the worldwide electronics industry
n www.accelera.org
22

23. Shivoo
+
Verification Languages
n Verification languages can raise the level of abstraction
n Best way to increase productivity is to raise the level of abstraction used to
perform a task
n VHDL andVerilog are simulation languages, not verification
languages
n Verilog was designed with a focus on describing low-level hardware
structures. It does not provide support for high-level data structures or object-
oriented features
n VHDL was designed for very large design teams. It strongly encapsulates all
information and communicates strictly through well-defined interfaces
n Very often, these limitations get in the way of an efficient implementation of a
verification strategy. Neither integrates easily with C models
n This creates an opportunity for verification languages designed to overcome
the shortcomings of Verilog and VHDL. However, using verification language
requires additional training and tool costs
n Proprietary verification languages exist
n e/Specman from Verisity, VERA from Synopsys, Rave from Chronology etc
23

24. Shivoo
+
Methodologies
n OVM
n Open Verification Methodology
n Derived mainly from the URM (Universal Reuse Methodology) which was, to a large part, based on the eRM (e
Reuse Methodology) for the e Verification Language developed by Verisity Design in 2001
n The OVM also brings in concepts from the AdvancedVerification Methodology (AVM)
n System Verilog
n RVM
n Reference Verification Methodology
n complete set of metrics and methods for performing Functional verification of complex designs
n The SystemVerilog implementation of the RVM is known as the VMM
n OVL
n Open Verification Language
n OVL library of assertion checkers is intended to be used by design, integration, and verification engineers to check
for good/bad behavior in simulation, emulation, and formal verification.
n Acceleraa - http://www.accellera.org/downloads/standards/ovl/
n UVM
n Standard Universal Verification Methodology
n Accelera - http://www.accellera.org/downloads/standards/uvm
n System Verilog
n OS-VVM
n VHDL
n Acceleraa
24

25. Shivoo
+
OS-VVM (http://osvvm.org)
n At its lowest level, Open Source VHDL Verification
Methodology (OSVVM) is a set of VHDL packages that
simplify implementation of functional coverage and
randomization.
n OSVVM uses these packages to create an Intelligent
Coverage verification methodology that is a step ahead of
other verification methodologies, such as SystemVerilog’s
UVM
25

26. Shivoo
+
UVM : Universal Verification
Methodology
n UVM (UniversalVerification Methodology) is a
SystemVerilog language based Verification methodology
n UVM consists of a defined methodology for architecting
modular testbenches for design verification.
n UVM has a library of classes that helps in designing and
implementing modular testbench components and stimulus.
This enables re-using testbench components and stimulus
within and across projects, development of Verification IP,
easier migration from simulation to emulation etc.
n Relies on strong, proven industry foundations .The core of its
success is adherence to a standard (i.e. architecture, stimulus
creation, automation, factory usage standards etc.)
26

27. Shivoo
+
Goal of UVM: Automation
n Following can be automated using UVM
n Coverage Driven Verification (CDV) environments
n Automated Stimulus Generation
n Independent Checking
n Coverage Collection
27

28. Shivoo
+
Testbench Architecture
SV Testbench Architecture
UVM Testbench Architecture
28

29. Shivoo
+
Key elements of UVM
• Syntax
• RTL
• OOP
• Class
• Interface
System
Verilog
Language
• Constrained Random
• Coverage Driven
• Transaction Level
• Sequences
• Scoreboards
Verification
Concepts
• Base Classes
• Use Cases
• Configuration-
db
• Phases
Methodology
29

30. Shivoo
+
Note
n System Verilog Language syntax & semantics are pre-
requisite
n All System Verilog experience is directly relevant for UVM
(design/RTL, AVM,VMM, etc.)
n But be aware the verification part of language is much bigger
than that used for design!
n Design: RTL, Blocks, Modules,Vectors, Assignments, Arrays etc.
n Verification: Signals, Interfaces Clocking-block, scheduling,
functions, tasks, OOP, class, random constraint coverage, queues
etc.
n All verification experience is directly transferrable to UVM
30

31. Shivoo
+
UVM Testbench Architecture
31

32. Shivoo
+
UVM Testbench Architecture
32
n The UVM Testbench typically instantiates the Design under Test
(DUT) module and the UVM Test class, and configures the
connections between them.
n If the verification collaterals include module-based components,
they are instantiated under the UVM Testbench as well.
n The UVM Test is dynamically instantiated at run-time, allowing the
UVM Testbench to be compiled once and run with many different
tests.

33. Shivoo
+
UVM Testbench Architecture
33
n The UVM Test is the top-level UVM Component in the UVM
Testbench.
n The UVM Test typically performs three main functions:
n Instantiates the top-level environment
n Configures the environment (via factory overrides or the configuration
database) and
n Applies stimulus by invoking UVM Sequences through the environment to
the DUT.
n Typically, there is one base UVM Test with the UVM Environment
instantiation and other common items.Then, other individual tests
will extend this base test and configure the environment differently
or select different sequences to run.

34. Shivoo
+
UVM Testbench Architecture
34
n The UVM Environment is a hierarchical component that groups
together other verification components that are interrelated.
n Typical components that are usually instantiated inside the UVM
Environment are UVM Agents, UVM Scoreboards, or even other
UVM Environments.The top-level UVM Environment encapsulates
all the verification components targeting the DUT.
n For example: In a typical system on a chip (SoC) UVM Environment,
you will find one UVM Environment per IP (e.g., PCIe Environment,
USB Environment, Memory Controller Environment, etc.).
Sometimes, those IP Environments are grouped together into Cluster
Environments (e.g., IO Environment, Processor Environment, etc.)
that are grouped together eventually in the to- level SoC
Environment.

35. Shivoo
+
UVM Testbench Architecture
35
n The UVM Scorecard’s main function is to check the behavior of a
certain DUT.
n The UVM Scoreboard usually receives transactions carrying inputs
and outputs of the DUT through UVM Agent analysis ports
(connections are not depicted in Figure), runs the input transactions
through some kind of a reference model (also known as the
predictor) to produce expected transactions, and then compares the
expected output versus the actual output.
n There are different methodologies on how to implement the
scoreboard, the nature of the reference model, and how to
communicate between the scoreboard and the rest of the testbench.

36. Shivoo
+
UVM Testbench Architecture
36
n The UVM Sequencer serves as an arbiter for controlling transaction
flow from multiple stimulus sequences. More specifically, the UVM
Sequencer controls the flow of UVM Sequence Items transactions
generated by one or more UVM Sequences.
n The UVM Sequence is an object that contains a behavior for generating
stimulus. UVM Sequences are not part of the component hierarchy.
n UVM Sequences can be transient or persistent. A UVM Sequence instance can
come into existence for a single transaction, it may drive stimulus for the
duration of the simulation, or anywhere in-between.
n UVM Sequences can operate hierarchically with one sequence, called a
parent sequence, invoking another sequence, called a child sequence.
n To operate, each UVM Sequence is eventually bound to a UVM Sequencer.
Multiple UVM Sequence instances can be bound to the same UVM Sequencer.

37. Shivoo
+
UVM Testbench Architecture
37
n The UVM Agent is a hierarchical component that groups together other
verification components that are dealing with a specific DUT interface
(shown in fig below)
n A typical UVM Agent includes a UVM Sequencer to manage stimulus
flow, a UVM Driver to apply stimulus on the DUT interface, and a UVM
Monitor to monitor the DUT interface. UVM Agents might include other
components, like coverage collectors, protocol checkers, a TLM
(Transaction Level Model), etc.

38. Shivoo
+
UVM Testbench Architecture
38
n The UVM Driver receives individual UVM Sequence Item transactions
from the UVM Sequencer and applies (drives) it on the DUT Interface.
Thus, a UVM Driver spans abstraction levels by converting transaction-
level stimulus into pin-level stimulus. It also has a TLM port to receive
transactions from the Sequencer and access to the DUT interface in
order to drive the signals.
n The UVM Monitor samples the DUT interface and captures the
information there in transactions that are sent out to the rest of the UVM
Testbench for further analysis.Thus, similar to the UVM Driver, it spans
abstraction levels by converting pin-level activity to transactions. In
order to achieve that, the UVM Monitor typically has access to the DUT
interface and also has a TLM analysis port to broadcast the created
transactions through.
n The UVM Monitor can perform internally some processing on the transactions
produced (such as coverage collection, checking, logging, recording, etc.) or
can delegate that to dedicated components connected to the monitor's
analysis port.

39. Shivoo
+
UVM Class Library
39
n The UVM Class Library provides all the building blocks you need to
quickly develop well-constructed, reusable, verification components
and test environments.
n The library consists of base classes, utilities, and macros.
n Components may be encapsulated and instantiated hierarchically and
are controlled through an extendable set of phases to initialize, run, and
complete each test.These phases are defined in the base class library,
but can be extended to meet specific project needs.
n The advantages of using the UVM Class Library include:
n A robust set of built-in features
n Correctly-implemented UVM concepts
n Also provides various utilities to simplify the development and use of
verification environments.

40. Shivoo
+
UVM Class Library
40

41. Shivoo
+
Origin of UVM
41
Reference Verification Methodology
E Reuse Methodology
Universal Reuse Methodology
Advanced Verification Methodology
Verification Methodology Manual
Open Verification Methodology
41

42. Shivoo
+
Advantages of UVM
n Modularity and Reusability – The methodology is designed as modular components
(Driver, Sequencer, Agents , env etc) to enable re-use at different levels of verification
and across projects
n Separating Tests from Testbenches – Tests in terms of stimulus/sequencers are kept
separate from the actual testbench hierarchy and hence there can be reuse of stimulus
across different units or across projects
n Simulator independent – The base class library and the methodology is supported by
all simulators and hence there is no dependence on any specific simulator
n Sequence based Stimulus generation: There are several ways in which sequences can
be developed which includes randomization, layered sequences, virtual sequences etc
which provides a good control and rich stimulus generation capability.
n Configuration mechanisms simplify configuration of objects with deep hierarchy.
The configuration mechanism (using UVM config data base) helps in easily configuring
different testbench components based upon verification environment using it, and
without worrying about how deep any component is in the testbench hierarchy
n Factory mechanisms simplifies modification of components easily. Creating each
components using factory enables them to be overridden in different tests or
environments without changing underlying code base.
42

43. Shivoo
+
Disadvantages of UVM
n Steep learning curve: For anyone new to the methodology,
the learning curve to understand all details and the library is
very steep.
n Still developing and not perfect/stable: The methodology
is still developing and has a lot of overhead that can
sometimes cause simulation to appear slow or probably can
have some bugs
43

44. Shivoo
+
THANKYOU
shivoo.koteshwar@gmail.com
www.facebook.com/shivoo.koteshwar
44