1. May 1, 2013
Metric Driven Formal Verification
Raik Brinkmann
OneSpin Solutions

2. May 1, 2013
OneSpin Solutions - Who we are…
OneSpin provides
enduring solutions that
enable the most
thorough & easiest to use
logic verification available
OneSpin Solutions was spun
out of Infineon in 2005
Technologies include
Assertion-based
verification (ABV) and
equivalence checking (EC)
Production-proven
tools used in
thousands of designs
The company is a completely
independent entity owned by
Azini Capital and OneSpin’s
employees
The original technologies have
been augmented & enhanced
to provide the most thorough
formal capabilities available
Engineering HQ based
in Munich, Germany
Sales offices in US,
France, Germany,
Israel, Japan, and
Turkey

3. May 1, 2013
Formal Verification with OneSpin 360
• Unique push button observation coverage analysis for verification progress
• Faster assertion development in timing diagram style using transaction level assertions
• 100% functional coverage using unique gap free verification
• Unique structural assertion debugger and active value/driver tracing through RTL
• Incremental compilation of assertions for quick turn around
• Automatic clock and reset detection for easy design setup
Unique productivity features
• HDL-linting
• Structural assertion synthesis
• Coverage closure for simulation
• SoC connectivity verification
• Register verification
• Formal score boarding
• Protocol verification IP
• X-propagation verification
Large selection of push-button design verification solutions, included
• Verification of logic, sequential and power optimizations
• Supporting RTL and gate level for FPGA and ASIC
• Integrated with major vendors and many synthesis flows
Push-button equivalence checking

4. May 1, 2013
Outline
Importance of Progress Metrics
Coverage Metrics
Control vs. Observation Coverage
Practical Observation Coverage
Combining Control and Observation Coverage
Examples
Summary

5. May 1, 2013
Why do we need a metric?
DUV
Tests / Scenarios Checkers
Bus Functional
Model (BFM)
Test #1
Test #2
Stimulus / Scenario Generation
Constraints
Formal
Engine
Stimulus / Scenario Exclusion
Check #1
Check #2
Check #3
Check #4
Simulation
Formal
Verification Environment (simplified):
Verification Process:
• Plan: Verification planning
• Do: Write tests / properties and verify and fix DUV, tests, properties
• Check: Are we done, (still) making progress?
• Act: Adapt planning or get ready for tape-out
Progress Metric : Key Component of Verification Flow
Plan Do
CheckAct

6. May 1, 2013
Standard Formal ABV
Progress of Formal Verification?
• Unclear how many assertions to write and where to put them
• Unclear verification progress
• Unclear how formal verification affects overall verification quality
Integration with Simulation?
• Weak integration into verification flow and sign-off
• Additional effort to existing simulation effort
• Formal used as “point tool” for verifying specific aspects (e.g. hard to test
corner cases)
 Low acceptance of formal ABV
We need a practical progress metric!

7. May 1, 2013
Progress Metric & Coverage Taxonomy
Coverage
Structural Functional
Progress Metric
Bug Rate
Code Circuit
• Line/Statement/Block
• Expression/Branch/Condition
• FSM
• Toggle
• Latch
Scenarios
Implementation
artefacts
Analytical
Bugs
Fixed
100%
time
ok#
AssertionComplete
• Gap Free Analysis • Verification Plan
Anecdotal
Have I written enough tests / properties?
How much has been verified?
Where are the gaps in my verification?
Are we done and ready for tape-out?
if (…)
…
if (…)
else …

8. May 1, 2013
Control / Simulation Coverage:
• Focused on quality of stimuli
• How to judge quality of checkers?
Observation & Control Coverage
coverage metrics
stimulus generation checkers/assertions
plan
DUV
report
How good are my
test vectors &
constraints?
How good are my
checkers and
assertions?
How much of my DUV is verified?
When is my verification finished?
Observation Coverage:
• Focused on quality of checkers
• Exposes unverified DUV parts

9. May 1, 2013
Been there? Done what?
• Has the statement been reached?
• Idea:
– If a statement has not been reached
during verification, it can’t break a
check.
– If a statement has been reached, would
some check fail?
• Can measure quality of stimuli.
case (state)
…
burst:
if (cancel_i)
done_o <= 1
…
active
case (state)
…
burst:
if (cancel_i)
done_o <= X
…
mutate
• Has the statement been verified?
• Idea:
If a statement is modified, some check
should fail.
But would some check fail, if the
statement cannot be reached?
• Can measure quality of checkers.
Been there! Done that!
Example: Statement Coverage
Coverage
Control Observation
Been there! Done that!

10. May 1, 2013
Control Coverage for Formal
No explicit stimulus generation for formal
Formal considers all possible input stimuli by default (exhaustive verification)
However: Control coverage still an issue because
• Formal requires constraints excluding illegal environment behavior
• Constraints may accidentally be too strong and exclude vital functionality
• Constrained functionality can neither be controlled nor observed
(Structural) control coverage
quantifies degree to which
locations of a design have been
activated during verification.
control
controllable
reached
unreached
unknown
uncontrollable
dead
constrained

11. May 1, 2013
(Formal) Observation Coverage
Verification tries to answer the question:
• “Does a design satisfy a specification?”
(Observation) coverage tries to answer the question:
• “What causes a design to satisfy a specification?”
Causality:
• “When do we say A is a cause of B?”
Common approach: Counterfactual causality:
• “A is a cause of B if, had A not happened, then B would not have happened.”
• A code line is a cause of a design satisfying an assertion. Source: H. Chockler, IBM
(Structural) observation
coverage quantifies degree to
which locations of a design
have been responsible to make
checks pass.
observation
unknown
observable
observed
unobserved
unobservable

12. May 1, 2013
Practical Observation Coverage
Mutation coverage
• Particular changes depending on error model, usually static
• Replacing for example “a <= b;” with “a <= 1´b1;”
• Problems:
• Can lead to vacuously holding assertions
• Requires several modifications at each location, high run time
“Quantify MDV” metric
• Use abstraction by free variables, allow dynamic behavioral change
• Replacing for example “a <= b;” with “a <= free_b;”
• Advantages:
• No unintended vacuity
• One modification for each location leads to faster results
• Multiple locations can be checked at the same time to prove “unobserved”

13. May 1, 2013
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= c;
default: z <= <input>;
endcase
end
M5
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= <input>;
default: z <= 1'b1;
endcase
end
M4
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= <input>;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M3
Formal Observation Coverage Examplemodule select1(onehot, a, b, c, z, clk, reset);
input clk;
input reset;
input [2:0] i;
input a;
input b;
input c;
output reg z;
always @(posedge clk or posedge reset)
if (reset)
z <= 1'b0; // L1: not covered (reset case)
else
begin
case (i)
3'b001: z <= a; // L2: covered by assertion
3'b010: z <= b; // L3: not covered
3'b100: z <= c; // L4: not covered
default: z <= 1'b1; // L5: not covered
endcase
end
// if there is no reset, then 'a' is stored in 'z' if ‘i' is 3'b001
A: assert property
( @(posedge clk)
disable iff (reset)
i == 3'b001 |=> z == $past(a)
);
endmodule
Q: Which assignment locations Lx in design
M are observed by proven assertion A?
2. Re-Check property A for each M1..M5
always @(posedge clk or posedge reset)
if (reset)
z <= <input>;
else
begin
case (i)
3'b001: z <= a;
3'b010: z <= b;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M1
always @(posedge clk or posedge reset)
if (reset)
z <= 1’b0;
else
begin
case (i)
3'b001: z <= <input>;
3'b010: z <= b;
3'b100: z <= c;
default: z <= 1'b1;
endcase
end
M2
Assertion A holds on M1:
L1 not observed
Assertion A fails on M2: L2
is observed
M
A
L3 not observed
L4 not observed
L5 not observed
1. Modify each location L1..L5
of M: Producing M1..M5
A: The locations Lx for which A fails after
replacing the assignment with a free input.

14. May 1, 2013
Merging Observation and Control
Observation Coverage Control Coverage Merged Result
observed reached1) covered
unknown reached reached
unobserved reached unobserved
unknown unreached unreached
unobserved unreached uncovered
unobservable2) constrained constrained
unobservable2) dead dead
1. If a location is uncontrollable, it is also unobservable.
2. If a location is observed, it is also reached and thus controllable.
observation
unknown
observable
observed
unobserved
unobservable
control
controllable
reached
unreached
unknown
uncontrollable
dead
constrained
2.
1.
Been there! Done that!

15. May 1, 2013
Interfacing with UCIS
UCIS
• Coverage standard evolved from Mentor Graphics‘ UCDB
• Only supports control coverage, but not observation coverage
How to interface?
• Use user defined structures to store Quantify MDV results
• Use result merging to enhance standard UCIS coverage
Result Merging from UCIS to Quantify MDV:
• Covered bins -> Reached locations
Merging Quantify MDV to UCIS:
• Dead/Constrained -> Coverage Excludes
• Reached locations -> Covered bins
• Covered locations -> Covered bins
Conclusions on UCIS interfacing
• Merging results with UCIS is possible and useful now
• Direct support for observation coverage by UCIS is desirable

16. May 1, 2013
Simple Quantify MDV Example
• Mixed control and observation
coverage results indicate:
– holes indicating missing
assertions
– constrained code indicating
over-constrainingverification
hole
verified
code
constrained
code
dead
code
Management
overview

17. May 1, 2013
Industry Examples
Design LOCs #Assertions #Locations Runtime
FIFO 321 30 21 100s
FSM-DDR2-Read 839 6 93 106s
vCore-Processor 295 8 86 204s
SQRT 383 2 35 257s
IFX-Aurix-1*) 25563 85 2316 4d
IFX-Aurix-2 27374 157 1993 5d
IFX-Aurix-3 57253 253 5309 7d**)
*) “Quantification of Formal Properties for Productive Automotive Microcontroller Verification”, Holger
Busch, DVCon, San Jose - February 26, 2013
**) interrupted at 80% completion

18. May 1, 2013
Summary
Coverage is an important aspect of verification
New metric introduced: “Quantify MDV”
• Combines observation and control coverage
• Agnostic to assertion style, assertion language, design style and design
language
• Scales to relevant big industry designs
• Useful for
• Quick feedback on formal verification quality, as well as
• Criterion for (formal) verification closure
• Can be combined with simulation based verification and metrics

19. May 1, 2013
Thank You!

20. May 1, 2013
What about redundancy?
Redundant code cannot be observed!
• Redundant code will be marked uncovered if not treated specially.
What are sources of redundancy?
• Redundant logic not driving any outputs
• Safety logic meant to implement fail safe devices
How to mitigate?
• Identify redundant code and mark it
• Remove redundant code before running coverage
• Selectively activate/de-activate redundant parts

21. May 1, 2013
Functional vs. Structural
Coverage Type Functional Structural
Control How much of the specified behavior
has been activated?
How much of the DUV implementation has
been activated?
Observation How much of the specified behavior
has been verified?
How much of the DUV implementation has
been verified?
Functional vs. Structural
• Functional - relates to specification
– Measures completeness of requirements verification
– Can identify gaps in verification plan
• Structural - relates to DUV
– Measures completeness of design intent verification
– Can identify unverified parts of DUV
Control vs. Observation
• Control - relates to activation
– Measures quality of stimuli
– Can identify unreachable code / over-
constraining
• Observation - relates to causality
– Measures quality of checkers
– Can identify verification gaps
• Control coverage doesn’t imply any observation coverage.
• Observation coverage doesn’t imply any functional coverage.
• 100% observation coverage implies 100% control coverage.
• 100% Functional coverage + all assertions proven implies 100% observation coverage (if DUV contains no dead
code, no constrained code, no redundant code)

22. May 1, 2013
Verification Environment with Exhaustive Coverage Analysis
DUV
Tests / Scenarios Checkers
Spec
Verification Environment
Verification
Plan
Coverage
Analysis
Control
Coverage
Observation
Coverage
Functional
Coverage
Have I written
enough stimuli?
Which parts of
my DUV have
been exercised?
Which parts of my
DUV have been
checked?
Did I write
enough checks?
Are all specified
functions
implemented?
Are all specified
functions verified?
Functional
Structural

23. May 1, 2013
What does block level coverage mean
in the larger context?
System-level
design
Block-level
design (RTL)
Architecture-level
design
Design start
Verification closure
System-level
simulation
Block-level
simulation
Sub-system-level
simulation
Formal ABV with
Covage Analysis
UCDB
Aggregation of module / block
level coverage results with higher
level context
High-Level/RTL Design & Verification Flow