1. Heuristic Stimuli GenerationHeuristic Stimuli Generation
For Coverage ClosureFor Coverage Closure
Exploiting Simulation FeedbackExploiting Simulation Feedback
Giovanni Squillero
Politecnico di Torino - Italy
CAD Group ( )
giovanni.squillero@polito.it

2. GOALGOAL
• To propose a methodology for coverage-
directed stimuli generation based on
simulation feedback
• Such stimuli could be added as new content to
improve existing validation suites
DVClub 2010-01-18 giovanni.squillero@polito.it 2

3. AcknowledgementsAcknowledgements
• Danilo Ravotto
• Ernesto Sanchez
• Matteo Sonza Reorda
• Alberto Tonda
• + many others
DVClub 2010-01-18 giovanni.squillero@polito.it 3

4. OutlineOutline
• Proposed methodology
• Case studies
• Conclusions
DVClub 2010-01-18 giovanni.squillero@polito.it 4

5. Design ChoicesDesign Choices
• Being able to tackle real problems
• Develop a versatile and broadly applicable
methodology
– Compatible with different environment
– Compatible with any coverage metric
• Minimize effort to change goal/target
– Exploit common aspects
DVClub 2010-01-18 giovanni.squillero@polito.it 5

6. Feedback-Based ApproachFeedback-Based Approach
• Simulation-based approach
• Exploits feedback from simulation
• Incremental improvement/refinement of the
solution (trial-and-error)
• Trade-off between computational resources
and confidence
• May exploit heuristics or problem-specific
knowledge
DVClub 2010-01-18 giovanni.squillero@polito.it 6

7. Proposed MethodologyProposed Methodology
DVClub 2010-01-18 giovanni.squillero@polito.it 7
Stimuli
Generator
System
Stimuli
Feedback

8. StimuliStimuli
DVClub 2010-01-18 giovanni.squillero@polito.it 8
Stimuli
Generator
System
Stimuli
Feedback

9. StimuliStimuli
• Sequences of bits
• Sequences of keys
• Full fledged assembly language programs
• VHDL test case
• External world
• …
DVClub 2010-01-18 giovanni.squillero@polito.it 9

10. Stimuli GeneratorStimuli Generator
DVClub 2010-01-18 giovanni.squillero@polito.it 10
Stimuli
Generator
System
Stimuli
Feedback

11. Stimuli GeneratorStimuli Generator
• Exploit an Evolutionary Algorithm to generate
stimuli to maximize a given function
DVClub 2010-01-18 giovanni.squillero@polito.it 11

12. Evolutionary AlgorithmsEvolutionary Algorithms
• Meta-heuristic optimization algorithm based
on the concept of population and exploiting
some principles of natural evolution
DVClub 2010-01-18 giovanni.squillero@polito.it 12

13. Evolutionary AlgorithmsEvolutionary Algorithms
• Succession of random and deterministic steps
– A systematic way of throwing dices
– Better than pure random
“The great effect produced by the
accumulation in one direction,
during successive generations,
of differences absolutely inappreciable
by an uneducated eye”
DVClub 2010-01-18 giovanni.squillero@polito.it 13

14. Evolutionary AlgorithmsEvolutionary Algorithms
• Population
– Multiple solutions considered in each step
– More resistant than pure hill-climbing
– Different solutions may interbreed
DVClub 2010-01-18 giovanni.squillero@polito.it 14

15. Evolutionary AlgorithmsEvolutionary Algorithms
• Why using an evolutionary algorithm?
– Adaptative
– Able to find unexpected solutions
– Better than randomBetter than random
DVClub 2010-01-18 giovanni.squillero@polito.it 15

16. Evolutionary AlgorithmsEvolutionary Algorithms
• Problem: Fitness function
– GOAL: Optimize the wheel
– FITNESS: Minimize the number of “bumps”
DVClub 2010-01-18 giovanni.squillero@polito.it 16

17. Evolutionary AlgorithmsEvolutionary Algorithms
• Problem: Black magic
DVClub 2010-01-18 giovanni.squillero@polito.it 17

18. µGP (MicroGP)µGP (MicroGP)
• CAD Group general-purpose evolver
– 3 versions (only 2 released under GPL)
– Project started in 2002
– 11 developers + contractors, students, …
• Current version
– ≈ 300 file, > 40,000 lines in C++
DVClub 2010-01-18 giovanni.squillero@polito.it 18

19. µGP (MicroGP)µGP (MicroGP)
Evolutionary Optimization: the µGP
toolkit
E. Sanchez, M. Schillaci, G. Squillero
Springer, 2010
ISBN: 978-0-387-09425-0
DVClub 2010-01-18 giovanni.squillero@polito.it 19

20. µGP (MicroGP)µGP (MicroGP)
DVClub 2010-01-18 giovanni.squillero@polito.it 20
http://ugp3.sourceforge.net/
MicroGP++ (aka. ugp3, µGP3
)
• Information
• Download
• Credits

21. System & FeedbackSystem & Feedback
DVClub 2010-01-18 giovanni.squillero@polito.it 21
Stimuli
Generator
System
Stimuli
Feedback

22. SystemSystem
• Strongly problem dependant
• Model via simulation/emulation
– HDL (netlist to high-level)
– HW accelerated (e.g., exploiting FPGA)
– Architectural simulator
– ISA simulator
• Real device
DVClub 2010-01-18 giovanni.squillero@polito.it 22

23. Feedback (examples)Feedback (examples)
• From simulation
– Code coverage metrics (e.g., instruction coverage)
– HW specific metrics (e.g., toggle coverage)
– High-level information (e.g., FSM coverage)
• From the real system
– Performance counters
– Physical measures (e.g., temperature, time, power
consumption)
DVClub 2010-01-18 giovanni.squillero@polito.it 23

24. OutlineOutline
• Proposed methodology
• Case studies
• Conclusions
DVClub 2010-01-18 giovanni.squillero@polito.it 24

25. Design VerificationDesign Verification
• Devise a set of programs maximizing different
coverage metrics
DVClub 2010-01-18 giovanni.squillero@polito.it 25

26. FeedbackFeedback
• Code coverage metrics
– Statement coverage (SC)
– Branch coverage (BC)
– Condition coverage CC)
– Expression coverage (EC)
• HW specific metric
– Toggle coverage (TC)
DVClub 2010-01-18 giovanni.squillero@polito.it 26

27. SystemSystem
• DLX/pII
– Cleaned and simplified MIPS intended primarily
for teaching purposes
– 32-bit load/store architecture, 5-stage pipeline
– VHDL RTL description
• 4,558 statements
• 3,695 branches
• 193 conditional statements (1,764 expressions)
• 8,283 logic bits
DVClub 2010-01-18 giovanni.squillero@polito.it 27

28. Experimental ResultsExperimental Results
DVClub 2010-01-18 giovanni.squillero@polito.it 28

29. Experimental ResultsExperimental Results
DVClub 2010-01-18 giovanni.squillero@polito.it 29
230K
instructions
2,978
instructions

30. Experimental ResultsExperimental Results
DVClub 2010-01-18 giovanni.squillero@polito.it 30
20h 33h
27h
37h
95h
≈1 week

31. Post-silicon VerificationPost-silicon Verification
• Generate functional test programs for post-
silicon verification
• Activity performed in collaboration
with the ETM Group, Intel (Phoenix)
• Intel Pentium 4
– 42-55 millions of transistors
– 2GHz clock
– NetBurst architecture
DVClub 2010-01-18 giovanni.squillero@polito.it 31

32. System & FeedbackSystem & Feedback
• Real Pentium 4 (no simulation)
• Performance counters as metric
– Introduced in 1993 in IA32 architecture
– 48 event detectors + 18 programmable counters
– Instruction-tagging (for discriminating non-
speculative performance events)
DVClub 2010-01-18 giovanni.squillero@polito.it 32

33. Experimental ResultsExperimental Results
• Two sets of experiments:
– Mispredicted branches over the total branches
– Clock cycles in which the trace cache is delivering
µops to the execution unit instead of decoding or
building traces (hard metric!)
• Intrinsic features of the µarchitectural design
• Time (both): 12h/program
Haifa 2008 G. Squillero 33

34. Mobile PhoneMobile Phone
• 12m-activity performed in collaboration with
Motorola Research Center in Turin
(“Automation of Current Drain Measures”)
DVClub 2010-01-18 giovanni.squillero@polito.it 34

35. SystemSystem
• Real prototypical mobile phone
– P2K platform
– GSM/3G networks
– Complex applications
• Radio Communication Analyzer, anechoic
chamber
• Controllable power supply
• Custom hardware
DVClub 2010-01-18 giovanni.squillero@polito.it 35

36. StimuliStimuli
• Sequence of keys
• Power supply
• Network status
DVClub 2010-01-18 giovanni.squillero@polito.it 36

37. FeedbackFeedback
• Physical measures
– Power consumption (+ autocorrelation)
– Network status
• FSM transition coverage
DVClub 2010-01-18 giovanni.squillero@polito.it 37

38. Experimental ResultsExperimental Results
• Major bugs discovered in the prototype
– Test infrastructure
– LCD display
– Fake deep-sleep state
DVClub 2010-01-18 giovanni.squillero@polito.it 38

39. OutlineOutline
• Proposed methodology
• Case studies
• Conclusions
DVClub 2010-01-18 giovanni.squillero@polito.it 39

40. ConclusionsConclusions
• Simulation-based & feedback-based
• Evolutionary algorithm
• Broadly applicable
• Human resources
– Limited (set-up )
• Computational resources
– Easily parallelizable (generation level)
– Trade-off quality vs. effort
DVClub 2010-01-18 giovanni.squillero@polito.it 40

41. ConclusionsConclusions
• May exploit other methodologies
– Useful starting point
– Effective completion
• May be coupled with other methodologies
– Rule-based instruction randomizers
– Simulation-based approaches
DVClub 2010-01-18 giovanni.squillero@polito.it 41