High Level Simulation based Verification

1 . Advanced Topics in High-Level Simulation-based Verification Kerstin Eder 1 University of Bristol November 3, 2003 

2 .Verification Maturity Scale I Measures verification capability based on current verification methodology. (Verisity) 1. Manual Verification (MV) Directed testing approach with manual checking. 2. Automated Verification (AV) Automatic stimulus generation and checking. 3. Coverage-Driven Verification (CDV) 4. Scalable Coverage-Driven Verification (SCDV) 5. Component-Based Reuse (CBR) Verisity has found that organizations operating at the AV level outperform teams working at the MV level, finding more bugs sooner, and so on up the scale. Kerstin Eder 2 University of Bristol November 3, 2003 

3 .Directed vs Coverage-Driven 100% Directed % Functional Coverage Approach Time CDV requires more commitment: Coverage metrics + (early) RTL availability Kerstin Eder 3– i University of Bristol November 3, 2003 

4 .Directed vs Coverage-Driven 100% Coverage-Driven Directed % Functional Coverage Approach Approach Time CDV requires more commitment: Coverage metrics + (early) RTL availability Kerstin Eder 3– ii University of Bristol November 3, 2003 

5 .Scoreboarding - Transaction-based Verification Put created transactions/packets on the scoreboard. Compare emerging transactions/packets. Collect output transactions/packets. Find corresponding item on scoreboard & delete. Use unique IDs. Account for transactions/packets that disappear. At the end, check that scoreboard is empty. Scoreboard Input Transaction Input Transaction add() Input Transaction match() Input Transaction DUV Output Transaction Kerstin Eder 4 University of Bristol November 3, 2003 

6 .Scoreboarding (simple) example in e - I Assume: DUV does not change order of packets. Hence, first packet on scoreboard has to match received packet. import packet_s; unit scoreboard { !expected_packets : list of packet_s; add_packet(p_in : packet_s) is { expected_packets.add(p_in); }; check_packet(p_out : packet_s) is { var diff : list of string; -- Compare physical fields of first packet on scb with p_out. -- Report up to 10 differences. diff = deep_compare_physical(expected_packets[0], p_out, 10); check that (diff.is_empty()) else dut_error(‘‘Packet not found on scoreboard’’, diff); -- If match was successful, continue. out(‘‘Found received packet on scoreboard.’’); expected_packets.delete(0); }; }; Kerstin Eder 5 University of Bristol November 3, 2003 

7 .Scoreboarding (simple) example in e - II Recording a packet on the scoreboard Extend driver such that: When packet is driven into DUV call add_packet method of scoreboard.  Current packet is copied to scoreboard. Useful to define an event that indicates when packet is being driven! Checking for a packet on the scoreboard: Extend receiver such that: When a packet was received from DUV call check_packet.  Try to find the matching packet on scoreboard. Useful to define an event that indicates when a packet is being received! Kerstin Eder 6 University of Bristol November 3, 2003 

8 .Verification Maturity Scale II 1. Manual Verification (MV) 2. Automated Verification (AV) 3. Coverage-Driven Verification (CDV) Tracking verification progress via code and functional coverage. – Verification Plan defines coverage requirements. – Iterative simulation produces coverage reports. – Cumulative coverage analysis. Ensures corner cases are hit. – Use constrained/biased pseudo-random generation. Set of advanced techniques. Accurate insight into verification progress. 4. Scalable Coverage-Driven Verification (SCDV) 5. Component-Based Reuse (CBR) Kerstin Eder 7 University of Bristol November 3, 2003 

9 .Advanced Constraints keep constraint-bool-expr; where constraint-bool-expr is a simple or compound Boolean expression. States restriction on the values generated for fields in the struct.  keep kind!=tx or len==16; Describes required relationships between field values and other struct items.  struct packet { kind : [tx, rx]; len : int; keep kind == tx => len==16; --when tx packet { keep len == 16; }; exactly same effect }; Hard constraints are applied when the enclosing struct is generated. If constraints can’t be met, generator issues constraint contradiction message. Kerstin Eder 8 University of Bristol November 3, 2003 

10 .Biased pseudo-random Generation Using keep soft and select:  struct transaction { address : uint; keep soft address == select { 10: [0..49]; 60: 50; 30: [51..99]; }; }; NOTE: Soft constraints can be overridden by hard constraints!  extend instruction { keep soft op_code == select { 40: [ADD, ADDI, SUB, SUBI]; 20: [XOR, XORI]; 10: [JMP, CALL, RET, NOP]; }; }; In practice, getting the weights/bias right (for coverage closure) requires significant engineering skill. Kerstin Eder 9 University of Bristol November 3, 2003 

11 .Generation with keep Generation order is important: It influences the distribution of values!  struct packet { kind : [tx, rx]; size : byte; keep size>15 => kind==rx; }; 1. If kind is generated first, kind is tx about half the time because there are only two legal values for kind. 2. If size is generated first, there is only a 1 in 16 chance that size is less than or equal to 15, so kind will be tx about 1/16 of the time!  Consider using: keep gen (kind) before (size); Kerstin Eder 10 University of Bristol November 3, 2003 

12 .Randomized Test Generation needs... ...repeatability: Same testbench version + same test + same random seed = same stimulus data. Is this all? The testbench evolves over time! and random stability: Changes to the testbench should not affect orthogonal aspects!  Packet data structure: struct packet { ... payload: list of byte; ...}; Kerstin Eder 11– i University of Bristol November 3, 2003 

13 .Randomized Test Generation needs... ...repeatability: Same testbench version + same test + same random seed = same stimulus data. Is this all? The testbench evolves over time! and random stability: Changes to the testbench should not affect orthogonal aspects!  Packet data structure with interrupted field: struct packet { ... payload: list of byte; interrupted: bool; ...}; With same seed should give the same payload data! Kerstin Eder 11– ii University of Bristol November 3, 2003 

14 .Advanced Techniques: SN temporal checking SN Temporal Language Capture behaviour over time for synchronization with DUV, functional coverage and protocol checking. Language consists of: – temporal expressions (TEs) – temporal operators – event struct members to define occurrences of events during sim run – expect struct members for checking temporal behaviour NEW: PSL/Sugar compatible expressions (more later). Kerstin Eder 12 University of Bristol November 3, 2003 

15 .Temporal Expressions in e Each TE is associated with a sampling event. Sampling event indicates when the TE should be evaluated by SN. Syntax examples:  true(boolean-exp)@sample-event  rise/fall/change(expression)@sample-event Kerstin Eder 13 University of Bristol November 3, 2003 

16 .Events in SN Events are used to synchronize with the DUV or to debug a test. Events are struct members.  Automatic emission of events: <’ extend driver_s { event clk is fall(’calc1_sn.c_clk’) @sim; event resp is change(’calc1_sn.out_resp1’)@clk; }; ’>  Explicit emission of event: <’ extend driver_s { collect_response(cmd : command_s) @clk is also { emit cmd.cmd_complete; }; }; ’> Kerstin Eder 14 University of Bristol November 3, 2003 

17 .Synch between SN and Simulator Issue test Callback triggers all SN command events attached to @sim event. SN executes SN gives SN evaluates TEs, emits presim control events and executes TCMs. phase to sim sys.tick_start sys.tick_end Specman callback Simulator t=0 ns t=N ns t=N ns time Sim runs until value of a signal Sim continues attached to an @sim event changes. until next Issues callback and sim suspends. callback Kerstin Eder 15 University of Bristol November 3, 2003 

18 .SN Predefined Event: @sim  event clk is rise (’˜/top/clk’) @sim; @sim is special sampling event occurring at any simulator callback. Expression must be an HDL signal path in the simulated model. Signal does not have to be a clock. No restriction for signal to be periodic or synchronous. Might slow down simulation! Clock signal can also be emitted from e code and driven into DUV. (But usually more efficient to generate clock in HDL.) When not running with a simulator attached to SN, use @sys.any. Kerstin Eder 16 University of Bristol November 3, 2003 

19 .Conforming to Stimulus Protocol Must be able to react to state of DUV during simulation!  clock, signal changes, sequences of events t0 t1 t2 e language provides wait (till next cycle) and sync actions which allow to pause procedural code top/clk until event occurs. top/enable print a; print a; print b; wait true(’˜top/enable’==1)@clk; print b; print a; sync true(’˜top/enable’==1)@clk; print a; print b; print b; Kerstin Eder 17 University of Bristol November 3, 2003 

20 .Methods with a Notion of Time TCMs - Time Consuming Methods Depend on sampling event. Can be executed over several simulation cycles.  collect_response(cmd : command_s) @clk is { wait @resp; -- wait for the response cmd.resp = ’calc1_sn.out_resp1’; cmd.dout = ’calc1_sn.out_data1’; }; // collect_response Implicit sync action at beginning of TCM. TCM must be called or started to execute.  run() is also { start drive(); // spawn }; // run Normal methods can’t call TCMs because they have no notion of time. TCMs can (only) be started from a normal method! Kerstin Eder 18 University of Bristol November 3, 2003 

21 .Temporal Checking of DUV Behaviour Checking time relationship between events and values of DUV signals. Define legal sequences of events for protocol checking. Example Temporal Operators:  not(temp exp)  temp exp1 and temp exp2 (or)  Sequences: f temp exp1; temp exp2g  Yield operator (next cycle): temp exp1 => temp exp2  true(temp exp) (change, fall, rise) Kerstin Eder 19 University of Bristol November 3, 2003 

22 .Temporal Checking Methodology 1. Capture important DUV temporal behaviour with events and TEs. 2. Use expect struct members to declare temporal checks. Syntax: expect TE else dut_error(string); Example temporal checks: expect @req => {[..4];@ack} @clk else dut_error("Acknowledge did not follow request within 1 to 5 clock cycles."); expect @buffer_full => eventually @int @clk else dut_error("Buffer full, but interrupt did not occur."); eventually Sometime before the end of simulation! Kerstin Eder 20 University of Bristol November 3, 2003 

23 .Verification Maturity Scale III 1. Manual Verification (MV) 2. Automated Verification (AV) 3. Coverage-Driven Verification (CDV) 4. Scalable Coverage-Driven Verification (SCDV) Multiple CDT environments from module to system level run in parallel. – Simulation farms (server farms) – Hardware emulation/acceleration Employ Coverage-Driven Verification (CDV) environments in 24-7 mode. With growing number of parallel environments comes increased control/data management problem! Significant SW support to manage control/data of CDT environments. Kerstin Eder 21 University of Bristol November 3, 2003 

24 .Verification Maturity Scale IV 1. Manual Verification (MV) 2. Automated Verification (AV) 3. Coverage-Driven Verification (CDV) 4. Scalable Coverage-Driven Verification (SCDV) 5. Component-Based Reuse (CBR) Leveraging SCDT across multiple projects. Component-level reuse.  eVCs Introduction of coding standards.  AIM: Maximise return on verification investment. Verification Component - Bus X Masters Monitor and Bus Logic Protocol Slaves Checker Kerstin Eder 22 University of Bristol November 3, 2003 

25 .Summary Verification Maturity Scale 1. Manual Verification (MV) 2. Automated Verification (AV) 3. Coverage-Driven Verification (CDV) 4. Scalable Coverage-Driven Verification (SCDV) 5. Component-Based Reuse (CBR) Objective measure of verification capability of a verification team. Clarifies where methodology can be improved. Kerstin Eder 23 University of Bristol November 3, 2003