This presentation points out gaps in what we teach in systems engineering using a simulation of the ARRL Sweepstakes contest as an example or case study in developing part of a simulation.
1. Systems engineering: It’s an
enabler
Presentation at Orbotech 7 March 2011
Dr Joseph Kasser, CEng, FIET, CM, CMALT
1
2. Assumptions
You know something about systems
engineering
Talking about philosophy to make you think about
the way you apply it
You do not know about amateur radio
The application domain in the case study
You understand enough English to
understand this talk
Example of importance of stating assumptions
Facilitates communications
2
3. Topics
Systems engineering camps
Gaps in what we teach
Case study extract
Pointing out some of the gaps
Lessons learnt from the case study
Systems engineering: an enabler
3
5. The systems engineering camps
The process camp
Functional perspective
The discipline camp
Structural perspective
The problem camp
Operational perspective
The activity camp
Functional/operational perspective with an
objective definition 5
6. Parable of blind men and elephant*
“… And so these men of Indostan
Disputed loud and long,
Each in his own opinion
Exceeding stiff and strong,
Though each was partly in the right,
And all were in the wrong!
MORAL.
So oft in theologic wars,
The disputants, I ween,
Rail on in utter ignorance
Of what each other mean,
And prate about an Elephant
Not one of them has seen!”
* Yen, D. H., The Blind Men and the Elephant, 2008,
http://www.noogenesis.com/pineapple/blind_men_elephant.html, last accessed 26 October 2010 6
7. A matter of perspective.
Functional
Functional and
operational
Process
Take
over the
world
Operational
Perspectives are
Problem
incomplete and
there are gaps
7
8. Topics
Systems engineering camps
Gaps in what we teach
Case study extract
Pointing out some of the gaps
Lessons learnt from the case study
Systems engineering: an enabler
8
9. Amateur radio - context
World-wide hobby
Licensed by governments
National societies
American Radio Relay League (ARRL)
Radio Society of Great Britain (RSGB)
Singapore Amateur Radio Transmitting Society (SARTS)
Experimenting and applying
Emergency communications, terrestrial, satellite, hardware,
software, business, etc.
Platform for developing systems engineers*
* Kasser, J. E., "How synergy between amateur radio, systems and other engineering
can raise the technical quotient of a nation", proceedings of the 4th Asia-Pacific
Conference on Systems Engineering (APCOSE 2010), Keelung, Taiwan, 2010.
9
http://www.youtube.com/watch?v=nd0tTYgJWEo
10. Big picture - context
Simulated emergency message traffic
handling
Contest format
Exchange messages simulating traffic into
and out of simulated disaster zone
World-wide contests
Everybody contacts everybody
Local and regional contests
US Sweepstakes, Australia, BERU, etc.
Target area contests
Everybody tries to contact stations in a given area
US State QSO parties, ARRL DX, SAS, WAE, etc.
Go back in time to 1977/8 10
11. ARRL Sweepstakes contest
[1977]
Contact (work) as many other stations as
possible within 48 hour period
Weekends in November
Exchange simulated emergency message
Use different frequency bands with different
propagation characteristics
Score = number of contacts * multiplier
Multiplier is number of ARRL Sections contacted
Section only counts once irrespective of frequency band
11
13. Column A: a systems engineering
approach to problem solving*
3
1. Plan the work (Tasks 2-8)
2. Define the problem 1 2 5 6 7 8
3. Conceive solution options 4
4. Identify ideal solution evaluation
criteria Recursive or self similar
5. Perform trade off to find the
optimum solution Applies to each box
6. Select the preferred option Fits in one column of the
7. Formulate strategies and plans to HKMF
implement the preferred option
8. Milestone review to obtain Fits across several columns
authorisation to proceed to of the HKMF
implementation phase
* Tasks 2-7 from Hitchins, D. K., Systems Engineering. A 21st Century Systems Methodology, John Wiley & Sons 13
Ltd., Chichester, England, 2007., Figure 6.2
14. Focusing in on the problem on
Reliance
technology
Problem had been recognized
Technology
at least 12 years earlier
The time wasenable
may 1978
Understand the factors
orinprohibit
involved the ARRL
solution
sweepstakes contest well
enough to enable an operator
in Silver Spring, MD to contact
Problem all the Sections (multipliers)
statement is given the constraints of low
radiated power
different
14
Key words in red
15. Exploring solutions to determine
real need
1. Read about factors 2. Create contest simulation
Identify the factors Identify the factors
Identify relationships Identify relationships
between factors between factors
Develop understanding Develop understanding
Create simulation
Run simulation
Try approaches and
see what happens
Develop better
understanding 15
16. Radio Propagation
Sky wave
(refraction altitude dependent on frequency and time (solar effect)
Ground wave Interference
My QTH
Distant Section
16
17. ARRL Sections in 1977
Note: Numbers are assigned for this project not by the ARRL 17
18. The pertinent factors
The Sections
75, spread out across CONUS, and non-CONUS
Probability of propagation to Section
Radio frequency band
Time of day
Probability of someone in Section when wanted
Population distribution
Probability of interference from other stations
Transmitted power
Receiver front end characteristics
18
19. Statement of the problem
Develop a simulation that is:
1. Realistic enough to enable an operator in
Silver Spring, MD to contact all the
Sections given the constraints of low
radiated power if the operator has
developed an understanding of the
pertinent factors involved in contacting all
the Sections. Realistic enough to enable
successful completion of
2. Fun to play. mission at the appropriate
time 19
20. Feasibility study: Constraints
(implementation domain knowledge)
INTEL Hardware platform
Given constraint
8080 microprocessor (8 bits)
Assume software written in BASIC
Memory
32K RAM
Interpreter occupies 16K Bytes
Need some space for Stack and run time
variables
Leaves about 14K Bytes for program 20
21. Feasibility study: Risk management
(implementation domain knowledge)
Do some software code sizeWe often decouple
estimation
Table of Stations callsign array is hardware and
largest data element
(need to know there are 2707 maximum from published scores)
software development
Conclusion – feasible but with implementation risk
(Except in embedded
RAM Memory may be insufficient
Risk mitigation possibilities systems)
Use spaghetti code to cut down on memory use
Document source code accordingly, REM statements are discarded by
interpreter
Use IBM 360 if code won’t fit in Intelec MD8/80
TPM: RAM usage – track during software development
21
Domain knowledge is software programming and PC platform
23. What the operator does
Contact
Start Exchange data Store data
someone
Time out
(0-n minutes)
No Yes
Contest
over ?*
End
* Contest over := (24 hrs of operation [elapsed time]) or (end time GMT) 23
24. Functional flow diagrams
Start documenting and expanding ideas
Intuitive
Identify sequences
Functions can be simple or complex
Identify dependencies between functions
May not provide best grouping
Are a tool to provide a view
May not be best tool to create relationships
between functions
24
25. Alternate Function flow chart –
Call CQ (OS1.1) functional
perspective
Send Worked B4
Yes
Call CQ Reply? Yes
Duplicate?
Yes
Enough?
Send Message
Yes
Receive message Complete? Store data Say ‘Bye’
Request repeat
25
26. Function flow chart – Call CQ
(OS1.1) functional perspective
Send Worked B4
Yes
Call CQ Reply? Duplicate?
Yes
No No
Had
Enough Exchange data (QSO)
No ?
Yes
Store data Say ‘Bye’
26
27. Exchange data (QSO) (1)
functional perspective
Send Message1 Receive message2
Yes
Complete?
Request repeat
Note 1, sending station may also request you to resend message
Note 2, message may be incomplete due to interference
27
28. Exchange data (QSO) (2)
functional perspective
Receive message2
Send Message1
Complete?
Yes
No
Request repeat
Note 1, sending station may also request you to resend message
Note 2, message may be incomplete due to interference
28
29. Partial functional N2 chart
F_CQ o o o3
o F_RX o o1 o2 o o4
o F_CK o o o
o F_B4 o
o F_TXM o o o
o o F_RXM o o
o F_LOG o
o F_QSY o
o o o F_QRV
Outputs – horizontal squares, Inputs – vertical squares 29
30. F_QSO
Partial functional N2 chart
Single input, candidate
for aggregation
F_CQ o o o3
o F_RX o o1 o2 o
o F_CK o o o May need
a new
o F_B4 o interface
function
o F_TXM o o o here
o o F_RXM o o
o F_LOG o
o F_QSY o
Candidate for
o o aggregation with F_QSO o F_QRV
outputs – horizontal squares, inputs – vertical squares
30
31. How do you determine system
functions?
Top down?
Bottom up?
Middle out?
All of the above
Tendency to flowchart functions
N2 chart is better way
Both approaches is best way
Checks and balances 31
32. Do we need requirements?
Size of project
Very small
Concept of operations
Well understood
Likelihood of changes during SDLC
Very low
Number of people/organizations
involved
1
32
33. Requirements analysis -1:
Requirement for number of stations
taking part
Published scores showed maximum number of
contacts was 2707 for top scorer
Requirement
600. The system shall contain at least 2707 stations.
Can set number at 3000
(TPM: watch RAM usage, and drop to 2710 if necessary; use
parameter for value)
Critical:
Feasibility analysis (maximum) customer
3,000/24=125 contacts/hour = 2/minute involvement
Reasonable (application domain knowledge)
33
34. Requirements analysis -2:
Requirement for number of stations in
each Section
Published scores in QST list participation by Section
Two separate parts (weekends) to contest
Phone (SSB) and Morse code (CW)
Set requirements for stations in each Section based on
1. SSB participation
2. CW participation
3. Combination participation in the two parts
Assumption
Ratio of number of stations not submitting entries is about the
same as for those submitting entries
3
1 2 5 6 7 8
4
34
35. Requirements analysis -2:
Number of stations in each
Section
3
Selection criterion
1 2 5 6 7 8
4
At least one station in each Section?
If none of the options contains at least one station, then
a station may need to be inserted – see next slide for
options.
Criteria CW SSB Combined
At least one in each No No Yes
Section
Missing Sections WY,NWT CZ None
35
36. Dealing with WY, NWT and CZ
(1 participant)
3
Options 1 2
4
5 6 7 8
1. Set value at 1 (published results)
For all iterations of simulation
2. Set value at either 0, 1 or 2
At start of each iteration of simulation
Selection criteria
It’s a simulation, numbers are small, should
have minimum effect
Makes game more interesting (and real?)
Uncertain if a ‘clean sweep’ can be achieved
36
37. 630. Requirements for number of
stations in Canada1
631 The system shall contain 7 stations in MAR2.
632 The system shall contain 8 stations in QB.
633 The system shall contain 16 stations in ONT.
634 The system shall contain 9 stations in MAN.
635 The system shall contain 3 stations in SK.
636 The system shall contain 7 stations in AB.
637 The system shall contain 8 stations in BC.
638 The system shall contain 0, 1 or 2 stations in NWT.
Notes 1Numbers are determined by distribution expect for NWT
37
2 See Section TBD for abbreviations
38. Looking back at the problem
statement
Number of stations in each Section that is needed for the
understanding?
Exact or relative?
Let the number of stations in each Section vary by a
small percentage each time the simulation is run
Make simulation game more interesting
Makes for a different situation
Valid
distribution was assumed based on 1 set of entries
there was an assumption on the distribution in slide 64
Put ± tolerances on the stations in each Section
38
39. 630. Requirements for number of
stations in Canada1
631 The system shall contain between 6 and 8 stations in MAR.
632 The system shall contain between 7 and 9 stations in QB.
633 The system shall contain between 14 and 18 stations in ONT.
634 The system shall contain between 8 and 10 stations in MAN.
635 The system shall contain between 1 and 4 stations in SK.
636 The system shall contain between 6 and 8 stations in AB.
637 The system shall contain between 7 and 9 stations in BC.
638 The system shall contain 0, 1 or 2 stations in NWT.
1. Tolerances from application domain knowledge
39
40. Alternate approach to setting
requirements for number of stations
in Canada
Use probabilistic approach and calculate for each iteration
of simulation
Section calculation approach (F_Section)
Test feasibility of requirements
Calculate percentage of entries in each Section in contest from
published results
Whole contest or by call area (easier to manage)
Write software module to set up distribution of Sections based on
calculated percentages (± tolerance, allowing for a 0 value)
Exercise module several times and compare results of model to
published data from results to validate module
Write requirements to allow for both design approaches
40
41. Partial published results
CW SSB TOTAL % CANADA % CONTEST
MAR 4 3 7 11.86 0.26
QB 6 2 8 13.56 0.30
ONT 10 6 16 27.12 0.59
MAN 6 3 9 15.25 0.33
SK 2 1 3 5.08 0.11
AB 2 5 7 11.86 0.26
BC 4 4 8 13.56 0.30
NWT 0 1 1 1.69 0.04
SUM 34 25 59 100.00 2.19
41
42. 630. Requirements for number of
stations in Canada1
631 0.26±p% of the stations in the contest shall be in MAR.
632 0.30±p% of the stations in the contest shall be in QB.
633 0.59±p% of the stations in the contest shall be in ONT.
634 0.33±p%of the stations in the contest shall be in MAN.
635 0.11±p% of the stations in the contest shall be in SK.
636 0.26±p% of the stations in the contest shall be in AB.
637 0.30±p% of the stations in the contest shall be in BC.
638 0.04±p% of the stations in the contest shall be in NWT.
p% TBD
42
43. Functions and requirements
Function
Contact stations in Canada
Requirements
Specify the number of stations in each Section in
Canada
Requirements
are one way to quantify functions
And ..
43
44. Requirements analysis 3:
Platform dependency
101.1 The simulation shall be written in Microsoft
BASIC Version 2.0.
101.2 When executing, the simulation shall be
contained within 12K Bytes of RAM.
OR
101. The simulation shall execute on an INTEL Intelec
8/80 Microprocessor Development System equipped
with 32 K Bytes RAM.
[That is the maximum RAM for that architecture] 44
45. Tools and techniques used in
requirements analysis (summary)
Looked up data in application and execution domains
Used domain data to compute values for requirements
Developed models of probabilistic functions
Design and realized partial elements of solution system
Exercised models
Tested validity of models as part of breadboarding process
Used problem solving process a number of times
How the wording of requirements affects the design
45
47. Issues addressed in this phase
Writing the code (construction)
Propagation model
Section distribution model
47
48. Propagation model
Time Frequency Band (M) For W1/VE1 call area*
(Hrs) 10 15 20 40 80 Group of sections
0000 0 0 0 100 100 Limited to 10-80 M
0400 0 10 50 100 100 Time is EDT or Local
0800 100 0 100 100 0 Four hour time of day
1200
blocks
100 0 100 100 0
Number indicates
1600 50 10 50 100 50
probability of
2000 0 0 0 100 100 propagation
48
* From fig 5-10 in Kasser J., Software For Amateur Radio, TAB Books, 1984, page 83
49. Propagation model
Options
1. Logic using IF … THEN statements
2. Table driven approach
3
1 2 5 6 7 8
4
49
50. Logic approach
2230 IF B=B4 OR B=B5 AND H<12 OR B=B3 AND H>=20
AND RND(1)>.5 THEN 2810
2240 IF B=B3 AND (H<20 AND H>=12 OR RND(1)>.5 AND H>=8)
THEN 2810
2250 IF B=B2 AND (H>=20 OR H>=8 AND H<12) AND
RND(1)<0.1 THEN 2810
2260 IF B=B1 AND (H>=12 AND Q=2 AND H<20 OR H>=20
AND RND(1)>.5) THEN 2810
2270 RETURN: REM with Y5 as 0
2810 Y5=1:RETURN
B = Band
B1 … B5 amateur bands
H time of day [block]
50
51. Table driven approach
Use an array Contact(S, B, H)
Can be the Table of stations
Section, Band, Time of day block
Code
During initialization
Store Table values in Contact(S, B, H)
At run time
Set up value of S, B and H
Index into array by value in S, B and H and determine if contact is
possible (C1 = 1)
Y5 = Contact(S, B, H)
If Y5 = 0 THEN C1 = 0 ELSE
If Y5 = 1 THEN C1 = 0 ELSE
C1 = (RND(1) < Y5) : REM Returns Logical 0 or 1
51
52. Characteristics
Logic approach Table array
Does not require Requires knowledge of
knowledge of arrays arrays
Simple set up Complicated set up
Many lines of code for Few lines of code at
execution time execution time
Flexible probabilities Flexible probabilities
By changing values of By changing data
variables embedded in loaded into table
the code without programming
Bonus: Location can be
changed to any Section
By changing data
loaded into table
without programming
52
53. Selection Criteria
Development time
Array approach has learning curve
Schedule issue
3
1 2 5 6 7 8
4
53
54. Decision
3
Use logic approach 1 2 5 6 7 8
4
Lack of domain knowledge
Schedule driven
Risk of non-completion of software
Wrong decision from a ‘systems’ perspective
Hindsight
Completion of simulation was secondary objective
Understanding of the situation was primary
objective
Table-based software is very powerful and flexible
Used later in other software 54
55. Topics
Systems engineering camps
Gaps in what we teach
Case study extract
Pointing out some of the gaps
Lessons learnt from the case study
Systems engineering: an enabler
55
56. Lessons learned
Simulations should be realistic enough to enable successful
completion of mission at the appropriate time
The same problem solving process is used in all phases of the
system development lifecycle
Successful systems engineering needs knowledge and experience
in/of
Systems engineering, application domain, implementation domain
The customer/user needs to be involved in the development
Application domain knowledge
The need to focus on what is important
In M&S it is “the understanding”
Simulations don’t provide answers, they [should] provide understanding
56
57. More lessons learned
Functional flow diagrams may not be best tool to create
relationships between functions
N2 charts are powerful and versatile tools
Incorrect aggregation leads to aggravation
We probably don’t need as many system level
requirements as we think we do
The wording of requirements affects the design
Recursiveness and self-similarity of problem solving
process
Technology influences design decisions
There is knowledge in the development team that is not
delivered with the solution system
Work should be, and can be, fun
57
58. Topics
Systems engineering camps
Gaps in what we teach
Case study extract
Pointing out some of the gaps
Lessons learnt from the case study
Systems engineering: an enabler
58
59. Proposed Maturity Model for measuring
competencies of engineer-leaders
Type I Type II Type III Type IV Type V
Knowledge
Declarative Procedural Conditional Conditional Conditional
Systems engineering
Declarative Declarative Conditional Conditional Conditional
Domain (problem solution)
Cognitive characteristics
System Thinking
Declarative Procedural Conditional Conditional Conditional
Descriptive
No No Procedural No Conditional
Prescriptive
Confused fact Perpetual Pragmatic Pragmatic Strategic re-
Critical Thinking finder analyser performer performer visioner
Individual traits (sample)
Communications Yes Yes Yes Yes Yes
Management No Yes Yes Yes Yes
Leadership No No Yes Yes Yes
59
60. Holistic thinking the problem
Critical thinking
Can’t expect systems engineers to have
knowledge in all domains
Systems thinking
Use continuum perspective/lateral thinking
Reverse position of knowledge
Key questions
What else is used across all domains?
Is there a discipline which is used in all domains?
Answer
Mathematics
60
61. Systems engineering is an
enabler
Systems engineering is an enabler in “the
making things happen” function
In different disciplines in different domains
Applying systems thinking in problem solving
Activities that deals with parts and their
interactions as a whole
Similar to mathematics
61
62. An enabler
“[systems engineering] is a philosophy and a way of
life”
Hitchins, D. K., "Systems Engineering…In Search of the
Elusive Optimum", proceedings of Fourth Annual Symposium
of the INCOSE-UK, 1998.
Systems engineering is the art and science of
creating tangible solutions to complex problems and
issues… (Hitchins)
Application of holistic thinking in the workplace
Product (application domain)
Process (implementation domain)
62
63. Engineer-leaders
Are those people who apply holistic thinking in the
workplace to:
transform puzzling, troubling and uncertain situations into clearly
articulated problems;
Identify optimal conceptual solutions;
Realize those solutions within the constraints of the situation
They perform such functions defined as design, test,
integration, systems engineering, and project management
They need different knowledge and skills pertaining to the
domain and the area of the HKMF in which they are working
They have various job titles (roles)
63
65. A matter of perspective
Functional
Functional and
operational
Process
Take
over the
world Holistic
Operational
Enabler
Problem
65
66. Summary
Systems engineering camps
Gaps in what we teach
Case study extract
Pointing out some of the gaps
Lessons learnt from the case study
Systems engineering: an enabler
66