1. Measuring Settlement Outcomes
Tied to CIC’s Logic Model
2012 OCASI Conference
Dr. Gillian Kerr
gillian@gkerr.com
1

2. Agenda
• The context and background of CIC’s logic model
• How CIC’s policy goals should affect the way that you
design and deliver services
• How policy goals should affect the way that you
measure and communicate the impact of your services
QUESTIONS:
• Any specific topics you want me to cover?
• Who is involved in CIC’s Settlement and Immigration
Renewal Exercise? (correct me when I’m wrong)
• Who works in the federal or provincial government?
2

3. Key underlying concepts
• A budget is ideology stripped of rhetoric. To
understand policy, look at where the money is
going.
• Funders are not rational, and neither are
agencies, so don’t believe policy analysts.
• The most predictable impact of ‘pay for
performance’ is increased cheating. It takes
years to get it right.
3

4. Government of Canada’s 16 Strategic Outcomes
4

5. Breakdown of settlement-related
strategic outcomes
5

6. Settlement Program Architecture 2011-12
6

7. Settlement Program Logic Model – 2010
Policy Development, Program Design and Management Settlement Services
Language
Information &
Program Policy and Program Program Implementation and Needs Assessment Learning & Employment-related
Support Services Awareness Community Connections
Components Development Management and Referrals Skills Services
Services
Development
• Strategic plans • Operational plans • Initial and on-going Enabling services: • Information • Language • Labour market • Individual and
needs assessments – Childminding products training bridging community-level
• Policy, priorities, standar • Program delivery materials and – Transportation
ds and outcomes (including language • Other skills/ • Job search skills bridging, e.g.:
tools assistance • Orientation
assessments) – Provisions for life-skills training – Host/ mentor matches
• Performance • Functional guidance & training sessions
• Referrals to CIC- disabilities training – Volunteers engaged trained
measurement strategy • Promotion • Labour market and supported
and national reports • Data collection and funded and
Other support and outreach information
regional/local/SPO reports community • Cultural awareness, anti-
Outputs • Horizontal coordination services: • Workplace racism, and welcoming
• Regional, local and SPO settlement
• PT consultations services – POE reception services orientation communities services
coordination – Translation
• Research analysis/ and – Interpretation
reports • Service delivery capacity building – Settlement/ crisis
• Best practices and info sharing counselling
• Funding allocation
• Contribution agreements
E 6. Clients, service providers
and CIC are aware of
1. Policies and programming align with departmental and newcomer settlement
government priorities 8. Target population is aware of CIC settlement services
needs
Immediate 2. Program models are evidenced-based, informed by 9. Timely, useful and appropriate CIC settlement services are available in the Official Language of choice (in
7. Referrals and accordance with the Official Languages Act and Policy)
Outcomes stakeholder input and address the barriers & needs of both personalized settlement
newcomers and communities plans are based on 10. Clients obtain the CIC settlement services they need to deal with settlement issues as they emerge
3. Standards, tools, resources and program coordination support assessed settlement
the effective delivery of services) needs
4. Services are efficiently delivered
5. Provision of settlement services across Canada that achieve
A B C D
comparable outcomes 17. Clients are connected to the
15. Clients have knowledge broader community and social
11. Clients have timely, useful and 13. Clients have the official
of the Canadian work networks
accurate information needed to make language skills needed to
environment and are
informed settlement decisions function in Canadian 18. Program participants are aware
Intermediate connected to local
12. Clients understand life in Canada society of newcomers’ needs and
Outcomes labour markets
including laws, rights, responsibilities 14. Clients have the skills/life- contributions and are engaged in
16. Clients have the skills to newcomer settlement
and how to access community skills needed to function in
find and apply for
Key Outcomes resources Canadian society
employment
Expected
A - Orientation Results
B - Language/Skills
A - Orientation
C - Labour Market Access
Ultimate 19. Newcomers find employment commensurate with their skills and experience
Outcomes
DB Community Connection
- - Language/Skills
20. Newcomers enjoy their rights and act on their responsibilities in Canadian society
E - Development and Capacity Building
C - Labour Market Access 21. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian
society
D – Welcoming Communities 22. Newcomers contribute to the economic, social and cultural development needs of Canada (in PAA)
CIC Strategic E - Program and Policy Development
Outcome 3 23. Successful integration of newcomers into society and the promotion of Canadian Citizenship

8. Logic models (1)
• A logic model shows the links between your
daily activities and your long term outcomes
Short term Long term
Activities
outcomes outcomes
• Explains to funders why you are doing what you’re
doing
• Highly simplified summary of what you are trying to
achieve
• Helps to plan your programs to achieve your goals 8

9. Logic models (2)
9

10. Case study: Change to Citizenship Regulations
Proposed amendments to Citizenship Regulations (which govern
implementation of Citizenship Act) require objective evidence of English or
French language ability to be included with applications.
The required level is set to be equivalent to the Canadian Language
Benchmark level 4: “The proposed amendment to section 14 of the
Regulations uses CLB/NCLC benchmarks which are recognized as the official
Canadian standards for describing, measuring and recognizing the language
proficiency of adult immigrants in both English and French.”
• The indicator operationalizes the policy.
• SPOs are not expected to create valid language
tests to measure policy outcomes.
• SPOs should be involved in creating good indicators
to protect legitimate interests of newcomers & SPOs 10

11. Example of evidence-based logic model
11

12. Campbell’s Law for Indicators
"The more any quantitative social indicator is
used for social decision-making, the more
subject it will be to corruption pressures and the
more apt it will be to distort and corrupt the
social processes it is intended to monitor.“
Donald T. Campbell, Assessing the Impact of Planned Social Change, 1976.
12

13. Case study: Developing a
settlement needs assessment tool
In CIC logic model, settlement needs assessment comes first
13

14. What would you be measuring if you
took the logic model seriously?
What would you see if you were delivering services according to the logic model?
14

15. Does CIC care about the logic model?
Before investing in new measures, are they valued by the funder?
Relevant CIC Planning Highlights for Settlement Programs – 2011-2012:
• Complete a review of the Settlement Program to include recommendations on priorities, service
delivery mechanisms, performance measurement strategy and governance structure.
• Collaborate with provinces and territories to develop an evidence base on settlement outcomes for
immigrants and draft a pan-Canadian framework for settlement outcomes for consideration by
Ministers.
• Plan and implement the methodology, research approach and launch of the Settlement Program
evaluation.
• Complete implementation of action items derived from evaluations of the Official Languages
Program, the Language Instruction for Newcomers to Canada program, the Immigrant Settlement
and Adaptation Program, the Host Program and the Welcoming Communities Initiative, as well as
the audit of the administration of settlement programs.
• Complete field testing of the Portfolio-Based Language Assessment and launch a pilot project to test
the delivery infrastructure required for a new national test of language achievement. The objectives
will be to provide newcomers with a language credential and improve CIC’s ability to evaluate the
impact and outcomes of language programs.
• Implement a standardized approach to newcomer needs assessment and associated settlement
plans across service delivery organizations, and ensure collection of baseline data.
• Include a website on best practices in settlement to share evidence-based practices among service
delivery organizations across the country.
• Expand local partnerships and pilot projects from Ontario to other parts of Canada, focusing on
larger urban centres.
From CIC Report on Plans and Priorities 2011-2012 www.tbs-sct.gc.ca/rpp/2011-2012/inst/imc/imc-
eng.pdf

16. Are current program policy initiatives consistent with
logic model?
Settlement Information Renewal Exercise (SIRE)
CIC is committed to improving Needs Assessment & Referral
Services (NARS) in 3 ways:
• Establishing data collection and reporting protocol to collect
baseline data on settlement needs (iCARE’s first module is
NARS)
• Developing settlement planning tools for newcomers
(e.g., Living in Canada site)
• Developing guidance and standards for CIC staff and SPOs
Conclusion:
Yes, it looks safe to rely on the logic model, and it’s a
good time to influence CIC tool development.
16

17. Can you use the CIC settlement tools?
17

18. Consider cooperative development
• SPOs reluctant to share resources with
colleagues, resulting in a whole lot of half-baked tools
that turn well-intentioned staff into thieves or hoarders
and prevent improvement.
• Use open-sourced/open-access approach to create valid
and legitimate tools that are consistent with sector’s
values and outcomes.
• Work with CIC policy analysts to co-develop - they love to
be appreciated and it will result in more integration.
• Use other initiatives to lever sector resources –
e.g., Canadian Centre for Financial Literacy’s settlement
needs assessment tool.
18