SlideShare una empresa de Scribd logo
1 de 22
Descargar para leer sin conexión
#AIIM12




   Better Than the Black Box:
An Empirical Approach to Taxonomy
          Development

                  Patrick Lambe
          Straits Knowledge, Singapore
#AIIM12
Three Traps

 Chasing your tail




 Experts know best

                               K e e p O u t !!
                       T a x o n o m is t a t W o r k




 Enclosure

    #AIIM12
Being Empirical Means…
   Warrant - grounding taxonomy in
           Content warrant
           User warrant
           De facto standards in use
   Testing
           Open card sorts
           Closed card sorts
           Scenario-based tests
           Load/balance tests
   Consultation
           Targeted on gaps, technical accuracy, NOT general opinions
           Transparent, tough issue resolution process
For enterprise taxonomies, unmanaged exposure to unfocused opinions
                  are the Taxonomist’s greatest enemy
          #AIIM12
1. Warrant - the role of the
       knowledge audit



#AIIM12
What is a knowledge audit?

A knowledge audit is a survey-cum-
inventory of the different kinds of knowledge
that are:
   currently in use
   needed but not currently in use (gaps and
   desired improvements)


…to support the work and strategic
objectives of your organisation

                                                Natural




     #AIIM12
Knowledge Audit

 Half day workshop
           2-3 operational managers
           per dept
                                                                      QuickTimeª and a
           Knowledge maps, culture                                     decompressor
                                                              are needed to see this picture.

           analysis, pain points
 Site visits
 Gallery viewing




Ex a mp le o f a gal ler y v ie w ing exe rcis e
                   #AIIM12
Knowledge Maps
Input to activity    Div & Activity   Output from activity




                                                             Subscribing
                                                               divisions


                                                              Unique code
                                                             of knowledge
                                                                    asset


                                                                  Type of
                                                               knowledge
                                                                    asset


                                                               Description
                                                             of knowledge
                                                                     asset
      #AIIM12
Expected outcomes
Knowledge maps give us:
 Evidence for how people describe their knowledge assets (content warrant)
 Maps linking knowledge assets to activities (context of use - user warrant)

Site visits give us:
 Observational evidence for organizing principles in use (user warrant)
 Evidence (printouts, photos) of how folders, physical and digital documents,
    emails, shared resources are labeled and organized (content and user
    warrant)

Analysis of the knowledge maps gives insights into Knowledge gaps, Knowledge
  risks, Critical knowledge requiring protection, Sharing and collaboration
  opportunities, Knowledge flows and blockages - and helps identify
  knowledge priorities, useful in designing the taxonomy

      #AIIM12
2. Testing - evidence that people
     can use the taxonomy
            effectively



 #AIIM12
Testing Techniques
 Open card sorting - to find “natural
  categories” among users and variance
  between them - at the start of a
  taxonomy design
 Closed card sorting - to test usability of
  top level categories - to test a draft
  taxonomy
 Scenario based testing - to uncover
  navigation, predictability issues - to test
  a completed taxonomy - scenarios are
  derived from user warrant
                                                   The conversations are
 Balance/load testing - to analyze the
                                                    as important as the
  distribution of content across a
                                                          results
  populated taxonomy in use
                                                Donna Spencer, Card Sorting Rosenfeld
                                                Media 2009
        #AIIM12
Case Study

     Review of existing taxonomy in use
           Public health agency



#AIIM12
Balance


                53% of taxonomy folders
                have no content
                76% of populated folders
                have fewer than 5
                documents
                9% of populated folders
                have more than 15 items
                15% of populated folders
                (7% of all taxonomy folders)
                have 5-15 items and could
                be considered navigable


#AIIM12
Balance & Accessibility




                             35% of content is 5 or
                             more levels deep
                             83% of content is 4 or
                             more levels deep




#AIIM12
What we found

From the usability tests:
Avge time       Avge         Avge        Task      Confident   Consistency
 / task        clicks /    dead ends   abandoned     final     of decisions
                 task       / task                 decision




   2.5         18 clicks      1.5        25%         46%          35%
minutes                     folders
                            opened




     #AIIM12
Taxonomy design issues
 As Navigation Tool:
      Too narrow and deep
      Unpredictable, different principles of organisation applied in parallel
 As Controlled Vocabulary:
      Same categories repeated across the different parts of the structure -
       ambiguity
      No overall design principle to govern additions to taxonomy
 As Classification Scheme:
      Fewer than half the taxonomy is populated, extreme lack of balance in
       distribution of content
      Only 35% consistency in usability tests
Solution: break the hierarchy into three facets: Business Activities, Document
Types, Health Topics, using warrant from the existing classification plus
knowledge maps from a knowledge audit
       #AIIM12
Taxonomy Redesign
 This combination of facets gave a very big reduction in frustration
  levels, increased the richness of information captured about
  documents, and substantially increased consistency and confidence




  Because in a faceted system each document needs between 2 and 3 tags to achieve the same level of
  precision as in a single hierarchy, we have divided the actual consistency rate of individual tags by 2.5 to
  make a meaningful comparison.

            #AIIM12
3. Consultation - transparent
but rigorous issue resolution
           process



#AIIM12
Consultation Means…
 “We’d like you to examine and              Focus the feedback on the areas
  comment on the following sections          of the taxonomy where testing
  of the taxonomy because of your             indicates that you need advice
  expertise area”
                                                You’ll get these anyway - but
 “We do not want your opinions, we
                                                  you’ve made it possible to
  want targeted feedback on:                        Say “no” to opinions
      Gaps                                            without warrant
      Accuracy of language
      Accuracy of structure and
       relationships                          Try to ensure that all suggested
      Consolidating and simplifying    changes have some warrant underpinning
       categories”                        them - labels in use, standards in use,
                                          situations requiring this language; and
 “Please give reasons for your            that feedback represents the general
  suggestions, based on how people              user not the specialist user
  will actually use these categories”
       #AIIM12
Issue Resolution




#AIIM12
Issue Resolution




#AIIM12
Being Empirical Also Means…

 Being better equipped to
  manage the politics of
  taxonomy development
 Being able to establish
  credibility for the taxonomy
  development process and
  products - including being able
  to justify denying requests
 Reducing the risk of project
  delays resulting from claims on
  the taxonomy from narrowly
  informed special interests

      #AIIM12
#AIIM12




Any Questions?
www.greenchameleon.com


plambe@straitsknowledge.com

   #AIIM12

Más contenido relacionado

Similar a Aiim conference patrick lambe slides

Designing an effective information architecture
Designing an effective information architectureDesigning an effective information architecture
Designing an effective information architectureoptimalworkshop
 
Metadata and Taxonomies for More Flexible Information Architecture
Metadata and Taxonomies for More Flexible Information Architecture Metadata and Taxonomies for More Flexible Information Architecture
Metadata and Taxonomies for More Flexible Information Architecture jrhowe
 
S doherty counting_dragons_dita-reuse
S doherty counting_dragons_dita-reuseS doherty counting_dragons_dita-reuse
S doherty counting_dragons_dita-reuseStan Doherty
 
KMWorld Martin Briefing
KMWorld Martin BriefingKMWorld Martin Briefing
KMWorld Martin Briefingmartingarland
 
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for EveryoneGDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for EveryoneJames Anderson
 
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...Leading the Evolution of Work, Process and Technology: How to Plan and Execut...
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...CA Technologies
 
Containers, Reuse and Security: What’s in Your Wallet?
Containers, Reuse and Security: What’s in Your Wallet?Containers, Reuse and Security: What’s in Your Wallet?
Containers, Reuse and Security: What’s in Your Wallet?CA Technologies
 
Mesh Labs Introduction June 2012
Mesh Labs Introduction June 2012Mesh Labs Introduction June 2012
Mesh Labs Introduction June 2012Umesh Ramalingachar
 
How Search 2.0 Has Been Redefined by Enterprise 2.0
How Search 2.0 Has Been Redefined by Enterprise 2.0How Search 2.0 Has Been Redefined by Enterprise 2.0
How Search 2.0 Has Been Redefined by Enterprise 2.0Enterprise 2.0 Conference
 
Km ver 1.0 student
Km   ver 1.0 studentKm   ver 1.0 student
Km ver 1.0 studentplpictimatec
 
Data analysis – using computers for presentation
Data analysis – using computers for presentationData analysis – using computers for presentation
Data analysis – using computers for presentationNoonapau
 
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...Jai Natarajan
 
Data analysis – using computers
Data analysis – using computersData analysis – using computers
Data analysis – using computersNoonapau
 
Taxonomy Validation
Taxonomy ValidationTaxonomy Validation
Taxonomy ValidationDave Cooksey
 
Qualitative Data Analysis
Qualitative Data  AnalysisQualitative Data  Analysis
Qualitative Data AnalysisMaisa Rahman
 
Identifying Security Risks Using Auto-Tagging and Text Analytics
Identifying Security Risks Using Auto-Tagging and Text AnalyticsIdentifying Security Risks Using Auto-Tagging and Text Analytics
Identifying Security Risks Using Auto-Tagging and Text AnalyticsEnterprise Knowledge
 
7 dee finding the right methodologies marshall sponder - 9-12-12 - submitted
7 dee finding the right methodologies   marshall sponder - 9-12-12 - submitted7 dee finding the right methodologies   marshall sponder - 9-12-12 - submitted
7 dee finding the right methodologies marshall sponder - 9-12-12 - submittedMarshall Sponder
 

Similar a Aiim conference patrick lambe slides (20)

Julio berdegue from policy to research and back again
Julio berdegue   from policy to research and back againJulio berdegue   from policy to research and back again
Julio berdegue from policy to research and back again
 
Designing an effective information architecture
Designing an effective information architectureDesigning an effective information architecture
Designing an effective information architecture
 
Metadata and Taxonomies for More Flexible Information Architecture
Metadata and Taxonomies for More Flexible Information Architecture Metadata and Taxonomies for More Flexible Information Architecture
Metadata and Taxonomies for More Flexible Information Architecture
 
S doherty counting_dragons_dita-reuse
S doherty counting_dragons_dita-reuseS doherty counting_dragons_dita-reuse
S doherty counting_dragons_dita-reuse
 
KMWorld Martin Briefing
KMWorld Martin BriefingKMWorld Martin Briefing
KMWorld Martin Briefing
 
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for EveryoneGDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
GDG Cloud Southlake #17: Meg Dickey-Kurdziolek: Explainable AI is for Everyone
 
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...Leading the Evolution of Work, Process and Technology: How to Plan and Execut...
Leading the Evolution of Work, Process and Technology: How to Plan and Execut...
 
Containers, Reuse and Security: What’s in Your Wallet?
Containers, Reuse and Security: What’s in Your Wallet?Containers, Reuse and Security: What’s in Your Wallet?
Containers, Reuse and Security: What’s in Your Wallet?
 
Mesh Labs Introduction June 2012
Mesh Labs Introduction June 2012Mesh Labs Introduction June 2012
Mesh Labs Introduction June 2012
 
How Search 2.0 Has Been Redefined by Enterprise 2.0
How Search 2.0 Has Been Redefined by Enterprise 2.0How Search 2.0 Has Been Redefined by Enterprise 2.0
How Search 2.0 Has Been Redefined by Enterprise 2.0
 
Km ver 1.0 student
Km   ver 1.0 studentKm   ver 1.0 student
Km ver 1.0 student
 
Data analysis – using computers for presentation
Data analysis – using computers for presentationData analysis – using computers for presentation
Data analysis – using computers for presentation
 
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...
Enterprise Grade Data Labeling - Design Your Ground Truth to Scale in Produ...
 
Data analysis – using computers
Data analysis – using computersData analysis – using computers
Data analysis – using computers
 
Taxonomy Governance and Iteration
Taxonomy Governance and IterationTaxonomy Governance and Iteration
Taxonomy Governance and Iteration
 
Taxonomy Validation
Taxonomy ValidationTaxonomy Validation
Taxonomy Validation
 
Qualitative Data Analysis
Qualitative Data  AnalysisQualitative Data  Analysis
Qualitative Data Analysis
 
Identifying Security Risks Using Auto-Tagging and Text Analytics
Identifying Security Risks Using Auto-Tagging and Text AnalyticsIdentifying Security Risks Using Auto-Tagging and Text Analytics
Identifying Security Risks Using Auto-Tagging and Text Analytics
 
7 dee finding the right methodologies marshall sponder - 9-12-12 - submitted
7 dee finding the right methodologies   marshall sponder - 9-12-12 - submitted7 dee finding the right methodologies   marshall sponder - 9-12-12 - submitted
7 dee finding the right methodologies marshall sponder - 9-12-12 - submitted
 
Modul 3 knowledge mapping
Modul 3   knowledge mappingModul 3   knowledge mapping
Modul 3 knowledge mapping
 

Último

Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfAnhNguyen97152
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxWorkforce Group
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessAPCO
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakEditores1
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003believeminhh
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Lviv Startup Club
 
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBBPMedia1
 
Data skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsData skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsyasinnathani
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...Brian Solis
 
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...Khaled Al Awadi
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...IMARC Group
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toumarfarooquejamali32
 
Live-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarLive-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarNathanielSchmuck
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023Steve Rader
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Winbusinessin
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.ukaroemirsr
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGlokeshwarmaha
 

Último (20)

Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdfGraham and Doddsville - Issue 1 - Winter 2006 (1).pdf
Graham and Doddsville - Issue 1 - Winter 2006 (1).pdf
 
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptxCracking the ‘Business Process Outsourcing’ Code Main.pptx
Cracking the ‘Business Process Outsourcing’ Code Main.pptx
 
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for BusinessQ2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
Q2 2024 APCO Geopolitical Radar - The Global Operating Environment for Business
 
Tata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerakTata Kelola Bisnis perushaan yang bergerak
Tata Kelola Bisnis perushaan yang bergerak
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
The Vietnam Believer Newsletter_MARCH 25, 2024_EN_Vol. 003
 
Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)Michael Vidyakin: Introduction to PMO (UA)
Michael Vidyakin: Introduction to PMO (UA)
 
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John MeulemansBCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
BCE24 | Virtual Brand Ambassadors: Making Brands Personal - John Meulemans
 
Data skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story pointsData skills for Agile Teams- Killing story points
Data skills for Agile Teams- Killing story points
 
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
The End of Business as Usual: Rewire the Way You Work to Succeed in the Consu...
 
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...NewBase  25 March  2024  Energy News issue - 1710 by Khaled Al Awadi_compress...
NewBase 25 March 2024 Energy News issue - 1710 by Khaled Al Awadi_compress...
 
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
Boat Trailers Market PPT: Growth, Outlook, Demand, Keyplayer Analysis and Opp...
 
Lecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb toLecture_6.pptx English speaking easyb to
Lecture_6.pptx English speaking easyb to
 
Live-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry WebinarLive-Streaming in the Music Industry Webinar
Live-Streaming in the Music Industry Webinar
 
NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023NASA CoCEI Scaling Strategy - November 2023
NASA CoCEI Scaling Strategy - November 2023
 
Investment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV IndustriesInvestment Opportunity for Thailand's Automotive & EV Industries
Investment Opportunity for Thailand's Automotive & EV Industries
 
Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024Ethical stalking by Mark Williams. UpliftLive 2024
Ethical stalking by Mark Williams. UpliftLive 2024
 
Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024Borderless Access - Global Panel book-unlock 2024
Borderless Access - Global Panel book-unlock 2024
 
7movierulz.uk
7movierulz.uk7movierulz.uk
7movierulz.uk
 
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISINGUNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
UNLEASHING THE POWER OF PROGRAMMATIC ADVERTISING
 

Aiim conference patrick lambe slides

  • 1. #AIIM12 Better Than the Black Box: An Empirical Approach to Taxonomy Development Patrick Lambe Straits Knowledge, Singapore #AIIM12
  • 2. Three Traps  Chasing your tail  Experts know best K e e p O u t !! T a x o n o m is t a t W o r k  Enclosure #AIIM12
  • 3. Being Empirical Means…  Warrant - grounding taxonomy in  Content warrant  User warrant  De facto standards in use  Testing  Open card sorts  Closed card sorts  Scenario-based tests  Load/balance tests  Consultation  Targeted on gaps, technical accuracy, NOT general opinions  Transparent, tough issue resolution process For enterprise taxonomies, unmanaged exposure to unfocused opinions are the Taxonomist’s greatest enemy #AIIM12
  • 4. 1. Warrant - the role of the knowledge audit #AIIM12
  • 5. What is a knowledge audit? A knowledge audit is a survey-cum- inventory of the different kinds of knowledge that are: currently in use needed but not currently in use (gaps and desired improvements) …to support the work and strategic objectives of your organisation Natural #AIIM12
  • 6. Knowledge Audit Half day workshop 2-3 operational managers per dept QuickTimeª and a Knowledge maps, culture decompressor are needed to see this picture. analysis, pain points Site visits Gallery viewing Ex a mp le o f a gal ler y v ie w ing exe rcis e #AIIM12
  • 7. Knowledge Maps Input to activity Div & Activity Output from activity Subscribing divisions Unique code of knowledge asset Type of knowledge asset Description of knowledge asset #AIIM12
  • 8. Expected outcomes Knowledge maps give us:  Evidence for how people describe their knowledge assets (content warrant)  Maps linking knowledge assets to activities (context of use - user warrant) Site visits give us:  Observational evidence for organizing principles in use (user warrant)  Evidence (printouts, photos) of how folders, physical and digital documents, emails, shared resources are labeled and organized (content and user warrant) Analysis of the knowledge maps gives insights into Knowledge gaps, Knowledge risks, Critical knowledge requiring protection, Sharing and collaboration opportunities, Knowledge flows and blockages - and helps identify knowledge priorities, useful in designing the taxonomy #AIIM12
  • 9. 2. Testing - evidence that people can use the taxonomy effectively #AIIM12
  • 10. Testing Techniques  Open card sorting - to find “natural categories” among users and variance between them - at the start of a taxonomy design  Closed card sorting - to test usability of top level categories - to test a draft taxonomy  Scenario based testing - to uncover navigation, predictability issues - to test a completed taxonomy - scenarios are derived from user warrant The conversations are  Balance/load testing - to analyze the as important as the distribution of content across a results populated taxonomy in use Donna Spencer, Card Sorting Rosenfeld Media 2009 #AIIM12
  • 11. Case Study Review of existing taxonomy in use Public health agency #AIIM12
  • 12. Balance 53% of taxonomy folders have no content 76% of populated folders have fewer than 5 documents 9% of populated folders have more than 15 items 15% of populated folders (7% of all taxonomy folders) have 5-15 items and could be considered navigable #AIIM12
  • 13. Balance & Accessibility 35% of content is 5 or more levels deep 83% of content is 4 or more levels deep #AIIM12
  • 14. What we found From the usability tests: Avge time Avge Avge Task Confident Consistency / task clicks / dead ends abandoned final of decisions task / task decision 2.5 18 clicks 1.5 25% 46% 35% minutes folders opened #AIIM12
  • 15. Taxonomy design issues  As Navigation Tool:  Too narrow and deep  Unpredictable, different principles of organisation applied in parallel  As Controlled Vocabulary:  Same categories repeated across the different parts of the structure - ambiguity  No overall design principle to govern additions to taxonomy  As Classification Scheme:  Fewer than half the taxonomy is populated, extreme lack of balance in distribution of content  Only 35% consistency in usability tests Solution: break the hierarchy into three facets: Business Activities, Document Types, Health Topics, using warrant from the existing classification plus knowledge maps from a knowledge audit #AIIM12
  • 16. Taxonomy Redesign  This combination of facets gave a very big reduction in frustration levels, increased the richness of information captured about documents, and substantially increased consistency and confidence Because in a faceted system each document needs between 2 and 3 tags to achieve the same level of precision as in a single hierarchy, we have divided the actual consistency rate of individual tags by 2.5 to make a meaningful comparison. #AIIM12
  • 17. 3. Consultation - transparent but rigorous issue resolution process #AIIM12
  • 18. Consultation Means…  “We’d like you to examine and Focus the feedback on the areas comment on the following sections of the taxonomy where testing of the taxonomy because of your indicates that you need advice expertise area” You’ll get these anyway - but  “We do not want your opinions, we you’ve made it possible to want targeted feedback on: Say “no” to opinions  Gaps without warrant  Accuracy of language  Accuracy of structure and relationships Try to ensure that all suggested  Consolidating and simplifying changes have some warrant underpinning categories” them - labels in use, standards in use, situations requiring this language; and  “Please give reasons for your that feedback represents the general suggestions, based on how people user not the specialist user will actually use these categories” #AIIM12
  • 21. Being Empirical Also Means…  Being better equipped to manage the politics of taxonomy development  Being able to establish credibility for the taxonomy development process and products - including being able to justify denying requests  Reducing the risk of project delays resulting from claims on the taxonomy from narrowly informed special interests #AIIM12