1. Evaluation and Ranking of National Research Institutes Project Group of National Research Institute Ranking Institute of Policy and Management, Chinese Academy of Science Presented by Prof. WB Liu Kent Business School, University of Kent, UK 2010-07-01 ， Canterbury, UK

11. Approach Overall Subject 1 … Subject N E1 output E2 efficiency E3 E2 E1 Subjects Ranking and then Overall Ranking SSM ANALYSIS E3 effectiveness

14. Sample- Scale difference

15. Sample- subjects distribution USDA 50% 4% 0% 2% 14% 2% 1% 0% 5% 0% 0% 1% 1% 1% 19% 1 ． BiSci 2 ． Chem 3 ． Comp 4 ． Eng 5 ． Envi 6 ．地学 7 ．材料 8 ．数学 9 ．医学 10 ．神经 11 ．物理 12 ．社科 13 ．多学科 14 ．免疫 15 ．农业 CAS 11% 27% 2% 6% 3% 9% 10% 2% 2% 1% 24% 0% 2% 0% 1% 1 ． BiSci 2 ． Chem 3 ． Comp 4 ． Eng 5 ． Envi 6 ．地学 7 ．材料 8 ．数学 9 ．医学 10 ．神经 11 ．物理 12 ．社科 13 ．多学科 14 ．免疫 15 ．农业

16. Indicators 2000 European Commission Office （ EC ） Knowledge Science and Innovation （ KSI ） 2003 Economic and Social Commission for Western Asia （ ESCWA ） New Science and Technology Indicators (NST) 2002-2006 National Science Board （ NSB ） Science and Engineering Indicators-US (SEI) 2003 World Economic Forum （ WEF ） Global competition indicators （ GCI ） 2003 （ OECD ） Main Science and Technology Indicators （ MSTI ） 2004 Lausanne （ IMD ） World Competitiveness Yearbook （ WCY ） 2004-2006 1991-2001 1000-2004 2006 WuHan University Research institute of Management Science, GuangDong, Graduates Development Centre in MOE, Evaluation and Ranking of Chinese University (CUE) 2005 WuHan Univeristy World University Research competitiveness Index (WUSRCI) 2006 CAS CAS Innovation Index (CASCCI) 2004 MOST, China Chinese Science Technology Indicator (CSTY) 2005 MOST, China Chinese Science and Technology Year Book (CSSTY) 2003—2005 Bureau of statistics, China Chinese Statistic Year Book (CSY) year Organization Title

19. Indicators system  3Es theory Inputs Outputs internal operational system External environment E2=Efficiency E1=Efficacy E3=Effectiveness

24. A state-owned system to improve the originality, significance, reputation and sustainability of CAS basic research in the natural sciences by developing the research capabilities and infrastructure, and by improving resources utilisation of CAS in order to benefit Chinese social and economic development and to enrich human knowledge. C: World scientific community A: CAS headquarters and institutes T: Improvement in significance, reputation and sustainability W: Goal of CAS is improvement in research outcomes O: Chinese state council E: Economic and cultural constraints, current research infrastructure (R): Institutes, scientists, money, equipment Basic research in CAS — CAS as a whole

25. A CAS-owned system to enrich the world’s knowledge in a particular scientific domain with original and significant research by identifying potential areas of discovery, developing the capabilities to undertake appropriate research, carrying out the research and disseminating it through prestigious channels in order to enhance the reputation, significance and sustainability of CAS and the Institute. C: World scientific community in discipline, CAS (reputation) A: Institute personnel T: Generating significant new knowledge W: Role of an Institute within CAS is to generate significant new knowledge O: CAS E: CAS procedures, funding (R): Scientists, other staff, funding, equipment Basic research in CAS — Generic Institute

26. Monitor Control Criteria for Efficacy Efficiency Operational Control Strategic Control 1. Identify potentially significant areas of discovery 2. Appreciate current resources, capabilities and projects 3. Decide which opportunities to pursue 4. Obtain necessary resources 5. Conduct research 6. Disseminate results through prestigious channels 7. Contribute to the reputation, resources and sustainability Generic Institute Conceptual Model Monitor Control Criteria for Effectiveness

27. Example: Areas of possible research in related disciplines Areas of significance identified RD1 A system to identify research opportunities for institute X that are significant and original bearing in mind the resources and capabilities needed, by effective external scanning and by improving internal discussion and communication in order to contribute to decide which opportunities to pursue which contribute to enhance reputation, resources and sustainability of the Institute. Resources: Funding, hardware investments, staff Research levels Research reputation CAS procedure 1.1 Scan the external environment for research opportunities 1.2 consider current resources and capabilities 1.3 Encourage discussion and communication within and among institutes 1.4 decide on those which are potentially possible 1.5 specify extra requirements for potentially possibilities E1: does it produce possible further research topics? E2: E1/resources (eg time, funds, people) E3: do we get significant research done which contributes research reputation and resources enhancement? CM1. Identify potentially significant areas of discovery

29. Primary extracted efficacy indicators

30. * + E3 High quality graduates * * + + + E1 Graduate enrollment + + E1 Joint projects + E1 Joint-lab + E1 Exchange graduates E1 Megic-science infrastructure others * E2 Rate of excellent scientist E2 Invited talks per staff * E2 Awards per staff * * + * E3 Excellent scientists E3 Invited talks * * * + * * E3 Awards impact E2 Profit of owned company * E1 Vendibility of owned company * * * E2 Internal funding: external funding * + + + + E1 Rate of capital asserts + + E1 Competitive research funding + + E2 Research funding per staff + + + + + + + + E1 Research expenditure Financial measure + * * * E2 Award Patents per staff + * * * E3 Commercialized patens E3 Core patens awards * * * + * + + * * E1 Number of patents awards * + * * * * E1 Number of patents application patents * E2 SCI ， SSCI papers/funding * E2 Percentage of Top papers * E2 Percentage of cooperation papers * * E2 Top 0.1%-10% paper per staff * * * E2 SCI ， SSCI paper per staff E3 Top papers based on megic-science infrastructure + * E3 Top10% papers + * E3 Top 1% papers + * * E3 Top 0.1% papers + + + E2 SCI ， SSCI cited times per papers + + + + + E3 SCI ， SSCI total citations + + + + E1 SCI ， SSCI cooperation papers + + * + + + + + E1 SCI ， SSCI papers Biblo- Metrics KSI NST SEI GCI MSTI WCY FUCY CUE WUSRCI CASCCI CSTY CSSTY CSY 3Es Indicators Comparisons 3E indicators

33. Random Sensitivity Analysis E3(Max) Change (5%) Change (10%) NO. 机构名称 Plan1 Plan 2 Plan 1 Plan 2 Plan 1 Plan 2 Average Change 0.0978 0.0477 0.1363 0.0666 1 美国农业部 0.877238 0.8772 0 0 0 0 2 法国农业科学研究院 0.192529 0.2536 0.0257 0.0097 0.0247 0.0093 3 西班牙科学研究委员会 0.150366 0.1796 0.0317 0.0132 0.0324 0.0135 4 加拿大农业及农业食 0.110103 0.1493 0.0302 0.0111 0.0247 0.0091 5 日本农林水产省 0.050875 0.0518 0.0490 0.0240 0.0829 0.0407 6 澳大利亚联邦科学与工业研究组织 0.112674 0.1512 0.0554 0.0206 0.0590 0.0220 7 中国科学院 0.116528 0.1735 0.0235 0.0078 0.0482 0.0161 8 印度农业研究理事会 0.021313 0.0213 0.1763 0.0881 0.4217 0.2108 10 英国研究理事会 0.111147 0.1403 0.0426 0.0168 0.0572 0.0226 11 印度科学与工业研究理 0.05148 0.0550 0.1300 0.0607 0.1448 0.0676

38. Case study － aggregation method based on rank-added method 36 8 13 27 34 5 23 23 10 13 7 21 43 15 263 Unit28 6 37 25 11 31 12 29 4 29 31 36 9 1 14 261 Unit20 23 12 10 33 23 13 2 21 5 12 28 40 37 13 259 Unit17 17 25 29 24 20 25 17 13 16 11 23 20 7 12 247 Unit13 8 31 11 14 15 33 32 12 31 19 27 6 2 11 241 Unit15 14 15 8 12 7 16 24 31 26 3 16 13 31 10 216 Unit9 1 27 2 1 9 24 46 15 34 8 20 1 19 9 207 Unit5 19 14 12 8 24 10 12 17 8 18 19 15 28 8 204 Unit10 12 11 18 16 25 9 14 9 22 14 9 12 6 7 177 Unit8 9 16 19 9 10 11 13 27 7 9 10 16 17 6 173 Unit6 27 7 23 17 1 8 10 1 2 20 8 18 20 5 162 Unit3 7 1 4 20 5 3 6 19 3 6 4 14 29 4 121 Unit4 13 5 7 22 3 2 8 8 4 5 3 11 8 3 99 Unit7 10 2 1 7 6 6 1 6 15 4 1 3 18 2 80 Unit1 5 4 3 4 2 4 3 5 1 2 2 4 14 1 53 Unit2 Soci Phys Multi- Med math Mate Geo Envi Eng. Comp Che BioS Agr. Rank Score

39. Case study － aggregation method based on score-added method 10.26 6.78 6.25 13.74 11.05 5.25 5.89 6.31 5.40 9.68 7.90 5.49 8.78 MAX 0.31 1.68 1.24 0.14 0.40 1.26 4.58 1.05 2.40 1.08 0.25 0.14 0.13 14 14.6666 Unit17 1.06 0.06 1.24 0.84 0.89 0.09 0.20 1.56 0.32 0.73 0.32 2.78 5.18 13 15.2697 Unit15 1.03 1.38 0.66 1.34 1.29 1.33 1.25 0.47 2.11 1.68 1.54 0.99 0.49 12 15.5748 Unit6 0.15 0.07 0.01 0.04 4.16 0.00 0.02 0.05 1.55 9.68 0.04 0.12 0.00 11 15.8998 Unit54 0.74 1.78 0.68 0.72 0.28 1.44 1.15 1.88 0.72 0.99 1.55 1.13 2.87 10 15.926 Unit8 2.78 0.08 2.62 4.97 0.84 0.24 0.04 0.13 0.14 1.86 0.66 2.29 0.32 9 16.9447 Unit11 0.02 2.40 3.04 2.50 0.15 0.69 0.27 0.15 2.01 0.90 0.77 3.96 0.11 8 16.965 Unit18 1.32 0.04 0.41 1.13 0.09 1.30 0.24 2.80 0.34 0.26 0.12 1.83 8.78 7 18.657 Unit20 0.25 2.74 0.42 0.58 11.05 1.82 1.63 6.31 4.97 0.60 1.77 0.93 0.39 6 33.4436 Unit3 1.18 6.78 3.08 0.48 2.37 4.54 2.92 1.16 4.56 2.62 4.09 1.01 0.22 5 35.0232 Unit4 0.72 4.00 2.36 0.42 7.58 5.25 2.03 1.89 3.81 2.66 4.15 1.19 1.68 4 37.7177 Unit7 10.26 0.08 5.77 13.74 1.38 0.36 0.05 1.42 0.24 1.73 0.68 5.49 0.39 3 41.5793 Unit5 0.97 6.09 6.25 1.90 2.15 2.92 5.89 2.64 1.50 3.18 7.90 3.56 0.48 2 45.4129 Unit1 1.67 4.17 4.32 2.69 9.45 4.48 4.20 2.69 5.40 4.12 4.65 3.52 0.81 1 52.1937 Unit2 Soci Phys Multi- Med math Mate Geo Envi Eng. Comp Che BioS Agr. Rank Score

40. Case study Unit 1 Unit 2 Unit 7 Unit 10 Unit 6

42. Thanks