We work on behalf of members to promote their interests to organisations which have an impact on the development studies community.
At the moment, we are most active in consultations with DFID, ESRC and HEFCE.
This section of the website also provides an opportunity for members to voice their ideas and opinions on topical issues. Please email contributions to email@example.com
Department for Education and Skills (DfES) Consultation on post RAE 2008 metrics-based research assessment proposal - The DSA's Response
- Cecile Jackson, President
Development Studies Association
The document which is the subject of these comments is quite explicitly mainly related to the STEM (sciences) subjects/disciplines, has passing references to the Arts and Humanities, and mentions Economic and Social Research directly only once. Much of the document follows on from a 2004 report called the “Next Steps” which was largely STEM based. The lack of attention to the economic and social sciences is a serious shortcoming, and this affects Development Studies.
The 5 proposed STEM metrics models all are based on
- research income (weighted by source) as the main indicator
- volume measures (numbers of research active academic staff, numbers of postgraduate research students and PhDs awarded
- bibliometric data (impact indices, citation rates)
- research products such as patents, spin-out companies
- perceived quality – end-of-award reports, international panels, institutional research plans
For arts and humanities, the consultation document envisages either
a) a continuing, but streamlined, role for peer review in arts and humanities.
b) A modified metrics model with super-panels which agree on output metrics, with varied weightings to incentive or disincentivise behaviours.
The metrics under the second option would include
Input metrics (research council income, other research income, user-led income, research council success rate)
Volume metrics (PhD numbers/completions, staff with measureable outputs)
Quality/output metrics (bibliometrics, user impact, research council evaluation eg end-of-award reports, peer esteem).
Our response is based on both the STEM proposals and the arts and humanities proposals as it seems likely that the social sciences may experience some kind of hybrid.
The main criticisms of the RAE in the ‘Next Steps’ document are that it is too costly, distorts behaviour in recruitment, publication and character of research undertaken, and that it disadvantages user-driven and interdisciplinary research.
However, we hope that RAE 2008 will overcome these problems and we think the solution lies not in abandoning the current system but in allowing sub-panels freedom to develop distinctive criteria for excellence.
Since development studies has been criticised in previous RAE exercises as too donor-driven, and since its interdisciplinarity has been an enduring problem in winning subject recognition and proper peer review, we may be expected to favour a turn to metrics. The one area where the metrics approach scores better than the current RAE model is in respect of user-focussed research. However we believe that in this respect the proposals go to the other extreme and effectively provide public subsidy for commercial research whilst reducing funding for what QR was intended – blue skies and curiosity driven research. Reducing the latter kind of research funding would have negative consequences for development studies since it encompasses blue skies, applied and directly user-oriented research. We would not wish to see distortions within development studies research as a result of favouring that which is very fundable over that which is more basic. That both are equally important is a proposition which is consistent with the consultation document’s position on equal value for different kinds of research, and its conclusion that on how QR must continue to ‘support strategic, long term research and it should enable speculative research’ (3.8).
As regards the cost of the RAE, we have not seen a full costing which shows that the metrics system would be less costly, and we believe that it could be much more so if the full costs for all parties are accounted for, and the consequences of additional resources diverted into making proposals, and processing the much greater volume of applications, calculated. This could dwarf the cost of the RAE.
Distortions in recruitment and research character are seen as problems with the RAE, but these are issues which could be eased within the current framework, and the behavioural distortions are likely to be worse with the proposed metrics models, as staff with a record of winning grants would be greatly favoured. Incentives would be sharpened since QR would reward the same people who succeed with funders, whilst at present QR supports a dual system of incentives.
There are detailed questions over the application of the metrics approach which have been unresolved even within the peer review system which is being used for the RAE 2008. These include attribution issues (joint authorship and teams) and the shortcomings of the bibliometrics approach with respect to the assessment of the quality of publications. Some of these apply to all social sciences and some are specific to development studies, where research impact requires dissemination strategies which are not readily captured by standard bibliometrics.
We therefore would like to see development studies assessed on the basis of peer review. Insofar as the commitment to metrics has already been made, we would like to see them simply inform the peer review process, with the type and weighting of metrics to be determined by development studies sub-panel.