Programme

10:15 – 10:30 Coffee and Registration

10:30 – 10:45 Welcome

10:45 – 11:45 Professor David Byrne, University of Durham: ‘Mixed methods, methodology and the impacts of Social Policy Research: a challenge to simplistic explanations for a complex world’

11:45 – 1:15 Mixed Methods and Methodologies of Social Policy Research

Dr Michelle Farr, University of Bath: Evaluation and impact: using mixed methods and realism to understand complexity

Many public service organisations are needing to prove their impact within an increasingly competitive public procurement process. The Open Public Services agenda and the Health and Social Care Act (2012) create distinct challenges for public service organisations in negotiating commissioning and procurement practices to ensure that they can continue to provide services in a context of austerity and local government cuts. In addition the Public Services (Social Value) Act (2012) directs local authorities to consider how services procured improve economic, social and environmental well-being of areas. Many different evaluation methodologies and impact measurement tools are developing and being employed by a range of different social purpose organisations within this policy framework. These methods are often based on models that assume linear input, process and output pathways. It is assumed that disaggregation is both possible and desirable and that different organisations can detach their own interventions from others’ actions and outcomes. These methods follow in the philosophical footprints of evidence based policy, which assumes that the policy world can be broken down and analysed through the separation of “independent interventions” (Room 2013).

This paper uses different forms of realism and complexity theories (Pawson 2013, Byrne 2013 and Room 2013) to provide a philosophical framework for developments in evaluation methods that contest this linear approach and combine quantitative, qualitative and action research methods. It reflects on current research projects that have been co-produced with different practitioners in the fields of health and welfare services and illustrates how different modes and methods of evaluation can be integrated within a realist perspective that takes account of the complexity within which public service organisations work. It describes methods of evaluation that have been co-produced with practitioners and service users to understand the impact of services for service users and illustrates how mixed methods can provide deeper understanding of the impact of policies and public services for both commissioners and practitioners. This co-production of research with public service organisations brings certain opportunities and challenges for the academic researcher, providing clearer routes to impact yet incorporating new challenges in the areas of the power and politics of evidence-based interventions and research co-production.

Please click here for a copy of the presentation.

Serena Romano, Università degli Studi di Napoli Federico II: Policy design, ideas and the construction of poverty: new perspectives in social policy analysis.

One of the main elements in the current debate over poverty concerns its “constructed” nature. Most contemporary studies recognise the fact that poverty is never a neutral phenomenon and that it is by all means constructed socially and politically. Far from being a distinctive feature of past English Poor Laws, a more or less explicit distinction between alleged “deserving” and undeserving” categories of poor is a major aspect in the definition of all contemporary social policy-making actions. The definition of “the deserving poor” is mainly constructed upon the specific ideas concerning deservedness, merit and social justice that are found to be predominant in each society. Yet, while there is a strong interest in policy design and social policy evaluation as domains of welfare studies, a combination of the two is still hardly discussed and used as a methodological/analytical tool for studying how poverty is really constructed in our society. Understanding how ideas and discourses affect the overall public representation of poverty, how institutional actors refer to ethical frameworks to design and legitimise welfare reforms and, eventually, how policy design ultimately shapes poverty, becomes even more relevant as austerity measures are put in force almost everywhere across Europe and while the debate over alleged categories of “welfare scroungers” is gaining momentum both in Europe and in North America. The discussion will address the construction of poverty as a topic and the need to adopt both qualitative (e.g. the analysis of ethical perspectives emerging from public debates, official declarations, national reports and so forth) and quantitative methods (that can assess the distribution and the character of poverty) for a proper understanding of its main dynamics. The combination of qualitative and quantitative methodologies can provide an innovative perspective for comprehending how different normative orientations underlying welfare reforms in times of austerity do have different distributive outcomes at domestic level (as Eurostat data indicate). Such perspective can break new ground not only for a better understanding of poverty and social exclusion from a sociological point of view, but also for the discussion of the relationships existing between discourses, ideas, policy tools and their implications for poverty in our society.

Please click here for a copy of the presentation.

Matthew Jones, Coventry University: What opportunities and challenges do mixed methods and methodologies of social policy research present?

Paradoxically, mixed methodologies whilst plausibly paramount for future research, endures terminology which is somewhat inchoate. As scholars such as Robson (2011: 161) and Bryman (2004) have suggested, it is superior to refer to mixed methods as multi-strategy designs, given the former underplays the significance of a multiple research strategies. Mixed methods are not simply limited to utilising a variety of research methodologies (such as qualitative and quantitative) but also require significant consideration of the numerous research strategies (such as fixed and flexible designs) to maximise methodological rigour. Despite these conceptual issues, mixed methodologies are beginning to gain momentum due to its potential to ascertain a greater breadth of understanding.

As a first year PhD research student examining trust and leadership, opportunities are rife for multi-strategy (mixed method) designs. A large volume of literature advocates one methodological paradigm over the other, with arguably opportunities overlooked for greater analysis. Attempting to overcoming the so-called ‘incompatibility thesis’ (see Robson 2011: 162, Guba 1987: 31) will not only benefit broader social policy research, but will also be advantageous to trust and leadership scholars alike. This will hopefully encourage discipline and interdisciplinary peers to seize the benefits of mixed methodologies (such as triangulation, completeness, and explanation, see Bryman 2006).

Although scholars of trust and leadership are plentiful, it is clear that several ambiguities remain. The author’s pending research seeks to add further clarity to some of this confusion by utilising the benefits of a multi-strategy (mixed method) sequential explanatory design. Rather than attempt to argue or justify the superiority of one technique over another, the methodological argument is to strive to realise the benefits of collective of approaches in order to gain a fuller understanding of the linkages between trust and leadership. For example, in order to address research questions such as “how leaders create and sustain trust?” will plausibly require a research design which can holistically draw from tangible and intangible (or less tangible) data sources. This presentation therefore acknowledges some challenges facing mixed methods but emphasises a suite of opportunities mixed methods create.

Please click here for a copy of the presentation.

1:15 – 1:45 Lunch

1:45 – 2:45 Mixed Methods and Methodologies of Social Policy Research

Eveleigh Buck-Matthews, Coventry University: Festivals: Spaces of Exception.

Complex spaces require a complex research design in order to identify the intricate nature of the processes and relationships that are taking place. It is only by utilising a mixed methods approach to these complexities that we are able to gain a deeper understanding of the participatory behaviour that people engage with in festival space and how to translate these into social policy. Drawing on my previous masters research and my current PhD research this presentation will highlight some of the complex challenges that exist when researching  the way that young people are represented in the U.K. Previous work highlighted youth interactions at a ground level through qualitative data collection, mainly interviews and focus groups. It also combined frame analysis to expand the scale of the investigation by examining documentary evidence when looking at the representation of young people at a national level. It concluded that a reframing of youth needs to take place to see the non-traditional ways and processes in which young people participate within society. Building on this my current work; Festivals: Spaces of Exception will use mixed methods to expand the arguments and broaden debates on non-traditional forms of participation that take place in festival space.

Festivals provide a cultural space which many young people feel invested in and have had a part in creating.  The positive attributes of these spaces, if utilised can be transferred beyond these spaces to discover ways young people can feel part of wider society, deepening current debates on youth disengagement within society. There is a great opportunity using mixed methods to gain a better understanding of the processes that are taking place in festival space and to utilise new forms of participation that people are engaging with. Proposed methods include qualitative; interviews, focus groups and observational research. Also quantitative mapping of the ways that people participate. I will also generate data from frame analysis of media representations of festival space. Seeking to see how these forms of participation could potentially be implemented in wider policies and organisational structures to build a more inclusive system.  I believe that refectivity becomes complicated when utilising positivist quantitative methods with post structural qualitative approaches, I can identify challenges surrounding the potential to contradict an earlier theoretical stand point. Although I think that the potential that mixed methods present for expanding the understanding of participation and opening up processes in festival space is worth the challenges that they create.

Claire Preston, Anglia Ruskin University: The usefulness of mixed methods in understanding online collective action in a social care policy setting

The spread of the internet over recent decades means that to understand contemporary engagement in policymaking, it is now necessary to understand its manifestation online. The online environment is a constantly evolving terrain but one which provides access to unprecedented amounts of ‘naturally occurring’ data, as opposed to data provoked by the researcher. This presents methodological opportunities as well as challenges, typified in debates about ‘big data’. The research presented here uses a mixed-methods design to address some of these challenges.

The context is a consultation over social care, which provoked a campaign to defend disability benefits. The campaign encouraged people to post comments on the 2009 Green Paper, Shaping the Future of Care Together. By the end of the consultation, there were nearly 3,000 comments on the paper’s executive summary. This presentation focuses on the analysis of these comments, which formed part of a wider doctoral research project. The research draws on a social psychology conceptual framework, envisaging that group-based feelings of identity, efficacy and injustice are drivers of collective action. The aim of the analysis was to establish whether and how the drivers of collective action were expressed in the comments. Inductive thematic coding of a sample of comments uncovered and explored expressions of the drivers. It also enabled the development of a detailed coding system which, applied manually, quantified markers of these expressions in the whole set of comments. With the use of simple algorithm, it was also possible to classify comments according to the pattern of four characteristics they displayed.

The research found that most commenters expressed collective identity and group injustice in their posts but that the incidence of expression of these sentiments varied according to certain factors. The focus in this presentation, however, is on the implications for methods. Sentiment analysis software has facilitated various recent big data studies (for example, Thelwall et al, in press, Thelwall et al, 2011). More work on specifying the markers of collective identity and injustice might enable expression of these sentiments, likewise, to be identified in large bodies of textual data. However, as my own research suggests and as Lev Manovich (2011) argues, a back-and-forth between the nuanced interpretation made possible by close qualitative analysis and the pattern recognition brought by quantitative analysis makes meaningful conclusions from such analysis more likely.

Please click here for a copy of the presentation.

2:45– 4:14 Panel Discussion “Policy & Impact”: Mark Carrigan (Sociological Imagination), Sierra Williams and Jane Tinkler (LSE Public Policy Group/LASE Impact Blog)

4:15 – 4:30 Coffee break

4:30 – 5:30 Impact: Whose agenda and what for?

Kerris Cooper, London School of Economics: Balancing Impact with Accuracy, Effort and Academic Incentives

This short discussion will draw on my experiences from two recent research projects: Reading the Riots, a collaborative project between the LSE and the Guardian newspaper and Does Money Affect Children’s Outcomes?, a JRF funded project at CASE, LSE. Teaming up with a national newspaper was extremely beneficial for reaching a wider audience, but this came at a cost in terms of retaining control over how the findings were portrayed and what readers took to be the message of the research. With the smaller research project at CASE we retained control of the research findings, but reached a very limited audience (so far) and in a research team of just two people the time and effort taken to disseminate the findings was a great cost and does not tend to be incentivised or rewarded in academia. I will briefly summarise what I have learnt from these experiences so far, before ending with some points for discussion: how do we make use of traditional media and new media whilst preserving accuracy in the representation of our research? What can we do about the conflict between the time and effort required to reach a wider audience and the incentives to focus time on more academic outputs?

Please click here for a copy of the presentation.

Lisa McCrindle and Denise Coster, NSPCC: ‘Research and evaluation in practice: the NSPCC experience’

5:30 Close