What works? Evaluation of the DOSH financial capability programme

What works? Evaluation of the DOSH financial capability programme A report by Brunel University London and A2Dominion October 2018 1 This report is authored by Brunel University London in consultation with A2Dominion. Brunel University London evaluation team: Dr Emma Wainwright Dr Ellen McHugh Professor Ken Gilhooly Dr Laura Hills A2Dominion DOSH management team: Dele Ryder Christopher Langston 2 CONTENTS Page no. 4 1. Executive Summary 2. Overview of the Project 3. Overview of the Evaluation Approach 12 4. Key Findings: Outcome/ Impact Evaluation 17 5. Key Findings: Process Evaluation 28 6. Key Findings: Economic Evaluation 36 7. Limitations of the Evaluation and Future Evaluation 38 8. Implications and Recommendations for Policy and Practice 40 9. Sharing and Learning Activity 43 10. References 45 11. Appendices 47 8 3 1. Executive Summary 1.1 Project Summary The DOSH (Debts, Overdrafts and Savings Help) programme run by A2Dominion has been operating in Ealing since 2013 with the aim of supporting residents with their finances through close one-to-one therapeutic mentoring. Through the Money Advice Service (MAS) What Works Fund (WWF), A2Dominion has expanded the areas of DOSH delivery to Oxford and Spelthorne and developed a new financial capability application called Ask David to work alongside the therapeutic mentoring. The DOSH programme specifically aims to increase residents’ confidence, improve their money management, and enhance their ability to take control of their finances. It is a bespoke programme with DOSH mentors tasked with engaging residents through a series of one-to-one meetings, and drawing on the range of voluntary and community services (VCS), to assess and improve the wellbeing, resilience, and money-management skills of residents struggling with their finances. Since its development, DOSH mentors and partner organisations have encouraged residents to use the Ask David app to support and enhance their financial capability. During the WWF period, nearly 900 residents have been engaged in DOSH programme-related activities regarding their finances. 1.2 Evaluation Approaches The evaluation of the DOSH programme is based on answering the following question: Is digital support, therapeutic mentoring or a combination of these elements more effective in improving the financial capability of A2Dominion residents? Covering the period of funding, three approaches to evaluating the project were taken: Outcome Evaluation: this ran between August 2017 and March 2018 and involved a matched before and after quantitative survey with DOSH programme intervention (n = 34) and control (n= 24) group participants, and matched before and after qualitative case study interviews (n=12) with DOSH participants. Process Evaluation: this took place over the course of the funding period and draws together a range of data sources to reflect on the delivery and success of the DOSH programme. Economic Evaluation: this sets out the costs of delivering the DOSH programme and, using the Housing Association Charitable Trust’s (HACT) social impact measurement, analyses the return on investment of the intervention. 1.3. Key Findings 1.3.1 Outcome evaluation findings Therapeutic mentoring: • Therapeutic mentoring plays a key role in improving the financial capability of A2Dominion residents. • Most residents are positive about the therapeutic and personalised one-to-one support from their DOSH mentors, though a minority report inconsistent and patchy support during the later stages of the programme. Digital support: • As rolled out during this evaluation, the Ask David financial app has limited use. 4 Therapeutic mentoring and digital support: • The qualitative data from the case study interviews suggest that the digital app could usefully add value as part of a portfolio of interventions, alongside the therapeutic mentoring, to assist residents with their overall financial capability. Improving financial capability: • Quantitative and qualitative findings reveal improvement in financial capability among participants in the DOSH programme. • Quantitative findings show that DOSH is effective in: increasing residents’ ability to keep up with their bills and credit commitments; increasing residents’ confidence in budgeting and managing their finances; increasing residents’ financial knowledge; and, increasing residents’ self-perception on overall wellbeing. • There are two main motivating factors behind residents’ involvement with DOSH – (i) a desire to improve their financial situation; and (ii) ‘get back on track’ with their everyday lives. • Additional outcomes for residents on the DOSH programme include: learning about new services; gaining knowledge about available benefits and changes; and, one-to-one help and support with CVs, job searches and training. 1.3.2 Process evaluation findings • • • • • • DOSH mentors and volunteers were appointed and trained to engage residents about their finances. 169 internal referrals were made to DOSH via the Tenancy Support Officers. 373 residents were engaged through Brief Interventions. 125 Advice Plus visitors were referred to DOSH. 131 residents engaged in DOSH one-to-one support through therapeutic mentoring and completed a full programme of support from mentors. The Ask David app recorded 1,419 hits, 174 active users and 80 live conversations. 1.3.3 Economic Evaluation Findings • • • The overall cost of the project is £276,500. Using HACT’s Social Value Tool and Wellbeing Valuation approach, the DOSH financial capability programme achieved a Social Return on Investment (SROI) of £2,281,000. For every £1 spent on intervention, the estimated social value of the outcomes produced is £35. 1.4 Methodological Limitations 1.4.1 Intervention group drop off There was a drop off in respondent numbers between phases 1 and 2 of the outcome evaluation questionnaire: control group drop off was just over half (52%), and intervention group drop off was three quarters (75%). The sample size is therefore lower than indicated in the evaluation plan. There are a number of reasons for drop off. These include: the level of additional support needs some clients had, with reference to their mental health and general well-being, and mobile contact numbers being changed. In addition, some initial respondents and DOSH participants once an issue or concern had been resolved did not feel it was necessary to talk with a DOSH mentor or continue to participate in the programme. 5 In the future, to mitigate this level of drop off, A2Dominion would invest in an outcomes and tracking role, similar to posts within the old Connexions services which tracked 16-19 year old NEETs (not in employment, training and education). Such a role would track, via phone, letter, and/ or email, resident status and cross reference MAS outcomes over specific time periods. 1.4.2 Age profile of participants While evaluation data shows that DOSH is reaching its target group of struggling residents, questionnaire respondents and case study participants in the outcome evaluation included only one ‘young adult’ aged 16-24. The specific data on this cohort of residents is therefore limited and further research is needed. 1.4.3 Late roll-out of the Ask David app Delay in the roll-out of the Ask David financial app meant a shorter time for mentors to encourage DOSH participants to consider and use it. This had an impact on the Phase 2 intervention group outcome questionnaire findings on engagement with, and use of, the app and did limit the ability of the evaluation to fully answer the research question. 1.4.4 Longer-term monitoring To address limitations, it is suggested further evaluative research be conducted over a longer period. Ongoing monitoring of the benefits of one-to-one therapeutic mentoring will enable further understanding of how residents are coping with their financial problems and effecting behavioural change. In addition, given the less discernible outcome findings on, and the delays with, Ask David, there is need to continue evaluating its use and value as a supplement to therapeutic mentoring in improving financial capability for different groups of residents. 1.4.5 Applicability and transferability Recognising and learning from these limitations, the findings from this project have the potential to be applied and transferred to other housing associations. The sharing and learning activities are geared towards showcasing best practice and highlighting the potential transferability of such a financial capability intervention programme. 1.5 Learning and Sharing Activity Findings from the DOSH programme evaluations have been shared widely, influencing organisational development within A2Dominion and knowledge sharing with external partners. A visual Dashboard, representing the progress of key work strands within the project and interim progress reports produced by Brunel University London were distributed to senior leadership teams and key stakeholders within A2Dominion. A key stakeholder group of staff and external partners met quarterly to report on progress and share learning. Deloittes Digital led on the production of customer insight and segmentation work which successfully supported bids for further funding and HACT (The Housing Association Charitable Trust) led on a social media engagement strategy with the housing sector, including regular posts and blogs. A2Dominion and Brunel University London are working collaboratively to implement a future dissemination plan which includes: • Sharing the final report with the Senior Leadership Team of A2Dominion to inform future decision making and identify what approaches work when designing and delivering intervention programmes to support residents with their financial capability and overall wellbeing. 6 • • • • • Working with HACT to produce a framework model to deliver therapeutic one-to-one mentoring interventions drawing upon best practice. Sharing the report across the g15 – London’s 15 largest housing associations. Recognising the challenges encountered when engaging with vulnerable and hard-to-reach participants and sharing this with the housing association community and other stakeholders. Working with HACT to promote the report and DOSH programme across social media including A2Dominion’s Facebook and Twitter accounts. Publishing journal articles contributing to the wider academic literature on financial capability in the UK. A2Dominion will lead on sharing the report internally and coordinating related internal activities. Brunel University London will publish academic journal articles contributing to the wider literature on financial capability in the UK and disseminate findings through academic housing networks. 7 2. Overview of Project 2.1 Introduction This chapter provides an overview of the DOSH programme run by A2Dominion and explains how it has been expanded through the MAS WWF. It provides details of the programmes different elements and the key question addressed across the funding period and through this report. The project’s Theory of Change is used to highlight key inputs, activities, outputs and outcomes. This is set within the wider context of DOSH. Finally, the chapter looks at the delivery of the project and its outcomes. 2.2 Aims and objectives of the project The DOSH programme is delivered by A2Dominion and has been operating in Ealing since 2013. It aims to increase residents’ confidence, improve their money management, and enhance their ability to take control of their finances. The WWF enabled the programme to be expanded to cover three geographical areas – the London Borough of Ealing, Spelthorne Borough and Oxford City. With this, DOSH mentors were appointed for each of the areas. In addition, A2Dominion commissioned Deloitte UK to develop a new financial capability application called Ask David to work alongside the therapeutic mentoring element of the programme. The programme’s two key elements are explained as follows: Therapeutic mentoring: while mentoring is a well-known concept and practice in organisations, therapeutic mentoring is less so. It is an approach to working with service users that has developed in the USA and is derived from behavioural psychology. It merges mentoring with therapeutic services, and has emerged from practices associated with health and clinical sciences. A2Dominion adopted the therapeutic mentoring approach so as to include the wellbeing aspects of support and recognising DOSH mentors’ role in teaching alternative strategies; providing anticipatory guidance; using role plays and behaviour referrals; and, enhancing and practising conflict resolution, problem solving and social skills in everyday situations. Financial capability app: the ASK David app was developed by Deloitte and was a key component of the WWF project. Deloitte was tasked with creating a product that had the potential to deliver meaningful long term outcomes for users, as well as complementing the ethos of the therapeutic mentoring and the overall delivery strategy of DOSH (Appendix 1). In relation to these two key elements, the WWF project was aimed at answering the following question: Is digital support, therapeutic mentoring or a combination of these elements more effective in improving the financial capability of A2Dominion residents? The DOSH target population is identified as having basic financial knowledge and needing help with setting and achieving financial goals for unplanned life events. The evaluation of DOSH therefore aimed to capture A2Dominion residents with different tenancies, and of differing age, gender, employment status and BME backgrounds. The evaluation targeted the following specific groups: • • • • Young adults 16-24 Struggling Young adults 18-24 Squeezed Working age people Squeezed Struggling adults or young people in financial difficulty. The Money Advice Service (MAS, 2015) conducted a large scale analysis to classify and prioritise the needs and circumstances of the 50 million adults in the UK, and using this data divided the adult 8 population into three main segments: ‘struggling’ = 11.6 million people; ‘squeezed’ = 12.7 million people and; ‘cushioned’ = 24.5 million (MAS, 2017). For this project, all participants in DOSH and its evaluation were identified as falling into the ‘struggling’ or ‘squeezed’ MAS categories, with a majority defined as struggling. The MAS definitions of squeezed and struggling are as follows: Squeezed – Working age consumers with significant financial commitments but relatively little provision for coping with income shocks. They are digitally savvy and have high media consumption but this is more for entertainment than financial information. Struggling – They struggle to keep up with bills and payments and to build any form of savings buffer. They are the least financially resilient and the most likely to be over-indebted (MAS, 2017). 2.3 Project outline Residents needing extra help and support with their finances are identified, engaged and referred to DOSH in a range of different ways including through Income Officers and Tenancy Sustainment Officers, referrals via Advice Plus agencies and via Brief Interventions. Income Officers and Tenancy Sustainment Officers (TSOs): Income Officers alert the TSOs when a resident is in rent arrears. The TSO then contacts residents and, if considered appropriate, they refer them to the DOSH programme. Brief Interventions: these are a concept developed from the primary health care sector and serve as a tool for providers to have short, instructive conversations with potential service users. NICE describes them in the following terms: “A brief intervention involves oral discussion, negotiation or encouragement, with or without written or other support or follow-up. It may also involve a referral for further interventions, directing people to other options, or more intensive support. Brief interventions can be delivered by anyone who is trained in the necessary skills and knowledge. These interventions are often carried out when the opportunity arises, typically taking no more than a few minutes for basic advice.” (NICE, 2014: 46) In the DOSH programme, brief interventions have been used to open discussions about money with residents. Advice Plus: this is a multi-agency drop in service managed by A2Dominion and has been running successfully at the Beethoven Centre, Westminster, for eight years. The WWF was used to appoint an Advice Plus Coordinator to establish a weekly Advice Plus session in Ealing, located at the Lido Centre. Operating a triage system, the Advice Plus Coordinator engages clients and signposts them to the DOSH programme and DOSH mentors. Once referred onto the DOSH programme residents are contacted by the relevant area mentor who then provides a series of one-to-one meetings aimed at addressing issues of wellbeing, resilience and money-management skills. These meetings normally start with budget planning and an assessment of overall wellbeing. At this point, other Voluntary and Community Services (VCS) may be drawn on to further support the DOSH participant. In addition to this therapeutic mentoring, and if and when appropriate, residents are encouraged to use the Ask David app to support and enhance their financial management. Over the course of the DOSH programme, residents, with the help of the DOSH mentors, begin to identify the areas in which they need extra help and support to ‘get back on track’ and improve their overall financial capability and wellbeing. 9 Figure 2.1 DOSH Theory of Change The purpose of DOSH, as expanded through the MAS WWF, is to increase residents’ confidence, improve their money management, enhance their ability to take control of their finances, and improve and sustain budgeting and saving skills so they are better able to cope with planned or unplanned financial difficulties. The project sought to achieve these outcomes through a series of evaluation activities, as detailed in chapter 3. A Theory of Change was produced outlining the key inputs, activities, outputs, outcomes and impacts to ensure the DOSH financial capability programme achieved its goals (figure 2.1). The only change to the Theory of Change was the exclusion of Westminster in the DOSH WWF evaluation, as the capacity and additional resources to fully support a Westminster programme were not factored into the original grant application. 2.4 Context The UK’s Financial Capability Strategy (fincap.org.uk, 2015) focuses on individuals “taking control of their finances” while recognising that people have different needs at different stages in their lives. It emphasises the need for “improving people’s ability to manage money well, both day to day and through significant life events, and their ability to handle periods of financial difficulty” (2015: 6). This is particularly pertinent to the social housing sector as research has shown that social housing tenants are more susceptible to lower levels of financial capability (Atkinson et al, 2006) than others. Large proportions of social housing tenants are in no, or part time and precarious forms of, employment and therefore rely on welfare provision. Changes to this welfare provision in terms of mode of receipt and amount given are significant for both social housing agencies and tenants. These welfare reforms are an important opportunity to renew efforts to address the high levels of financial exclusion among social housing tenants: “welfare reforms should be viewed as an opportunity for landlords to review their current practice and put in place policies and practices that will deliver financial inclusion, as this will help to mitigate the risk of rent arrears and evictions”. (Williams, n.d.: 3) 10 Over the past decade, the social housing sector and individual agencies have given increasing attention to the financial capability of tenants (NHF and Toynbee Hall, 2008) with a proliferation of financial skills/ education programmes implemented at local level (e.g. Collard et al, 2012). Such programmes are important as social housing agencies benefit from lower rent arrears and cost savings as a result of fewer failed tenancies (Chartered Institute of Housing, 2011; NHF, 2010). Directly related to the roll-out of Universal Credit, recent research evidences that many tenants are experiencing difficulties with direct payment and landlords' arrears are rising markedly (Hickman et al, 2017; Tanner, 2018). The DOSH programme provides a potentially important addition to the range of financial capability programmes run by social housing agencies as it brings together both one-to-one personalised support with app technology. As the UK’s Financial Capability Strategy highlights, “where advances in technology can be used assist people to improve their financial capability” there is a need to ensure they are used (fincap.org.uk, 2015: 4). The DOSH programme, with its Ask David app, provides a new case study through which to explore the role that such technology can play in increasing financial capability, alongside more traditional approaches. The extension of the DOSH programme through the WWF follows on from four evaluations conducted by and for A2Dominion. The first was a process mapping carried out by Price Waterhouse Coopers in 2016 as part of A2Dominion’s business transformation programme ‘Fit for the Future’. This WWF project and its evaluation support this wider transformation programme with the development of tenancy support and a pilot well-being team. The second was the DOSH plus survey with 90 participants which looked at the different types of services residents used; the types of problems they were experiencing and seeking advice on; their internet access and usage; their confidence in using the internet; and the different tasks they carry out online. This survey was carried out in the summer of 2016 to inform A2Dominion research bids as part of a process to understand how they can better support their residents with their finances and overall wellbeing. The third was a financial inclusion health check conducted by Toynbee Hall. This highlighted further improvement A2Dominion could make to consolidate and improve its work, including working with frontline staff to develop their skills in identifying poor financial health (Toynbee Hall, 2016). The fourth, an ‘Evaluation Review for the DOSH Project’, again conducted by Toynbee Hall (2017), made three keys recommendations: implementing a thorough monitoring and evaluation plan for DOSH; broadening the DOSH offer to everyone in the community and expanding the geographical scope; and, considering longer term programme sustainability including opportunities for volunteers and digital engagement. Through its current form and implementation, A2Dominion are heeding the findings of this work through its DOSH programme. 2.5 Project Delivery The project was delivered as originally intended. A short delay in the roll-out of the Ask David app resulted in a shorter time frame in which residents were able to use the app before phase 2 of the data collection. There was a three-month delay in recruitment of the Advice Plus Coordinator and various staff changes including the project manager leaving, which affected delivery at certain stages. 11 3. Overview of the evaluation approach 3.1 Introduction This chapter introduces the research question for the evaluation of DOSH and highlights the links to MAS WWF outcomes and UK Financial Capability Strategy priorities. It details the different elements of the evaluation and their related methods, data collection and activities, and reflects on changes from the original evaluation design. The role of the DOSH project evaluation is underlined with reference to existing knowledge, highlighting how this evaluation complements and adds to this. 3.2 Research Questions The evaluation of DOSH focuses on improving the financial capability of A2Dominion residents. The main research question is as follows: Is digital support, therapeutic mentoring, or a combination of these elements, effective in improving the financial capability of A2Dominion residents? Building evidence linked to the social housing context, this research question aligns with the MAS policy question relating to young adults, work-age people and those facing financial difficulties. In particular, the project is interested in: • • • How can we help 16-to-24 year-olds, who have left school and are transitioning to independent living, to prepare for and make better financial decisions? How can we help working age adults to improve their financial capability, develop budgeting and tracking habits, build up a savings buffer to withstand financial shocks and/or set financial goals for key life events? How can we identify and help people who are at risk of falling into problem debt? The evaluation therefore focuses on the following specific target groups: • • • • Working age adults who are struggling; Working age adults who are squeezed; Young adults (16-24) who are struggling and/or squeezed, and: Adults, any age, who are facing financial difficulties. This main research question was designed to link to the MAS Outcomes framework and the components of behaviour, wellbeing, mindset, ability and connection (MAS, 2017). The following elements have been critical in the evaluation of DOSH: Behaviour – managing money well day to day Wellbeing – keeping up with bills; emotional wellbeing Mindset – financial confidence Ability – budgeting and knowledge about products Connection – increasing digital and internet engagement The DOSH programme and its evaluation link directly to the UK’s Financial Strategy priorities (fincap.org.uk, 2015) for young and working-age adults including: improving financial knowledge; planning and management; and, identifying best support approaches. It also aims to provide rigorous evidence and evaluation as stressed in the UK’s Financial Capability Strategy (2015). 3.3 Type of Evaluation This research question has been addressed through three different, but complementary, evaluations: 12 3.3.1 Outcome/ impact evaluation The outcome/ impact evaluation is aimed at evidencing and understanding the impact of the DOSH programme on participants. It measures the outcomes of the DOSH programme linked to the above elements of Behaviour, Wellbeing, Mindset, Ability and Connection. A2Dominion commissioned Brunel University London to undertake the design, implementation, and analysis of this evaluation. With a focus on the main evaluation question above, the Brunel evaluation was framed around the following: • • • • Identifying and evidencing the factors that lead to a positive experience of DOSH and improved financial capability; Evaluating the experiences and perceived effectiveness of the relationship between DOSH mentors and participants; Exploring the role of mentoring and the effectiveness of the therapeutic relationship; Evaluating the experiences of using, and the effectiveness of, the new app in improving participants’ financial capability. The methodology and methods used in the outcome evaluation are detailed in section 3.5.1 and the findings are presented and analysed in chapter 4. 3.3.2 Process evaluation The process evaluation is aimed at understanding the effectiveness of the DOSH programme from a delivery perspective. Importantly, this enables a reflection on the success of the programme as well as consideration of what worked less well in terms of programme delivery. This allows a better understanding of how the programme could be changed or improved as it moves forward. The process evaluation draws on a wide set of data collected by A2Dominion and DOSH partner organisations. The approach taken in the process evaluation is detailed in section 3.5.2 and the findings are presented and analysed in chapter 5. 3.3.3 Economic evaluation The economic evaluation is aimed at fully understanding the costs and benefits of the DOSH programme. The HACT Social Value Tool and Wellbeing Valuation approach is used to understand potential longerterm economic impact of participation in DOSH for individuals, A2Dominion and wider society. The approach taken in the economic evaluation is detailed in section 3.5.3 and the findings are presented and analysed in chapter 6. 3.4 Complementing/adding to the existing evidence base As highlighted in section 2.4, the UK’s Financial Capability Strategy has spearheaded calls for improving people’s ability to manage their money well. Part of the strategy is focused on ensuring robust evaluation of interventions targeted at specific groups (fincap.org.uk, 2015). The evaluation of the DOSH programme, which is targeted at social housing residents, is an important addition to existing financial capability knowledge focusing on specific groups e.g. on youth (Johnson, 2007), immigrants (Barcellos et al, 2016), retirees (Lloyd and Lord, 2015). Previous qualitative research exploring residents’ use of DOSH highlights its value among those experiencing financial and other difficulties (Wainwright and Marandet, 2017). As emphasised in the UK’s Financial Capability Strategy (fincap.org.uk, 2015) there is a need to look at those groups who have been affected by welfare reforms and those receiving Universal Credit, and many of the participants in the DOSH programme fall into these categories. The evaluation of DOSH also enables a view across the programme with an appraisal of the therapeutic mentoring and financial app elements. In addition, given its now well-established link with financial capability (Taylor et al, 2009; 2011) a focus on the wellbeing of residents is critical and this evaluation looks at a series of wellbeing indicators. 3.5 Methodology 13 3.5.1 Outcome/ impact evaluation The outcome/ impact evaluation was conducted between August 2017 and March 2018 across the three DOSH areas. In close collaboration with A2Dominion, Brunel University London devised a mixed methods evaluation based on survey questionnaires and case-study interviews. Outcome evaluation data was collected through two key stages: Stage 1 consisted of a matched two-phase questionnaire with the DOSH intervention group. Phase 1 was conducted between August and November 2017 and at the start of residents’ engagement with DOSH and phase 2 between December 2017 and March 2018 towards the end of the DOSH process. The questionnaire survey looked at employment status, sources of income, levels of worry about current financial situation, keeping up with financial commitments, confidence in managing finances, and knowledge of different financial budgeting tools (Appendix 2). Questions were designed using MAS’s Financial Capability Outcomes Framework for Adults which contains a comprehensive selection of outcomes, indicators and questions for use in surveys to measure changes in people’s financial capability. The phase 2 questionnaire omitted key demographic/ profile questions and added in measures on the effectiveness of the DOSH programme, its mentors, and the Ask David financial app (Appendix 3). Phase 1 questionnaires were completed with/ by DOSH mentors. Given the questions about experiences of DOSH mentors, phase 2 questionnaires were completed with the support of other A2Dominion staff. Included in stage 1 was a matched two-phase questionnaire with a control group. This was used to control against changes in the intervention group and substantially enhances the quality of the data collected and analysis made on key variables. The control group followed a similar profile to the intervention group and was selected from residents identified as struggling with their finances. Brunel University London collected all control group data. The control and intervention group questionnaires were run during the same period. Matched completions for questionnaire phases 1 and 2 are shown in table 1. Though smaller than planned, the size of groups has enabled statistically significant results to emerge. Table 3.1: Questionnaire completions DOSH questionnaire Phases 1 and 2 Intervention Group 32 Control Group 24 Of the 32 intervention group participants all were defined as ‘squeezed’ or ‘struggling’ (MAS, 2017). Stage 2 complemented the predominantly quantitative data collected in the questionnaire through a two-phase case study interview with DOSH participants. Opportunistic sampling was used to recruit participants from the DOSH intervention group. All case study participants completed the questionnaire survey. A case study profile was developed for each participant giving basic demographic and biographical details (Appendix 4). These interviews were aimed at gaining greater insight into the experiences of those involved in the DOSH programme. Interviews focused on what led the residents to become aware of DOSH, and their experiences and outcomes of involvement (Appendix 5). Matched completions for phases 1 and 2 are shown in table 2. Twelve participants engaged in both phase 1 and 2 interviews. Table 3.2: Case-study interview completions Case study interviews Phase 1 Completions 14 Phase 2 12 14 Stages 1 and 2 of this evaluation were used to capture outcomes related to DOSH participation and the MAS WWF components. Analysis of data sought to triangulate between the quantitative and qualitative data sets to ensure better understanding of, and enhance confidence in, key findings. Quantitative questionnaire data was analysed using SPSS. This analysis aims to see if the change between phases 1 and 2 is greater for the intervention group than for the control group. A two-way analysis of variance, where one condition is within-participants (Time of Test) and one is between participants (Group), and examines key variables, was used. This was followed by analyses of composite scales for Financial Knowledge and SelfPerceptions. Qualitative case study interview and questionnaire data was analysed using thematic analysis. Thematic analysis focuses on identifying, describing, and then analysing common themes across the data set using a coding approach. Both sets of data and analysis are used to answer the main evaluation question. In this report, individuals who took part in the questionnaire survey are referred to as ‘respondents’, while individuals who took part in the case studies interviews are referred to as ‘participants’. As highlighted in section 3.5.1, the 14 case study participants are also among the 32 intervention group questionnaire respondents. 3.5.2 Process evaluation The process evaluation focuses on project delivery and includes monitoring and evaluating the number of residents going through DOSH, seeking one-to-one support and/ or accessing the financial app, as well as drawing on experiences and perceptions of DOSH implementation from mentors and partner organisations. This evaluation draws on a wide range of data sources to explore the different elements of DOSH. Table 3.3 provides an overview of the data sources used in relation to the different DOSH elements: Table 3.3: Data sources on delivery of DOSH components Overall Dosh delivery Therapeutic mentoring Ask David App DOSH dashboard ✓ ✓ ✓ DOSH mentor interviews Director of Housing interview App scoping interviews and personas Partner evaluations ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Google analytics ✓ Common Purpose/ Queen Mary programme ✓ The DOSH dashboard, implemented and managed by A2Dominion over the course of the funding period, has tracked and monitored residents participating in the different elements of the DOSH programme. 15 Brunel University London conducted individual interviews with three DOSH mentors who are responsible for the day-to-day running and implementation of the DOSH programme in Ealing, Oxford and Spelthorne. These interviews were conducted between May and August 2017 and are aimed at providing context to the implementation, role and success of the DOSH programme and its different elements (Appendix 6). In the latter stages of the funding period, Brunel University London conducted an interview with A2Dominion’s Director of Housing to explore DOSH delivery, potential and success from an organisational perspective. This provides further important context for the DOSH programme and the understanding of its different elements for improving the financial capability of residents and improving outcomes for A2Dominion (Appendix 7). Prior to the development of the app (January to March 2017), Deloitte UK conducted 40 app scoping interviews with residents to explore and understand how they manage their day-to-day finances; their monthly expenses and savings; the different types of technology they own and use on a daily basis; their main sources of income; how they feel about their current financial situation; and if they seek advice or support from external agencies about managing their finances. The findings from these 40 interviews were used to inform and create 4 personas (Appendix 8). Personas are ‘fictitious representations of target users’ created using data collected from interviews with potential users (Pruitt and Adlin, 2006). As the Ask David app was trialled, A2Dominion undertook a series of evaluations with partner organisations who are involved in its roll-out. Evaluations were collected via emails to/ from A2Dominion staff and focus on potential usability. Google analytics is a service offered by Google to track and report website traffic. A2Dominion use this to collect data on engagement with and use of the Ask David app. The google analytics data used in this evaluation covers the WWF period from August 2017. Common Purpose UK contacted A2Dominion via HACT to offer co-curricular opportunities for students at Queen Mary, University of London. The challenge, devised by A2Dominion for students, was to find an answer to the following: ‘Is Digital support, through apps like Ask David, the most effective way to help young adults build confidence to manage their money’. Seven students were involved in this and their research included engaging young residents in conversations about how they use technology (Appendix 9). Linking directly to the key evaluation question, the data collected from these different sources is presented, analysed and discussed in chapter 5. 3.5.3 Economic evaluation To provide an economic evaluation of DOSH, A2Dominion draw on the HACT social value calculator. HACT has created a large bank of methodologically consistent and robust social value indicators to enable as assessment of social impact and evidence of value for money. HACT are the sector leaders in measuring the social impact of Housing Association community investment programmes. The social value indicator included relevant financial capability outcomes for the programme. 3.6 Changes to Methodology There were a small number of changes made to the original evaluation methodology: • Timescale of outcome evaluation: due to the delay in the roll-out of Ask David, the outcome evaluation commenced in August 2017 and was completed by March 2018. 16 • Further data sources: given the delay in the roll-out of the Ask David app and its anticipated impact on the outcome evaluation, A2Dominion sought to better understand the potential for engaging residents with the app and, through Common Purpose UK and students from Queen Mary, University of London, ran a small-scale scoping study to extend understanding of the use of Ask David among young adult residents. 17 4. Key Findings: Outcome/ Impact Evaluation 4.1 Introduction This chapter reports on the outcome/ impact evaluation data and findings. It starts by providing a profile of the residents involved in the DOSH evaluation before providing context on residents’ access and involvement with the programme. Findings are presented and analysed in relation to main research question: Is digital support, therapeutic mentoring, or a combination of these elements, effective in improving the financial capability of A2Dominion residents? In keeping with the MAS Outcomes, findings are organised into the following categories: (i) behaviours, (ii) mindset, (iii) ability, (iv) wellbeing and (v) connection. In each section that follows, the quantitative findings are presented to highlight change in relation to the key research question and the case study interviews used to provide further detail and analysis. Although the sample size is small, analysis of the matched two-phase questionnaire with 32 intervention group participants shows a strong statistically significant improvement in financial capability among participants of the DOSH programme – a key Theory of Change outcome for the evaluation programme. Two-phase case study interviews with DOSH participants reinforce these findings. Notably, analysis of the questionnaire and case study interviews show that therapeutic mentoring alone plays an important role in improving the financial capability of residents. There is little evidence that the Ask David financial app as rolled out in this programme adds value. However, there is some evidence from the case study interviews that the app could add value as part of a portfolio of interventions to assist residents with their overall financial capability and wellbeing. 4.2 Matched Questionnaire Respondent Profile As demonstrated in table 4.1, all survey respondents in control and intervention groups come from the three geographical areas of Ealing, Oxford and Spelthorne, with the largest number from Spelthorne. Table 4.1: Questionnaire completions by geographic area DOSH questionnaire Intervention Group TOTAL Phases 1 and 2 32 Ealing 9 Oxford 8 Spelthorne 15 Control Group 24 8 9 7 Table 4.2 shows the gender for all survey respondents in the intervention group and demonstrates a majority of female participants. Table 4.2: Intervention group by gender DOSH questionnaire Female Male Intervention Group 20 12 As demonstrated in table 4.3, the majority of questionnaire respondents are aged between 25 and 54 with a mean age of 45. Only one respondent falls into the ‘young adult’ category. Table 4.3: Intervention group by age DOSH questionnaire 18-25 years old Intervention Group 1 18 25-34 years old 35-44 years old 45-54 years old 55-64 years old 65-74 years old 75 years old 8 8 8 6 1 0 4.3 Case Study Participant Profile As demonstrated in table 4.4, participants from the matched two-phase case study interviews came from the pool of questionnaire completions and are therefore also from the same three geographical areas. Detailed bibliographic information on the case study DOSH participants can be found in appendix 4. Table 4.4: Case study interviews by geographical area Case Study Interviews Participants Ealing 4 Oxford 5 Spelthorne 5 Table 4.5 shows the gender for all case study interview participants and demonstrates a small majority of female participants. Table 4.5: Case study interviews by gender Case Study Interviews Female Male Participants 8 6 As demonstrated in table 4.6, the majority of case study participants are aged 35-64 with a mean age of 48.4. No participants fell into the ‘young adult’ category. Table 4.6: Case study interviews by age Case Study Interviews 18-25 years old 25-34 years old 35-44 years old 45-54 years old 55-64 years old 65-74 years old 75 years old Participants 0 1 5 4 4 0 0 4.4 DOSH Access and Involvement All survey respondents and case study participants were identified by A2Dominion as falling under the ‘struggling’ and ‘squeezed’ categories (MAS, 2017). These individuals were referred to the DOSH programme as being in need of support. Case study interviews show that while participation in DOSH is entirely voluntary, most participants found themselves on the DOSH programme after referrals from A2Dominion Income Officers and TSOs due to rent arrears. “Basically, at the time, A2Dominion was unfortunately seeing me go into rent arrears. It was not a considerable amount of money but it was enough to make their concern and they asked me would I like to be put in touch with a project which is called DOSH”. (Case Study C) 19 In addition, findings from the case study interviews identify two main motivating factors behind residents’ involvement with DOSH: • a desire to improve their financial situation • ‘get back on track’ with their everyday lives. 4.4.1 Improving financial situation More than two thirds of case study participants spoke of an overwhelming desire to improve their financial situation, clear their rent arrears and any outstanding debts, and avoid eviction. They also want to develop the necessary skills and knowledge to manage their finances and budget day-to-day independently and with increased confidence. “…there are certain steps they – we – can take to get me back on track and try and clear this debt and get my rent up and running because I’ve had a letter saying they are seeking possession of the flat and I think it was that that really kicked me into gear”. (Case Study L) 4.4.2 ‘Getting back on track’ Participants hope engagement with the DOSH programme will help them get ‘back on track’ with everyday life. A number spoke about experiencing increasing levels of stress and anxiety about their mounting debts and precarious financial situation before their involvement in the DOSH programme. Levels of stress, worry and anxiety are often compounded by cuts and changes to benefits which affect everyday activities and needs: “…cause it got to a point where I just wanted to run away. But I know it doesn’t solve the problem, but you know, it’s just getting on…I’m welling up at the moment, it’s just getting to me at the moment, you know, but I’m getting help with them”. (Case Study I) “I can’t afford to go shopping”. (Case Study A) 4.5 Key Research Outcomes This section reports on findings of key survey questions linked to financial knowledge, worry, and confidence, keeping up with bills and credit commitments and self-perception. Qualitative data is provided to triangulate our findings and offer richer insight. These are linked to Theory of Change outcomes identified in the Evaluation Plan which include: • • • • Improved financial capability Increase in confidence in managing finances and managing better day-to-day Improvement in emotional and mental wellbeing Increased access to advice services and residents actively seeking advice. 4.5.1 Behaviours: managing money day to day, improving financial situation There is a statistically significant beneficial effect of DOSH on residents’ keeping up with their bills and credit commitments. DOSH is effective in increasing residents’ ability to keep up with their bills and credit commitments. 20 Question 6: Which ONE of the following statements best describes how well you are keeping up with your bills and credit commitments at the moment? (Appendix 3) The graph indicates that the intervention group have a larger change in their Keeping Up scores in the expected direction (=39%) than did the control group (+3%). Statistically, the size of the interaction effect is partial η2 = .08, which is a medium to large sized effect. It is significant, F(1,54) = 4.58, p=.02. Control Group: N=24 Intervention Group: N=32 “…basically I’m up and running now, I’m back in full time employment, I’m paying the arrears of”. (Case Study C) “… And now I’ve managed to…now I’ve managed to help out, like my rent’s up to date, you know, and everything…everything’s going good”. (Case Study M). 4.5.2 Mindset: confidence, improved financial outlook/ capability There is a statistically significant beneficial effect of DOSH on residents’ confidence. DOSH is effective in increasing residents’ confidence in budgeting and managing their finances. Question 8: How confident do you feel in budgeting and managing your finances? (1 = Very confident; Fairly confident; Not very confident; 4 = Not at all confident) The graph indicates that the intervention group have a larger change in their Confidence scores in the expected direction (-20%) than did the control group (-5%). Statistically, the size of the interaction group effect is partial η2 = .05, which is a medium sized effect. It is significant F(1,51)= 2.68, p=.05. Control Group: N=24 Intervention Group: N=32 A major benefit of participation in DOSH is residents feeling that they can apply the practices and behaviours they have learnt to their individual circumstances: “Mentor 2 came around and like obviously helped me out and he advised…that’s how I got on the course. And now I’ve managed to…now I’ve managed to help out, like my rent’s up to date, you know, and everything…everything’s going good”. (Case Study M). A majority of residents feel the programme and the support of their mentors has brought structure to the way they manage their finances, how they prioritise and keep on top of their outgoings: 21 “That’s the idea of why I got them involved. I want them to give me a plan, put a plan in place what I can follow and…have a decent budget and still have enough money to last me until I get paid again”. (Case Study F) Residents feel it has taught them to identify and understand their most pressing issues, to learn how to deal with problems, and to apply the budgeting skills they have learnt from their engagement on DOSH to other scenarios. “…budgeting, to help me how to budget and if I’m falling behind with anything, she [Mentor 3] will ring up on my behalf, in my presence, and she will tell me what she’s going to do with this person, and what’s going to be happening. And so sometimes I might not understand, but what she does, she explains to me and she explains in a way that I understand”. (Case Study A) “I became more confident…I actually did begin to feel more confident, and I did things to budget more, so it [DOSH programme] did help…it was more of a help sort of talking really [to Mentors]”. (Case Study H) 4.5.3 Ability: financial knowledge There is a highly statistically significant beneficial effect of DOSH on residents’ self-rated Financial Knowledge. DOSH is effective in increasing financial knowledge. Question 10: Which statement best describes how knowledgeable you feel about the following items: Bank accounts; Credit cards Mortgages; Savings accounts; Credit Unions; Investments; Pensions; Insurance; Different types of financial fraud; Different types of loans and credit (1= I know nothing about this; 2= My knowledge is very limited; 3= I have basic knowledge, but do not understand the details; 4= I have reasonable knowledge; 5= I have good knowledge). The graph indicates that the intervention group have a far larger change in their Financial Knowledge total scores in the expected direction (34%) than did the control group (-2%). Statistically, the size of the interaction effect is partial η2 = .14, which is a large medium sized effect. It is highly significant F(1.50)= 7.89, p= .004. Control Group: N=24 Intervention Group: N=32 4.5.4 Wellbeing: worry, self-perception, increased wellbeing, confidence and optimism There is a statistically significant beneficial effect of DOSH on residents’ worry. DOSH is effective in reducing residents’ levels of worry about their current financial situation. There is statistically significant beneficial effect on residents’ self-rated perceptions of overall wellbeing (optimism, feeling useful, feeling relaxed, dealing with problems, thinking clearly and feeling close to people). DOSH is effective in increasing residents’ self-perception of overall wellbeing. 22 Question 5: To what extent, if at all, do you worry about your current financial situation? The graph indicates that the intervention group have a larger change in their Worry scores in the expected direction (+33%) than did the control group (10%). The statistical size of the interaction effect is partial η2 = .12, which is a large medium sized effect and is significant, F(1,50)=6.79, p=.01. Control Group: N=24 Intervention Group: N=32 Question 15: To what extent do each of the following apply to you: Feeling optimistic about the future; Feeling useful; Feeling relaxed; Dealing with problems well; Thinking clearly; Feeling close to other people; Able to make up your own mind (1= None of the time; 2= Rarely; 3= Some of the time; 4= Often; 5= All of the time) The graph indicates that the intervention group have a larger change in their SelfPerception Total scores in the expected direction (15%) than did the control group (-5%). The size if the interaction effect is partial η2 = .16, which is a large sized effect and is significant F(1,48)= 9.34, p= .002. Control Group: N=24 Intervention Group: N=32 More than half of the case study residents spoke about how their levels of stress, anxiety and worry about their financial situation have reduced since they started on the DOSH programme: “Very good [DOSH programme] and actually very encouraging, basically telling me not to worry, it’ll get sorted, and he’s [Mentor 2] going to help me. So yeah, it’s all positive”. (Case Study E) “And I’ve stopped worrying because of [Mentor 2], he explained things properly to me, I’m not overly concerned about trying to keep the roof over mine and the kids head at the moment”. (Case Study D) “It’s not as bad as it was. I think I can get on with my life and – you know – discuss my problems…”. (Case Study F) Residents feel that their confidence has increased since they had met with their DOSH mentors and gone through their finances and set up a budget plan: “cause when he [Mentor 2] came to me, I was having a really bad year, so I think it’s actually given me a little bit of a boost”. (Case Study M) 23 The support of mentors and having someone ‘on call’ to offer a sympathetic ear in a non-judgemental manner has given residents increased confidence in themselves and the future: “Oh, they gave me back all the confidence that I need, and I’m really grateful to them…”. (Case Study B) Residents also spoke about finding the ‘courage’ to deal with their finances which they didn’t have before the DOSH programme: “Before I wouldn’t have done it. I wouldn’t have had the courage to do it”. (Case Study A) Having ‘control’ over their finances, health and wellbeing has positive effects for residents and their extended family: “[It’s] about how far I’ve come…and just everything has changed…how happy I am, how confident I am and how I can just take control of things by myself…even my children notice it now”. (Case Study D) From the case study interviews, a majority of residents who live with their children report that the DOSH programme has had a positive impact on their children, their quality of life, and standard of living: “It’s made a real big impact in our life”. (Case Study D) The availability of one-to-one support is also crucial for increasing confidence, engaging successfully with DOSH mentors, and being able to manage their own finances: “The benefits of using the service with DOSH regarding myself as personally that there is someone to help you and therefore it gives you confidence and a life. You don’t have to fall into debt if you don’t want to because there is someone there to help you and you feel much better once you come to an agreement – like I did and therefore it takes a lot of pressure off the head and start thinking about all this that and the other – you are going to end up there, you are going to end up there. It’s rubbish. They are there to help you and they can help you”. (Case Study C) “I find [day-to-day life] easier. Before I wouldn’t go [anywhere] without my children. And now I am quite happy, I go to work, go find a new job. I go and do anything I need to by myself. But I don’t have an issue with it at all no more”. (Case Study D) Over two thirds of the case study participants report feeling more confident about the future and their ability to manage their finances and this is mainly due to the positive experience of their engagement with their mentors and the DOSH programme: “I’m hundred percent now back to work and I got a big weight off my mind – not knowing full well – oh, you have got to pay this – you have got to pay that. There is no such thing. They help you. They discuss with you. They are there to help you”. (Case Study C) They also express hope for the future which involves returning to work and being ‘successful’ and ‘busy’ with a life free from debt, worry and stress: “Well, just getting it all sorted. So that I can try and get on, get better and hopefully get back to work basically. Just try and survive with my kids, and not be struggling basically”. (Case Study E) 24 “I need to now think about me and try and better myself for my children as well, really….I want to get…I want to get back in work, I want to be working for myself…that’s how I see myself being successful, being busy, being out there”. (Case Study M) 4.5.5 Connection: signposting of other services and benefits, job applications and benefits One other outcome of residents’ engagement on the DOSH programme is learning about new services available through A2Dominion and partner organisations. Gaining knowledge about available benefits and changes is vitally important for residents who often do not possess the necessary skills, or lack knowledge about the different types of support and benefits available to them. The one-to-one support from the DOSH mentors is critical in identifying residents who may need extra support and access to other services: “I am quite stressed and depressed, and so I…you know, the relaxation courses, I’ve been to one of, it’s quite good…”. (Case Study H) After their involvement in DOSH, 74% (n= 23) of phase 2 questionnaire respondents state that they would be very likely to seek advice from an external advice organisation to deal with money worries. It is important to recognise that residents who have complex needs may need more comprehensive help and support from external agencies and service/ agency providers. This intensive and targeted support in most cases falls outside the scope and remit of DOSH mentors; however, this does not diminish their role or the work, and the support and level of expertise they provide to residents in general. It is the help, support and guidance from DOSH mentors which encourages and empowers residents to seek, access and accept external advice and support. Digital connection is discussed in 4.6.2. 4.5.6 Other outcomes Other important outcomes of residents’ engagement with DOSH include one-to-one help and support with CVs, job searches and training. In some cases, residents returning to work after long-term sickness are given references from their mentors: “I speak to him [Mentor 2] quite regular as well because I have just applied for a new job so he’s been my reference for that”. (Case Study D) One resident spoke about how her DBS check had been paid for by A2Dominion and this had allowed her to find a job: “…and I have my DBS, it’s done already…they pay [for it]…I can tell you, without the help they’ve given me, I don’t think I would have got a job”. (Case Study B) The DOSH programme offers very practical support, as well as advice and guidance. 4.6 Experiences of DOSH 4.6.1 Therapeutic mentoring 62.6% (n= 20) of survey respondents report that DOSH has been effective in helping them identify their needs and 68.8% (n = 22) report that DOSH had given support in the areas most needed. “Received good support [from DOSH programme]”. (Questionnaire) “Ongoing help and someone there to assist me”. (Questionnaire) “The support [from the DOSH programme] has been fantastic”. (Case Study C) The role of the DOSH mentors is critical in providing support and 71.9% (n = 23) of survey respondents report that their DOSH mentor really supported them. 65.7% (n = 21) report that their DOSH mentor 25 encouraged them to engage with the programme and work towards improving their financial skills and knowledge. 68.8% (n = 22) survey respondents feel their DOSH mentor gives them helpful advice in terms of managing their day-to-day financial budgeting and getting back on track. A majority of case study participants are very positive about the personalised one-to-one support from their DOSH mentors and respond positively to their approachable and friendly nature. This one-to-one contact with and tailored support from the DOSH mentors is critical in developing a meaningful relationship, establishing trust, and engaging with residents: “Knowing that they are still there to fall back on and still supporting you”. (Case Study D) “The DOSH Mentor assisted with my Benefit Appeal for ESA and Housing Benefit. The mentor represented me at a tribunal and got my ESA reinstated after 10 months of non-payment. He then got me a back date for Housing Benefit of £1773.00 which took me out of rent arrears and reinstated my Housing Benefit claim”. (Questionnaire) Case study participants value the support and guidance they receive from their contact with their DOSH mentor and other A2Dominion staff, and this is key to increasing residents’ wellbeing and ability to manage their finances independently and with increased confidence. There are however a number of participants who felt dissatisfied with their mentoring in the later stages of their involvement with DOSH: “I’ll tell you now she [Mentor 3] is really difficult to get hold of, and I’ve certainly been trying to call her, and she’s never at her desk, on the Friday she’s not in. And she never returns my phone calls now, it’s kind of like you know, I kind of feel differently now to what I said about her before…she knows that I need help...just basically she’s left me with it, and she does know what’s going on. But she hasn’t helped me at all”. (Case Study K) Ten respondents (31.25%) from the intervention group questionnaire report that they do not feel their DOSH mentor has encouraged them, and nine respondents (28%) feel that their DOSH mentor has not supported them. In addition to this, nine respondents report that they were not given support in the areas they needed the most, and ten respondents feel that their DOSH mentor has not given them helpful advice. Over the course of the programme, a small number of case study participants reported that they had to seek external advice, help and support as their DOSH mentor did not have the necessary knowledge or authority to assist them with their financial problems, as highlighted in the questionnaire and case study interviews: “…needed to be referred to other agencies as DOSH mentor could not deal with assisting me in reducing debts”. (Questionnaire Respondent) A small number of case study participants felt that their mentor had failed to adequately meet and engage with them after the initial catch ups and meetings: “I feel that I’m in a situation where I need, I do need help from their side. And it seems like they are just leaving me to it, and they know I’ve got five children they know I’m battling my mental health, and just seems like they’re just not willing to help me at all”. (Case Study K) Social housing residents often struggle with poor mental health, alcohol, drug and substance misuse and long term sickness. In this research, a number of residents spoke candidly about their own mental health problems, chronic health conditions and alcohol and drug addiction, and the impact this had on their ability to cope with everyday life. The levels of intersecting problems experienced by many of the 26 participants makes it very difficult for any intervention programme to have an impact across all life experiences and financial management issues, and this may account for some of the negative reports of the DOSH programme. 4.6.2 Ask David app In terms of understanding the effectiveness of the financial app in improving residents’ financial capability, the findings are less discernible. 64.5% (n = 20) of phase 2 survey respondents do not use any budget tools and notably only one survey respondent reports using financial apps. While only three survey respondents report using the Ask David App on a regular basis to help them with the dayto-day management of their finances, it is important to note that the overall sample size was small, and therefore it is difficult to determine the overall effectiveness of this element of DOSH in terms of improving residents financial capability and wellbeing. Data presented and analysed in chapter 5 further adds to this discussion on the Ask David app. The low figure on Ask David use can be partly explained by some confusion about the app among residents. A number of participants in the case study interviews report that they were not aware of the app or did not understand how it could help them with their day-to-day budgeting and financial management: “I saw it [email about Ask David from A2Dominion] initially, and to be honest, I didn’t know what it was. It came through as an email, and I thought ‘what the heck is that?’” (Case Study E) A number of case study participants also report that they do not have access to the internet or a smartphone which restricts their ability to download and use the app: “I’ve got no broadband”. (Case Study F) “Our internet is terrible, so I would probably prefer to phone [mentor] than use the app”. (Case Study M) “I haven’t got the phone [smartphone], so it wouldn’t be any good to me…I haven’t got a tablet…I would love to have one [smartphone], but I can’t afford it”. (Case Study A) Low usage of the app can also be explained by the lack of IT skills and knowledge among some residents: “I can’t use things like [apps] that. I don’t know how to use technology like that, I don’t have a computer, a table or anything”. (Case Study I) For case study participants where English is a foreign language, there is an issue around understanding the potential benefits of the app and how it can help support their financial skills and knowledge. This resulted in a lack of engagement with this element of the DOSH programme: “[I don’t know] a lot about that [Ask David App]…I don’t want to use it…I don’t have a reason to use it…How can [it] help, the app is….what kind of help?” (Case Study G) Furthermore, it should be noted that many of these residents lead challenging and complex lives, and struggle day-to-day with a wide range of financial, social and em otional problems which can leave them little time or inclination to engage with different layers of financial intervention programmes: 27 “At the moment, no, because I’m struggling to sort of do my finances which is why I had this organisation more or less doing it for me”. (Case Study K) While it is clear that one-to-one therapeutic mentoring is effective in improving the financial capability of A2Dominion residents who have acute financial difficulties, there is potential for this to be supplemented by less intensive elements, such as the Ask David app. A small number of residents identified from the case study interviews report that they would be willing to use the Ask David app to assist them with their financial and day -to-day management of their budget: “I would be encouraged to do so, if anything was mentioned to me, and I was in financial difficulty as I was, yes”. (Case Study C) Residents are more likely to use the Ask David App with confidence if they have personalised input and support from their mentors: “I think it is something that I would consider to look at, but like I said due to what I’m going through, it’d be good if I could have someone that I could speak face to face to, and like on a one-to-one basis, rather than just doing it on my own”. (Case Study A) Moreover, case study participants need to understand how to use the Ask David App and are encouraged to seek support and advice in using it from either their mentor or other A2Dominion contacts. The Ask David app could add value for residents with lower levels of complex problems and higher levels of IT literacy than the current participants in this research. Further research is therefore necessary to fully understand the potential benefits of fina ncial apps in improving financial capability and overall wellbeing among younger and more tech savvy residents. 4.7 Contribution to WWF Objectives This chapter has highlighted a number of key findings which make a significant contribution to the WWF objectives and areas of focus. The findings from the qualitative and quantitative data reveal significant improvements in financial capability among participants in the DOSH programme – a key Theory of Change outcome for the evaluation programme. Alongside this, the research findings also show an increase in residents’ confidence in managing finances and managing better day-to-day; an improvement in emotional and mental wellbeing; and, increased access to advice services and residents actively seeking advice – all key Theory of Change outcomes. While the questionnaire data from the intervention group indicates that the DOSH programme has a positive impact on residents’ overall financial confidence and wellbeing, the case study interviews give a richer, more rounded and robust picture of its impact. Case study interviews effectively identify and highlight the factors that lead to a positive experience and improved financial capability, stress the different experiences both positive and negative, of the relationship between residents and mentors, and showcase the effects of one-toone therapeutic mentoring on overall improved financial capability and wellbeing. While the questionnaire data showed that just over a third of respondents did not view their one-to-one mentor support as being effective in identifying their needs and offering them helpful advice, findings from the case study interviews indicate a more nuanced and largely positive opinion of the DOSH mentors. This suggests that using targeted one-to-one therapeutic support in programmes such as DOSH may benefit more vulnerable and hard-to-reach residents who experience complex circumstances leading to challenging everyday lives. While the effectiveness of the Ask David app in improving residents’ financial capability and overall wellbeing is not evident, findings from the case study interviews suggest that this type of intervention can supplement more intensive programmes such as one-to-one therapeutic mentoring. It is essential that financial capability interventions that require a certain set and 28 level of skills and competencies are targeted at those who are most likely to engage with and benefit from them. 29 5. Key Findings: Process Evaluation 5.1 Introduction This chapter looks at the process of delivering the DOSH financial capability programme and whether, why, and how, it has been successful. It draws on a range of data sources detailed in chapter 3 to consider the different elements of project delivery. It also critically reflects on the delivery process to consider what worked less well and how the project can be improved in the future. 5.2 Defining ‘success’ At its core, the DOSH programme is aimed at ensuring A2Dominion residents are financially capable so that they are able to pay their rent and keep hold of their tenancies. Success is therefore defined in terms of effectively delivering the DOSH programme, engaging residents about their finances and in the different DOSH elements and, in the longer term, positively impacting on the work of the Income Officers and TSOs. In addition, and as described by the DOSH mentors, success is framed in the following terms: “It's about empowerment, it's about trying to teach them how to handle their money. It's not us just wanting to get our rent”. (Mentor 2) “[Trying] to create, how would I put it, a positive habit, to try and help them sort that out”. (Mentor 3) Success is therefore also about effecting longer term behavioural change in relation to residents’ personal financial management, ensuring they feel empowered and develop good financial habits. To monitor the success of the project, a DOSH programme steering group was set up at the start of the project that comprised members of the core A2Dominion team and representation from the local authority and VCS across the three DOSH areas. The steering group was responsible for sharing their learning and embedding DOSH principles with frontline staff and volunteers during the WWF period. To ensure prompt roll-out, A2Dominion staff were updated on DOSH programme developments with the details and explanations given on the key elements of mentoring and the financial application. In particular the TSOs, who refer residents to DOSH, were briefed about the programme’s aims and objectives to ensure appropriate use of referrals. This integrated internal working at A2Dominion has been critical to the success of engaging residents in the programme. As highlighted in the Theory of Change, there are different elements to the delivery of the DOSH programme and here we reflect on the relative success of these as evidenced by the programme delivery data. 5.2.1 DOSH mentors and volunteers Three DOSH mentors were appointed, one for each of the geographical areas of the DOSH programme. The key responsibilities of the DOSH mentors are: • • • To provide one-to-one financial capability interventions to DOSH participants To support the DOSH project manager and key partners in the design and development of the new service offer To engage residents in the development and testing of a new financial capability app. Crucial to the ‘therapeutic’ dimension of this mentoring is ensuring mentors have close knowledge and/ or experience of the challenges faced by social housing residents, whether due to their experience in the social housing sector, or through personal experiences. In addition, mentors received training based 30 on the skill/will coaching model, focusing on techniques to improve the will of participants to make small changes to how they manage their money, as well as build resilience. In addition, DOSH mentors aim to work holistically with residents to ensure a range of health and social needs and wider wellbeing are addressed. The necessary empathetic and understanding dimensions of the mentor role are critical for successful mentoring, as highlighted by DOSH participants: “It's just… she just understands… she's got kids herself, so she can understand, she can relate as a mother, to what I'm going to through. She understands how the system works, she knows that it's unfair at times to people who are in dire situations, because she's seen it herself. And she's just… I guess it's just experience making her more understanding to what I'm going through”. (Case Study K) “Just having someone to talk to about issues”. (Questionnaire) In April 2017, DOSH mentors completed a 3-day motivational interviewing programme. This aimed to equip the mentors with the skills to conduct practical and effective interviews with residents with a focus on supporting residents to change their financial patterns of behaviour and reducing debt. Internal programme evaluations evidence enhanced understanding of different motivational interviewing skills (e.g. active listening, questioning, change talk, etc.) among mentors. Residents’ experiences of the DOSH mentors are important for understanding the largely positive impact they have, but also for understanding when the mentoring is considered less successful, as evidenced in chapter 4. This was reiterated by A2Dominion’s Director of Housing in her appraisal of the success of DOSH: “That's what I think the mentors are really good at …, I think they've got… they've got sort of quite a wide, open mind about what could help somebody. And I think that's really important”. (Director of Housing) As DOSH mentor case load numbers increased over the WWF period, it becomes ever more challenging to work in this holistic ‘therapeutic’ way that positively impacts on residents. For success to be sustained an audit of mentor-resident ratios is needed to monitor caseloads with continued financial investment in this dimension of the programme. DOSH volunteer training: A2Dominion devised a training course with GGT solutions called ‘Beyond the Fence’ to train and engage staff and volunteers from partner organisations in skill/will coaching to support residents struggling with finances. This course is a motivational, behavioural change workshop aimed at supporting the work of the DOSH mentors and is accredited by the Institute of Commercial Management. 25 Beyond the Fence sessions were run during the WWF period including with Ealing Equality Council and Ealing Community and Voluntary Service. 5.2.2 Engaging residents about finances A2Dominion’s commitment to engaging residents about their finances is an integral first step towards expanding the DOSH programme and the DOSH dashboard has been used to collect and collate data on the number of residents engaged during the WWF period. Several different approaches are taken which include the work of Income Officers and the TSOs, Advice Plus sessions, and the use of Brief Interventions. The success of these is discussed in turn: Income Officers alert the TSOs when a resident is in rent arrears. DOSH is therefore an important channel through which to support those residents who find themselves in contact with the TSOs and DOSH mentors describe the DOSH programme as an “offshoot of our tenancy sustainment programme, where we provide assistance to a vulnerable tenant”. This is articulated by the Director of Housing and mentors: 31 “The DOSH project is really part of a tool that the team can use, to… for that purpose really. So it's quite a crucial part. I mean, the [TSO] team are responsible for collecting £183 million worth of rent every year”. (Director of Housing) “I get a lot of my referrals from the Tenancy Sustainment Officers, and Income Officers… so it kind of gives them another avenue to send people down, before they get to eviction. And we kind of complement the work of the Tenancy Sustainment Officers, who work with people around sort of getting the correct benefits and addressing any problems that might lead to eviction, in terms of finances and stuff. So we're just kind of an extra specialist arm”. (Mentor 1) Mentors perceive this to be working effectively with residents struggling with their finances being referred to them for further support. 169 internal referrals via TSOs and Income officers were made to the DOSH mentors during the WWF period. Advice Plus: an Advice Plus Coordinator was appointed to establish a weekly Advice Plus session in Ealing, located at the Lido Centre. Operating a triage system, the Advice Plus Coordinator engaged clients and signposted them to the DOSH programme and DOSH mentors. These weekly sessions in Ealing are set for extension to two sessions per week. During the WWF period, Advice Plus had 125 visitors. Given its success in Ealing, weekly Advice Plus sessions will be introduced in the Spelthorne area in 2019 and forecasted projections for the next six months indicate a continuous rise in the engagement of residents and non-residents alike. Brief interventions are used to assess an individual’s ability to manage finances and whether they would benefit from assistance. A2Dominion’s brief interventions consisted of a short questionnaire for staff and partners to use with residents and took no more than five minutes to complete (Appendix 10). Brief interventions were undertaken at Job Fairs, A2Dominion’s Beethoven Centre, Aspire, and the Ealing Equality Council. Aspire and Ealing Equality Council are A2Dominion partnership agencies. Aspire, based in Oxford, is a social enterprise with the aim of creating work placements and employment opportunities for local people who face barriers to securing meaningful employment. They operate a drop-in centre in Oxford and refer individuals to the DOSH programme. The Oxford DOSH mentor also visits the centre to engage A2Dominion residents about their finances and discuss the DOSH programme. A large number of Oxford referrals come through Aspire and this has been a successful method of engaging residents about finances. Ealing Equality Council is located in central Ealing and is a charity aimed at promoting and advancing equality and human rights. They run fortnightly free workshops providing free legal aid including on debt and housing-related matters and have referred individuals to the DOSH programme. During the course of the WWF period, 373 residents were engaged through brief interventions. Figure 5.1 shows the outcomes of the brief interventions questionnaire on finances. Those with a low score (107 residents) were offered a referral to DOSH, those with a mid-range score were be prompted to download Ask David or referred to DOSH (227 residents) and those with high scores (39 residents) had no further action. The brief intervention proved successful as it encouraged external partner involvement: of the brief interventions completed, 270 were completed by partners. It also encouraged referrals with 25% of the 107 participants identified for referrals agreeing to this. Through volunteer training, there have been increasing opportunities to advise, motivate and review residents’ financial capability through the use of brief interventions. This method of engagement continues within the core functionality of Advice Plus. 32 Figure 5.1: Brief intervention results Brief Interventions questionnaire on finances Score Range - Fantastic you are managing your finances 39 Score Range - You may benefit from some brief intervention about your financies 227 Score Range - Benefit from speaking to someone - Refer to DOSH 107 0 50 100 150 200 250 Strong, and often established, relationships with key partners have enabled A2Dominion to engage residents about finances and deliver DOSH. The use of Service Level Agreements has successfully communicated the aims and objectives of DOSH to partners and established their role in DOSH delivery. 5.2.3 DOSH one-to-one support During the WWF period, the DOSH Dashboard recorded 131 residents engaging in DOSH one-to-one support through therapeutic mentoring and completing the full support from the mentors. As relayed in 5.2.1, the delivery of mentoring has been successful and outcome evaluation data in 4.6.1 suggests that the one-to-one support has been largely successful for programme participants. There is a perception among mentors that DOSH is offering an important service that does not currently exist in the field of financial capability or in the geographic areas in which it is operating: “There's not a lot of financial advice or people that work with people around debts and stuff”. (Mentor 1) This largely revolves around the ‘intensive’ level of support given to DOSH participants. The importance of having an ‘in-house’ service of this kind is considered a positive, as this enables mentors to work first-hand with residents rather than referring them to external organisations: “And a lot of other housing associations don't have stuff like that even, I don't think. It would be again signposting rather than having someone in-house”. (Mentor 2) Given the wealth of knowledge in A2Dominion, it has “the resources to contact the right people, and know what needs to be done” (Mentor 2) in relation to benefits, employment and other resident needs. When in-house provision is not suitable or sufficient, DOSH mentors refer to a range of external services to support residents. “I would either try and do what I need to do, or refer them to agencies outside, or within A2Dominion, that can help them get on track, and stable”. (Mentor 1) This is reiterated by the Director of Housing: “[O]ne of the DOSH mentors met with a resident, and we got them food vouchers, we got them… we do a back to work, we do a bike recycling scheme. And the guy had sort of said that he was interested in this type of work, we got him onto training, we got him a new bike, you 33 know, that whole thing is like a big package, so that actually you can see your interventions may be quite a few, but what a massive difference you're making to somebody's life. And I think that's, for me that's the important thing”. The one-to-one mentoring enables a range of A2Dominion and external agencies to work together to make numerous changes in residents lives that can affect behavioural change over the longer term enabling greater financial capability. 5.2.4 DOSH digital support The development and roll-out of a digital financial inclusion product was a central component of the WWF project. A2Dominion partnered with Deloitte UK in developing the Ask David financial application. This is a mobile phone app available for download on Android phones. Ask David has two main functions: I. II. A newsfeed of information and links. This function enables the user to engage by scrolling through and picking relevant and useful information A direct messaging service to a DOSH mentor. This function enables users to have remote access to support and guidance as through the chat function they can send messages to a DOSH mentor. The design of this function is aimed at complementing the one-to-one element of the DOSH programme by providing DOSH participants additional access to a mentor outside the one-to-one appointments. Deloitte produced one-page descriptions of 4 personas (Appendix 8) that included background information about the user; their goals and challenges; technology use; their behaviour patterns in terms of income, spending, budgeting and saving; and advice and support they have sought. Figure 5.2 provides graphical analysis of the personas. Together, appendix 8 and figure 5.2 highlight where residents need support with IT skills to benefit from Ask David. These personas largely concur with the case study profiles created through the outcome evaluation case study interviews (Appendix 4) and demonstrate that many residents do require additional support to access and engage with app technology. 34 Figure 5.2 Deloitte persona analysis To support the app development and testing phases, partner organisation evaluations were collected by A2Dominion. Evaluation feedback raised issues relating to the following: offline use; availability of answers to user questions; ‘sharing’ options; search and filtering options. Issues were then fed back to Deloitte for final launch. Since January 2018 Ask David has been available for download on the Google Play store and it has moved from the Beta testing phase to open availability. A2Dominion aimed to engage up to 500 residents online though the app and Google analytics data shows the following since it went live: 1,419 hits 174 active users. 80 live conversations A2Dominion monitor and collate the questions asked and feedback given via Ask David. A redacted and anonymised version of this is in appendix 11. Though in its formative stages when interviews were conducted, mentors expressed optimism that the extended use of technology through the Ask David app will have a positive impact on residents. In particular, use of app technology is perceived as empowering residents to take control of their finances. “Just knowing that whatever question they've got at that moment, or any problem they've got at that moment, could be sorted without them having to come in or call, or book an appointment, or wait for the person to get back to the to the office, or call them back, and just being able to ask the question there and then, and have it answered, definitely will help.” (Mentor 3) “I think we're still [having] teething issues, but… so what we're trying to do is actually give tenants that means of accessing support. We don't necessarily have to have the face to face support, but to give them that resource.” (Mentor 2) 35 Mentors and DOSH delivery partners, tasked with raising awareness of the app, are now encouraging residents to use the app, though this is still in a relatively early stage: “It's started running. Nobody has used it in Oxford yet. Nobody wants to, so far. I keep trying to persuade people to download it”. (Mentor 1) Findings from the Common Purpose/ Queen Mary programme on engaging residents found that engagement with the app is limited as it is only available on android and not on IOS. In addition, though younger residents are reading the app articles, they are less likely to use the ‘chat’ option and therefore this requires further attention. As highlighted in chapter 7, this points to the need for more research in this area. 5.3 Project Delivery: What Worked and What Did Not Work The project has been delivered as intended. There were delays due to changes in core DOSH team management, appointment of an Advice Plus Coordinator and with the Ask David app. These had some impact on project delivery timescale: There was a small change to the core DOSH team in the early stages of the WWF period. However, impact of this was mitigated with quick replacement and DOSH partners were informed. There was a three-month delay in recruitment of the Advice Plus Coordinator which led to a delay in engaging residents in this way. While research on, and development of, the app were relatively swift, there were complications with the testing and roll-out phases of the Ask David app. Two key challenges impacted on the delivery of this component of the DOSH programme: • Organisational considerations: A2Dominion is one of the largest housing associations in the UK with a range of departments with different roles and priorities. The development of a financial app is a new extension to the work of the community investment team. As a result, there were discussions beyond the team about whether such an app, created by an external agency, would fit with the organisation’s ethos, values and corporate IT and communications strategies. Though ultimately resolved, these internal conversations and the divergence of opinions had an impact on the Ask David development and delivery. As the Director of Housing explains: “The issue that we have about the Ask David app is that we are currently going through transformations, … it's whether it fits in with our corporate IT strategy. …It's about how we can evidence that that is crucial to our business”. • Closed Beta stage testing: this gave the project team an opportunity to test the functions of the App in a real-world scenario. Downloading involved numerous steps, including referral, invitation and sign up. This proved difficult for most users and resulted in slow uptake of the app. This in turn rendered it difficult to test the efficacy of the app at the earlier stages of the project. These challenges were reviewed and highlighted the lack of digital skills among some potential users; lack of access to the IOS platform; and, the complexity of the sign-up process for the app in the closed Beta phase. Since January 2018, Ask David has been available for download on Google Play store and has moved from the Beta phase to open availability. 5.4 Improving Processes A2Dominion recognise that the DOSH programme is evolving as it has developed and extended over the WWF period. The process of engaging with residents about their finances has changed as partner organisations have become increasingly aware and enthusiastic about DOSH and recognise its 36 different elements. A2Dominion are keen to learn from the DOSH delivery to improve experiences of, and the support available to, residents: “We're taking on comments and feedback from residents, to sort of continually shape it, so that it's not one dimensional”. (Director of Housing) In addition, knowledge about the app element continues to increase among all those involved in the programme – A2Dominion staff and residents, partner organisations and DOSH participants. This is echoed at senior level within A2Dominion with a reflection on the apps potential, especially among certain groups: "I think the Ask David app is… will play a really crucial part in engaging, … maybe young… groups”. (Director of Housing) With knowledge gleaned from the outcome and delivery evaluations, A2Dominion continue to look for ways to engage residents with the potential to make use of it. However, it also recognises that the Ask David app is not, and will not be, appropriate and beneficial for all: “Let those people that can. Also let's train those that want to, but can't, and then… leave us to deal with the people that probably will never be able to deal with these things on their own”. (Director of Housing) From an organisational perspective, the overriding view of the DOSH programme is a positive one, and it is considered vital to the core business of A2Dominion as its delivers its financial strategy: “I'm starting to form the opinion that it's something that'll need to become part of our strategic objectives, as a business, and part of our corporate strategy. And I think that will be supported within the business. The outcomes are so positive…” (Director of Housing) In chapter 9 we discuss the sharing and learning activities over the course of the delivery project in more detail, and outline how A2Dominon intend to capitalise on the positive delivery of the DOSH programme as part of its broader strategic objectives. 37 6. Key Findings: Economic Evaluation 6.1 Introduction This chapter sets out the costs delivering the DOSH programme and estimates the return on investment using the HACT Wellbeing Valuation approach. 6.2 Delivery Costs The overall cost of delivering A2Dominion’s DOSH programme through the WWF is detailed in Table 6.1. The total project cost is £276,500. Table 6.1: Cost of delivering the DOSH programme Project Outcome evaluation £24,500 Delivery of project Advice Plus £30,000 Ask David £56,000 DOSH Mentors Salaries £99,000 Project Management/training/administration £41,000 Brief Interventions/volunteer mentors (??) £26,000 TOTAL £276,500 3 x FTE DOSH Mentors 6.3 Outcomes Value HACT is the housing sectors ideas and innovation agency and aims to provide what it calls ‘futureoriented solutions, projects and products’ for the UK social housing sector. Working collaboratively with social housing agencies and others in the sector, HACT aims to deliver support, leadership, solutions and research through four specialist areas: value, insight, health and local. In recent years, HACT, in partnership with Simetrica, has produced social impact measurement research and developed practical tools for the housing sector, including ‘Measuring the Social Impact of Community Investment: a Guide to Using the Wellbeing Valuation Approach’ (2014) and the accompanying Value Calculator. Wellbeing Valuation is the latest thinking on valuation of non-market goods and in social impact measurement. This approach also features as part of HM Treasury Green Book (2018) guidance on how to appraise and evaluate policies, projects and programmes. This particular approach has proven to be useful for the social housing sector as it allows housing providers to understand the impact their interventions and programmes have on residents’ lives through the management of their homes, the provision of support to find work or manage finances, and their work, often in partnership with other service providers. This approach also assesses the associations between outcomes and subjective wellbeing, and between income and subjective wellbeing, and estimates monetary values for those outcomes (Vine et al, 2016). The economic evaluation uses HACT’s Wellbeing Valuation approach to calculate the return on investment of the DOSH financial capability programme. The HACT Wellbeing Valuation approach uses 38 self-reported measure of wellbeing (subjective wellbeing) to measure an individual’s welfare, using measures such as life satisfaction. Using this Wellbeing Valuation approach, the DOSH financial capability programme project achieved a Social Return on Investment (SROI) of £2,281,000. For every £1 spent on the intervention, £35 was achieved in social value. 39 7. Limitations of the Evaluation and Future Evaluations 7.1 Introduction This chapter provides an overview of the limitations of the DOSH programme evaluation approaches, highlighting methodological limitations and transferability of evaluation design to other partner organisations. The chapter also outlines plans for future DOSH programme evaluations. 7.2 Appropriateness of Evaluative Approach The three evaluation approaches were appropriate and necessary for the DOSH programme and together provide a robust appraisal of the project’s successes, as well as highlighting areas for development. In particular, the use of a mixed-method outcome evaluation approach methodology using a matched two-stage quantitative questionnaire with intervention and control groups, and complemented by two phases of case study interviews, enabled triangulation of findings and control for external factors impacting on changes in financial capability. Though the sample numbers are small, this has enabled strong and significant findings to emerge. 7.3 Methodological Limitations There are a small number of limitations linked to the different approaches and which impacted on evaluation findings. These relate to size and drop off of intervention group, age profile of participants, and late delivery of the Ask David app. 7.3.1 Intervention group drop off Questionnaire: There was a drop off in respondents between phases 1 and 2 of the outcome evaluation questionnaire; control group drop off was just over half (52%), and intervention group drop off was three quarters (75%). The sample size is therefore lower than indicated in the evaluation plan. While these rates are disappointing, it is important to note that all questionnaire surveys experience drop-off rates, and trends show that rates have been rising in recent years (Mathers et al, 2009). These drop off rates need to be placed within the context of the population being researched. At the time of the outcome evaluation, a number of respondents were vulnerable and hard-to-reach for myriad reasons and, in some instances, were unwilling to engage in Phase 2 due to poor experiences of A2Dominion in areas unrelated to financial capability or the DOSH programme. The dual role of housing associations as rent collectors and service providers has been highlighted as an issue for lack of resident engagement (Wainwright and Marandet, 2017), and this clearly impacted on some respondents perceptions of A2Dominion at the time of Phase 2 questionnaire data collection. This can also be compounded by residents’ negative experiences of A2Dominion in areas such as repairs, tenancy management and rent arrears letter reminders which, in some cases, act as a deterrent and resulted in a loss of contact or refusal to be re-contacted about financial capability. It should also be noted that conducting questionnaires by telephone with social housing residents who are often in vulnerable social and economic situations has a number of drawbacks. In the second phase of data collection, a considerable number of telephone numbers (landline and mobile numbers) did not work which can be attributed to lack of credit or changes in number. Furthermore, contact details were often not updated or amended on the A2Dominion central database and consequently this had an impact on follow-up calls. In addition, some initial respondents and DOSH participants once an issue or concern had been resolved did not feel it was necessary to talk with a DOSH mentor or continue to participate in the programme. Finally, drop off was compounded by a small number of duplications and no match questionnaires. These were omitted from the final numbers and data analysis. 40 When contact was established with participants, most questionnaires were fully completed. Given the questionnaire set up, limited opportunity was given for ‘item non-response’ (missing data) and therefore this has not been a significant issue in the data analysis. Where questionnaire break off occurred, it was ‘unit non-response’ and therefore not included in the analysis. Case study interviews: For the reasons above, there was some drop off in one-to-one case study interviews with 12 out of 14 interviews conducted in phase 2. However, the rich data collected from phase 1 interviews, as well as that from phase 2 interviews, mitigates this as a significant issue affecting the qualitative data collection. Case studies were prepared for all 14 participants as detailed in appendix 4. 7.3.2 Age profile of participants While the evaluation targeted specific groups, namely young adults 16-24, working age people, and those in financial difficulties, the findings show that the outcome evaluation included only one ‘young adult’ aged 16-24. The specific data on this cohort of residents is therefore limited and further research is needed. 7.3.3 Late delivery of the Ask David app Delay in the delivery of the Ask David financial app meant a shorter time frame for mentors to explain its application and encourage DOSH participants to use it. This had an impact on the Phase 2 intervention group outcome questionnaire findings on engagement with, and use of, the app. 7.4 Transferability of the Evaluation Design Recognising and learning from the challenges faced in the delivery and evaluation of DOSH, the findings from this project have the potential to inform and assist other housing associations in their design and delivery of financial capability intervention programmes. A2Dominon have already worked with other housing associations to deliver DOSH training, have secured funding to deliver DOSH training to Thames Valley Housing Association, and there is potential to further roll-out this training and evaluation model to other housing association providers in the near future. Furthermore, there is an opportunity to forge new links and connections with partner organisations by sharing good practice and lessons learned from this programme. 7.5 Plans for Continuation of and Further Evaluation To address limitations, it is suggested that further evaluative research be conducted over a longer period. In the outcome evaluation phase 2 questionnaire, respondents from the intervention group were asked if they could be contacted by Brunel University London in the future. Twenty respondents (62.5%) said yes and contact details have been provided. This cohort provides the potential for a further study and ongoing monitoring of the benefits of one-to-one therapeutic mentoring, and will enable further understanding of how residents are coping with their financial problems and affecting behavioural change. Furthermore, given the less discernible outcome findings on, and the delays with, Ask David, there is real value in expanding the evaluation of its use as a supplement to therapeutic mentoring in improving financial capability for different groups of residents. 41 8. Implications and Recommendations 8.1 Introduction This chapter looks at the different types of project activity and outlines the most successful elements of delivery. It also explains the complexities of working with social housing residents, and details the learnings, recommendations, and potential for future development, of the project and its wider application. 8.2 Activity Delivery Residents’ successful engagement with the project and increased levels of financial capability, confidence and wellbeing can be largely explained by the personalised one-to-one therapeutic support they receive from their mentors. The findings from the evaluation data demonstrate the importance of human interaction and conversation in building trust, rapport and confidence as a precursor to various forms of resident engagement. This project also highlights the importance of ensuring that the types of digital technology used to engage with different people are relevant for their needs and appropriate for their level of education, skills and knowledge. One other area that needs to be addressed when designing and delivering a digital intervention of this nature is securing buy-in from senior stakeholders within the organisation and alignment to broader strategic objectives: “…how do we evidence it’s supporting our residents, because obviously it’s open to the wider public, and I think that certainly if we were…if I was talking to our Chief Executive and said ‘you know, it’s got this, X, Y, Z and benefits’, he would be the one who would be saying, ‘yeah, but what about our residents’”. (Director of Housing, A2Dominion) 8.3 Target Group Working A number of residents on the programme are highly vulnerable and, at times, this impacted on their ability to fully engage with their mentor and the different interventions on offer. Interviews with the three DOSH mentors highlight the range of problems they encounter when trying to access and build relationships with residents who often display challenging and complex behaviours: “You might be working with someone who's been long term unemployed for 10 years, so it can be challenging in that you're asking them to do stuff that they might feel is out of their comfort zone”. (Mentor 2) “It’s just sometimes they’re just not either physically or mentally able to do so [engage], or think about it [finances], or prioritise it”. (Mentor 3) Moreover, mentors face an additional hurdle in the perceptions, sometimes negative, that residents can hold of A2Dominion staff. A previous negative encounter with a member of A2Dominion staff can arouse feelings of suspicion, fear and anxiety, which can lead to disengagement or reluctance to discuss their financial problems with mentors: “I think again, they’re [residents] just like scared of talking about it [financial problems], and most people that are referred, are in quite high rent arrears. And so they don’t want to talk about it, because they think that I will evict them, but it’s getting that message across to people that it’s not my role at all”. (Mentor 1) Residents display varying levels of skills and knowledge of finances and the internet in general. Some residents do not possess the necessary IT literacy skills to avail of the Ask David app. Moreover, a 42 large number of residents do not have the money to purchase a smartphone and pay for continuous internet access. This is something that needs to be taken into consideration and reviewed when rolling out an intervention of this nature which places an onus on individuals to be in possession of the correct technology - often expensive smartphones which are beyond their financial reach. In addition, residents also need to have readily available access to the internet. Nevertheless, there are residents who are already, or who are keen to be, IT literate and tech savvy, and have access to smartphones. These residents are keen to learn more and demonstrate potential to download and use phone apps. This cohort of residents would benefit from a financial app intervention programme. 8.4 Organisational/ Sector Learning Given the ongoing changes to the way benefits are administered, housing associations have an important role to play in ensuring their residents become digitally literate and have access to the internet. In a project of this nature, DOSH mentors are uniquely placed to identify residents who need extra support and help with their IT skills and who could potentially benefit from different forms of engagement and intervention (McHugh and Wainwright, 2017). With this in mind, it may be worthwhile running face-to-face workshops for residents where they are encouraged to improve their IT literacy skills and knowledge, and are also shown how to download and use phone apps. 8.5 Recommendations As a result of the findings from this project, Brunel University London and A2Dominion make the following recommendations: Recommendation 1: Continue to provide tailored one-to-one assistance to ‘vulnerable’ residents who need ongoing and intensive therapeutic mentoring and support. Recommendation 2: The potential for residents to engage with other services and intervention programmes including the Ask David App is something that could be explored further by rolling out the programme to residents who possess the necessary IT skills to use it successfully and with little difficulty. Recommendation 3: A2Dominion should continue to reach out to residents and work with them to improve their IT skills. Running face-to-face IT skills workshops helps residents engage with interventions that rely on technology and high levels of IT literacy. A2Dominion have an important role to play in ensuring their residents become digitally included and have internet access. Recommendation 4: Continue to monitor the numbers engaging with DOSH and its different elements, and audit mentor-resident ratios to ensure mentor workload is sustainable. 8.6 Future Development and Wider Application A2Dominion plan to continue to run the DOSH programme with both the one-to-one therapeutic mentoring and the Ask David app. In addition, they intend to continue using the app with residents who have expressed interest in it and demonstrate the necessary digital skills. A2Dominion would like to: • • Consider employing more DOSH mentors to ensure the programme is delivered over a longer period of time and reaches out to all residents who would benefit from one-to-one therapeutic support and guidance. Consider how best to engage and support people who express an interest in using the app alongside one-to-one mentoring by finding practical and workable ways of increasing residents’ IT literacy levels. 43 • Explore ways of identifying residents who currently use apps on their phones and would benefit from less intensive forms of financial capability interventions such as the Ask David app. 44 9. Sharing and Learning Activity 9.1 Introduction This chapter provides an overview of the sharing and learning activities of the project over the course of the delivery period. It also outlines how the findings will be disseminated with key stakeholders and partners. To date, the following sharing and learning activities have been carried out: • • • • • • • • • Blog pieces outlining the DOSH programme have already been published in partnership with HACT. ‘Communicating Key Findings from the DOSH Project 2017/19’ presentation for the WWF Webinar 3 – Communicating Evidence Effectively. Sharing of preliminary findings at the Total Housing Conference. DOSH training to Thames Valley Housing Association. What Works Financial Capability Workshop and staff newsletter, November 2017. Findings included in 2017/18 Community Investment Report. Updates at Ealing Local Strategic Partnership. Development of mobile and interactive Money Savings Stand for monthly Employment Fairs hosted by A2Dominion (Appendix 12). The stand has been used at Ealing Job Fair in May 2018 and at Slough Job Fair in June 2018. 6 week Fitness Training Course for residents in Ealing and Southall (Appendix 13) These sharing and learning activities helped secure additional funding for the further development of the Ask David app and prompted commissions from other housing associations. A2Dominion and Brunel University London will continue to work collaboratively and devise a strategy outlining how the findings of the final report will be disseminated with key stakeholders and partners. The following sharing and learning activities will be carried out in the near future: • • • • • • • The report will be circulated and shared with the Senior Leadership Team of A2Dominion to inform future decision making and identify what approaches work when designing and delivering intervention programmes to support residents with their financial capability and overall wellbeing. A2Dominion will work with HACT to produce a framework model to deliver therapeutic one-toone mentoring interventions drawing upon best practice. The report will be shared across the g15 – London’s 15 largest housing associations. A2Dominion will draw attention to the challenges encountered when engaging with vulnerable and hard-to-reach participants with the housing association community and other stakeholders. A2Dominion will work with HACT to promote the report and DOSH programme across social media including A2Dominion’s Facebook and Twitter accounts. An application will be submitted to London North West University HealthCare NHS to fund a small community wellbeing team in Ealing who will offer one-to-one therapeutic mentoring to residents identified as ‘struggling’ financially and experiencing poor mental health. Brunel University London will publish journal articles contributing to the wider academic literature on financial capability in the UK. 45 References Atkinson, A., Mckay, S.D., Kempson, H.E. and Collard S.B. (2006) Levels of financial capability in the UK: results of a baseline survey (Consumer Research 47), Financial Services Authority Barcellos, S., Carvalho, L., Smith, J. and Young, J. (2016) Financial education interventions targeting immigrants and children of immigrants: results from a randomized control trial, The Journal of Consumer Affairs, Summer: 263-285 Chartered Institute of Housing (2011) Improving financial inclusion and capability in social housing. Coventry, CIH Collard, S., Finney, A., Hayes, D. and Davies, S. (2012) Quids in: the impact of financial skills training for social housing tenants, Bristol, Personal Finance Research Centre Fincap.org.uk (2015) Financial Capability Strategy for the UK https://www.fincap.org.uk/uk_strategy HM Treasury (2018) The Green Book: Central Government Guidance on Appraisal and Evaluation, London Hickman, P.., Kemp, P., Reeve, K. and Wilson, I. (2017) The impact of the direct payment of housing benefit: evidence from Great Britain, Housing Studies, 32: 1105-1126 Johnson. E. (2007) From financial literacy to financial capability among youth, Journal of Sociology and Social Welfare, xxxiv: 119-145 Lloyd J. and Lord, C. (2015) Defined capability: pensions, financial capability and decision-making among retirees, London, Strategic Society Mathers, N., Fox, N. and Hunn, A. (2007) Surveys and Questionnaires, The NIHR RDS for the East Midlands / Yorkshire and the Humber. McHugh, E. and Wainwright, E. (2017) Financial Capability and Resident Engagement: What works? London, Brunel University London Money Advice Service (2017) The Money Advice Service 2017/18 Business Plan, London, Money Advice Service National Housing Federation (2008) A guide to financial capability for social housing tenants, London: National Housing Federation NICE (2014) Behaviour change: individual approaches, London, NICE Pruitt and Adlin, (2006) The persona lifecycle: keeping people in mind throughout product design, San Francisco: Morgan Kaufmann. Tanner, B. (2018) Housing related arrears reveal ‘dismal failure’ of Universal Credit, https://www.24housing.co.uk/news/housing-related-arrears-reveal-dismal-failure-of-universalcredit/#.Wx5-xwV38gc.twitter Taylor, M., Jenkins, S. and Sacker, A. (2009) Financial capability and wellbeing: evidence from BHPS, Essex, Institute for Social and Economic Research Taylor, M., Jenkins, S. and Sacker, A (2011) Financial capability and psychological health, Journal of Economic Psychology, 32: 710-723 Toynbee Hall (2017) An evaluation review for the DOSH project, London, Toynbee Hall 46 Toynbee Hall (2016) The Toynbee Hall financial inclusion health-check for organisations. Report for A2Dominion, London, Toynbee Hall Vine, J., Adams, M.R., Knudsen, C., Lawton, R. and Fujiwara, D. (2016) Valuing Housing and Local Environment Improvements using the Wellbeing Valuation method and the English Housing Survey: Results and Guidance Manual, London, UK. Wainwright, E. and Marandet, E. (2017) Social housing and new forms of tenant engagement: welfare, governance and active citizenship, London, Brunel University London Williams, S. (n.d.) Increasing the parity of the pound for social housing tenants: how systematic financial inclusion can help mitigate the impact of welfare reform, London, Toynbee Hall 47 Appendices Appendix 1: Ask David Financial App Flyer produced by A2Dominion 48 Appendix 2: Phase 1 Intervention Group Questionnaire 49 50 51 52 53 54 55 Appendix 3: Phase 2 Intervention Questionnaire 56 57 58 59 60 61 Appendix 4: DOSH Participants Case Study Interviews - Bibliographic information Below is a brief case study for each of the fourteen DOSH participants interviewed. The information compiled here comes from questionnaire data and phase 1 and phase 2 interview discussions: Case Study A Resident A is a female in her mid-30s and has been an A2Dominion resident in Ealing for two years. In 2015, she had a serious accident and is currently unable to work. Recent changes to her benefits have resulted in rental arrears, mounting debts and an increasing reliance on food banks to feed her young family. She has had several meetings with different members of the A2Dominion support network over the past 6 months, and more recently with her DOSH mentor. Currently pregnant and experiencing poor ill health, her mentor has been providing her with support and reassurance. Case Study B Resident B is a female in her early 60s, has lived in Ealing since 1979, and has been in social housing for 29 years. She currently works as a carer and lives alone. A2Dominion has previously provided employment training and help with finding Resident B a job after she found herself unemployed following the collapse of her business. More recently, A2Dominion have been supporting Resident B with her budgeting through the DOSH programme after she found herself falling into rent arrears. Case Study C Resident C is a male resident in his mid-40s who lives in Oxford and has been an A2Dominion resident for more than twenty-five years. He lives alone and is employed full-time. He recently returned to work after long-term sickness and is currently being supported by the DOSH programme after falling into rent arrears while unable to work. He has had several meetings with his DOSH mentor. His mentor has recently attended court on his behalf to negotiate an agreement on the repayment of rent arrears. He feels confident about the future and managing his finances independently. Case Study D Resident D has recently moved to Spelthorne and is a female in her mid-30s and has been a resident with A2Dominion for just over a year. She has four children and is employed part-time, working 22 hours a week. After receiving support in the form of employment training from A2Dominion in the past, she contacted them about her mounting debts and money worries. She is now involved in the DOSH programme and praises her mentor for the respect he has shown her over the course of their meetings and now feels confident applying for tax credits and using the benefit calculator. Case Study E Resident E is a female in her mid-40s and has been a Spelthorne A2Dominion resident for 7 years. She is single, has two small children and is employed full-time in retail. After falling into rent arrears she made contact with A2Dominion and was put in touch with a DOSH mentor and placed on the DOSH programme. She has had several meetings with her mentor who has helped her fill out forms to claim additional benefits. Her mentor has also encouraged her to try swimming to improve her health and overall wellbeing. Case Study F Resident F is a male in his early 60s who lives in Oxford and has been an A2Dominion resident for more than five years. He has no formal qualifications; however, he has recently returned to college to study, and is currently involved in a training programme which requires him to do voluntary work. He is separated from his wife and three children. He was referred to A2Dominion after two organisations he 62 visits on a weekly basis for emotional and mental support became concerned about his mental health and wellbeing. Since his involvement with the DOSH programme, he has had several meetings with his DOSH mentor. His mentor has helped him produce a payment plan for his rent arrears and he has found it very helpful in budgeting and paying down his outstanding debt. His mentor also helped him write a letter to address ongoing delays to his Universal Credit payments. He is now in receipt of Universal Credit and making regular rent arrear repayments. Case Study G Resident G is a female in her mid-40s has been living in the London Borough of Ealing for 10 years and an A2Dominion resident for 4 years. She lives alone and works 10 hours a week in a school but would like to increase her hours to full-time in the near future. After falling into rent arrears, she was placed on the DOSH programme and has had three meetings with her DOSH mentor. Her mentor has helped her fill out forms for additional housing benefits and set up a monthly budget plan. Case Study H Resident H is a male resident in his early 40s, who currently lives in Oxford and has been an A2Dominion resident for 18 years. Currently long-term unemployed due to a mental health condition, resident H has been receiving support and guidance from A2Dominion about his financial situation and changes to his benefits. He is under the guidance of a DOSH mentor. His mentor has given him advice and on how to reduce his bills and stick to a monthly budget. With the help and guidance of his mentor, he also managed to successfully switch to a cheaper energy provider. Case Study I Resident I is a female in her early 60s who lives in Spelthorne and has been an A2Dominion resident for more than five years. She lives with her son, has no formal qualifications and is currently unemployed due to ill health. Recent family bereavements and depression have resulted in resident I falling into financial difficulties; this prompted her to make contact with A2Dominion who placed her on the DOSH programme. She has been receiving support and guidance from a DOSH mentor over the last 6 months. With the help and encouragement of her mentor, she has started attending a weekly depression support group and feels that this is having a positive impact on her overall wellbeing. In addition to this support, her mentor has helped her set -up direct debit payments for her TV Licence and given practical advice on paying utility bills. Case Study J Resident J is a male in his mid-50s who lives in Oxford and has been an A2Dominion resident for four years. He lives alone and is currently unemployed due to ill health. A2Dominion contacted him after he fell into rent arrears and placed him on the DOSH programme. He has met with his DOSH mentor on two occasions. Case Study K Resident K is a female in her early 30s and currently lives in Ealing with her five young children and partner. She has been an A2Dominion resident for three years. After her husband passed away last year, Resident K found herself falling into rent arrears and was contacted by A2Dominion and subsequently put on the DOSH programme. Since starting the programme she has met with her DOSH mentor on several occasions. She suffers with poor mental health and has recently being told to evict her property and attend a court hearing. Her mentor is currently helping her apply for discretionary housing payment. She would like more support and guidance from her mentor. 63 Case Study L Resident L is a male resident in his late 50s who grew up in Oxford and has been with A2Dominion for more than five years. He lives alone and has two daughters. He is registered disabled and has been out of work for more than twelve years. Following a Work Capability Assessment his benefits were stopped which resulted in him falling into rent arrears. A2Dominion have referred him onto the DOSH programme, and he has been having regular meetings with his DOSH mentor. Case Study M Resident M is a female in her late 50s who lives in Spelthorne with her four children and has been an A2Dominion resident for more than 14 years. Although she has no formal qualifications she has recently completed an employment training course and is hoping to return to work shortly. The breakdown of a relationship and changes to her benefit entitlement resulted in Resident M falling behind in her rent payments. After receiving a letter from A2Dominion about her rent arrears she was put on the DOSH programme and has had several meetings with her DOSH mentor. She has established a very good relationship with her mentor who is supporting her with applications, letter writing and general communication with the local council. She is feeling very confident about the future and her ability to stay on top of her bills and financial commitments. Case Study N Resident N is a male in his early 50s who has lived in Spelthorne and been an A2Dominion resident for more than 10 years. He lives alone and has three children. Ill health and a recent operation have left him unable to work, however he would like to find employment once he has made a full recovery. He was put on the DOSH programme after A2Dominion made a house visit. He has had several meetings with his DOSH mentor. Currently recovering from a major operation, he has fallen behind on his payments and is having to borrow money from family members to cover everyday essentials. 64 Appendix 5: Interview Guides for Phase 1 and Phase 2 Case Study Interviews Evaluation of the DOSH financial capability programme TOPIC GUIDE – Wave 1 INTERVIEWS with HA residents Introduction • Thank respondents for coming • Introduce self and Brunel University London • Explain purpose of the project and aim of the interview discussion • Ask participants to take time to read and sign the consent form (if not done ahead of time). Otherwise, reaffirm: - consent to participate and for the interview to be recorded (emphasise that the recording will only be heard by members of the project team and transcribers) - that the interview is confidential and pseudonyms will be used in any publications - that participants can choose not to answer individual questions - that participants can withdraw part way through with no implications • Hand out expenses • Encourage participants to speak openly, freely and honestly (and explain that the interview will be used to formulate recommendations for a report to housing associations and other interested parties) Introductions Ask the interviewee to introduce themselves • Name and age • How long have you been an A2D resident? • How long have you lived in this area? Biographical details • Explore background PROBE Education, employment history, family and children • Interests and aspirations for the future? Discussion to build up biographical picture of participant Information about A2D provision • What A2D services have you accessed over the past 12 months? Why? • How did you hear about them? Where did you find out about them? • Were you surprised that A2D provides this kind of services? If so who / what organisation would you expect to provide these services? • Do you think it is a good idea for HAs to provide services for their tenants? Are there any downsides? Pressures/ Motivations • What led up you to getting involved with DOSH? PROBE financial concerns and their impact on day-to-day living. Explore what led them to this point. Health, family, wellbeing, job etc. • Why DOSH? PROBE for reasons if different from above. • Did you seek advice from any other organization? 65 • Do you feel under pressure to engage with DOSH? If so, from whom? PROBE A2D, staff, self, family, wider pressures. Expectations of service provision • What are your expectations of DOSH? PROBE • What information have you received about the programme? • What do you think will be the benefits of using the services? PROBE for most important ones. Experiences of initial contact with A2D staff involved in DOSH • How did you initially hear about DOSH> ? • How was contact established? PROBE for role of TSO and income officers PROBE for how contact was established – house visit, phone call etc. • How many meetings have you had? How many more do you anticipate? • Any concerns about the programme? Expectations about the future • What outcomes do you expect after going through the programme? PROBE for employability, self-esteem, knowledge, wellbeing etc. Thank and close • Thank participants • Reminder about confidentiality 66 Evaluation of the DOSH financial capability programme TOPIC GUIDE – Wave 2 INTERVIEWS with HA residents Introduction • Thank respondents for coming • Re-introduce project, self and Brunel University London • Explain purpose of this second interview • Reaffirm: - consent to participate and for the focus group to be recorded (emphasise that the recording will only be heard by members of the project team and transcribers) - that the interview is confidential and pseudonyms will be used in any publications - that participants can choose not to answer individual questions - that participants can withdraw part way through with no implications • Hand out expenses • Encourage participants to speak openly, freely and honestly (and explain that the interview will be used to formulate recommendations for a report to housing associations and other interested parties) (Questions to be asked in conjunction with individual answers to exit questionnaire) Catch up Ask how things have been since last met. General discussion on any changes since last met. Experiences of DOSH • Recount their expectations from previous interview. Did your experiences of DOSH meet your expectations? PROBE • What do you like/dislike about this service? • Working with staff. PROBE mentoring and therapeutic encounter. How did this work? PROBE for process and content of meetings/ discussions How effective were these? PROBE for needs identified, support offered. How would you describe you mentor? PROBE responses in relation to encouragement, support and help. Other outcomes? What they liked/ disliked? Continued contact? • Working with the app? Did they/ didn’t they use this? Why/ why not? How effective is this? PROBE ease of use, any difficulties, what they liked/ disliked? PROBE support the app provides – including detailed examples./ Continued use? • Did you find any aspects of DOSH challenging? PROBE for details at various stages if relevant. • What are the benefits of having gone through DOSH? PROBE for sense of financial capability, financial knowledge, changes to behaviour, confidence etc. Identities 67 • • How do you feel about your future now? PROBE for range of emotions (link to wellbeing questions in survey) Has using being involved in DOSH made you feel differently about yourself? PROBE reasons. Expectations about the future • What outcomes do you expect from DOSH? PROBE for self-esteem, knowledge, wellbeing, financial sustainability, others etc. • What are your plans for the future? PROBE • Would you encourage others to use DOSH? Which particular aspects? • How could DOSH be improved? PROBE different aspects. Thank and close • Thank participants • Reminder about confidentiality • Ask them if they would like to be sent an executive summary of the research when it is completed (in 2018) and if so check their contact details (email or postal address). 68 Appendix 6: Interview Guide – DOSH mentors Evaluation of the DOSH financial capability programme Interview guide for DOSH mentors Introductions • • • • • • Introduce self Explain the purpose of the project and aim of the interview Ensure that participants have received and taken time to read the Info Sheet. If this is not the case, allow time for them to read it. Ask if they have any questions. Ascertain that respondents have about up to 1 hour to devote to the interview Ask respondent to sign consent form (or email it in the case of a telephone interview) Re-affirm: - That they can stop the interview and/or withdraw from the study at any time - That they consent to have the interview recorded - That they are happy for us to use quotes from the interview in reports and other publications using a pseudonym. - That their names and roles will not appear in any publications. The interviewee • Ask interviewee to introduce themselves (job title, description, day-to-day responsibilities and concerns) • Ask interviewee about the role of the organisation for which they work. PROBE for brief history of the organisation, sources of funding, partnerships with other organisations Role of the organisation in providing DOSH Information: • How long has DOSH been running in this area? • What kind of services does it include? • How many HA tenants are involved? PROBE for gender, age, ethnicity, % of social housing tenants etc. • Where is DOSH provided? PROBE for locations and catchment area Specificity of DOSH • How do you decide what to include in the DOSH programme? PROBE for interest/ popularity, local demographics and economy, etc. • How do you ensure you do not duplicate provision elsewhere? • Do you liaise with other organisations? PROBE for kind of organizations e.g. information and advice centre, council, HA, job centres, training providers, community groups, etc. • What is the DOSH programme’s ‘niche’? What do you do that other providers don’t do? • Do you feel there is a gap in the current provision? If so, PROBE for details and how this could be addressed. 69 • • • How does DOSH fit with other roles of HA? Are there potential conflicts between these roles? What are the main challenges you face as an organization with regards to providing DOSH? PROBE for specific role in providing services to HA tenants. Provision and take by HA tenants Motivations and barriers • What are the main aims of DOSH? • Who do you provide these services for? PROBE for tenants or residents in the local area. Particular types of tenants. How do you decide who to target? • How do you recruit tenants? PROBE for targeting of tenants/ use of profiling tools. • What do you feel are their motivation for engaging in DOSH? • What in your opinion are the main barriers preventing tenants/ residents enaging in DOSH? How do you address these? • How do you engage tenants/residents that are harder to reach? PROBE for strategies if suitable. • Do you feel your tenants receive sufficient information and advice and guidance before engaging IN DOSH? PROBE for reasons for choosing particular courses. • Are there any pressures for HA tenants to engage in DOSH? PROBE source – Has, Govt, Society etc. Expectations, experience and outcomes of DOSH • What are the expected/ actual outcomes of DOSH? PROBE for employability, identity, household debt, health, wellbeing etc. • What are the main challenges faced by HA tenants prior to engaging in DOSH? • What are the main challenges faced by HA tenants once they engage? • What are the main challenges faced by HA tenants once they finish? • Do you feel DOSH can be improved? PROBE for reasons • (Where applicable) How is the retention rate overall? Conclude • Thank interviewee • Do you have any questions about the research? • Tell them we’ll send them a copy of the summary report in January 2018 70 Appendix 7: Interview Guide – Director of Housing, A2Dominion What is the role of DOSH in your organisation? Describe its importance Part of core service delivery? How does it link to role of HAs in the social housing sector? As a Director, what benefits do you think it brings to your residents? What is its potential? Sustainability of the project? As an organisation, what are the expected outcomes for the DOSH programme? What are the anticipated outcomes? Which are the most important dimensions? (consideration of technology/ 1-2-1 support) Other short and long term outcomes? How do you see it shaping future policy development and service delivery within A2D? Anticipated expansion/ rollout? Other wider application and impact? Do you see it as having wider significance within the HA sector? Link to role of technology in service delivery Alongside one-to-one personalised support? 71 Appendix 8: Deloitte Ask David User Personas 72 73 Appendix 9: Queen Mary, University of London 74 Appendix 10: Brief Interventions – Questionnaire Survey on finances - A2Dominion residents Questions Scoring System 1 2 3 0 4 I regularly save from my weekly /monthly income Strongly disagree Disagree Neither Agree Strongly agree I am able to save for certain items or events Strongly disagree Disagree Neither Agree Strongly agree I know how to monitor my spending to make ends meet Strongly disagree Disagree Neither Agree Strongly agree I aim/try to live within my means and have a surplus left over each month Strongly disagree Disagree Neither Agree Strongly agree I know how to manage my finances and feel confident doing so Strongly disagree Disagree Neither Agree Strongly agree I try not to make impulse purchases/decisions unless I know I can afford to that week/month Strongly disagree Disagree Neither Agree Strongly agree I know how much money I have to spend each month/week and do not go over my budget Strongly disagree Disagree Neither Agree Strongly agree I know how and where to go for information and advice before I make financial decisions: Strongly disagree Disagree Neither Agree Strongly agree I worry about my financial situation Yes No Your Score Grand Total Scores: 0-10 11-26 27-32 You may benefit from speaking to someone about your finances: Refer to Dosh You may benefit from some brief advice about your finances Fantastic you are managing your finances great Name: Address: Contact Number: E-mail Address: 75 Not sure what to say to residents when they say they’re in financial trouble? We have some handy tips: 1. Collate and organise all documents related to your income/Benefits. 2. Collate and organise all documents related to expenditure (bills). 3. Use comparison websites to check prices for utility bills. 4. Look at your bank account regularly and download the bank’s app who you’re a customer with. 5. Write out all your income and expenditure & develop a budget plan here. 6. Review your outgoings on luxury items (satellite TV, Netflix). 7. Set yourself a savings goal (holiday, new car, Christmas, upcoming birthday). 8. Contact people that you owe money to if you can’t afford your bills. 9. Consolidate debts. 10. Download ‘Ask David’ from Google Play 76 Appendix 11: Ask David Questions, feedback and tips of the days A2Dominion resident question: Good Morning. If I receive personal independent payments will it interfere with my universal credit payments? Feedback from Ask David App: Hi there! If you’re an adult and getting either Personal Independence Payment (PIP) or Disability Living Allowance (DLA), it will continue to be paid along with your Universal Credit payment. You get these benefits if your condition is severe enough for you to qualify for them. They won’t affect the amount you get in Universal Credit. However, if you’re claiming DLA or PIP for a sick or disabled child, the rate of benefit you’re getting can affect your Universal Credit payment. A2Dominion Resident question: Hello Conor, I received the pip reconsideration. The decision is the same. They refer to slightly reduce and adequately dealing at face to face and clinical reports. My son having some major depression, he's at Bray side unit. I am more concerned about him now. I'll leave my claim as it stand'. Tks A2Dominion Resident question: Hi, do you help with shared ownership problems? Examples of Ask David ‘Today’s Tip of the Day’ Conversations Example 1: Ask David App: Hi xxxx, Today’s Tip of the day - Is today a financial struggle? It’s ok to talk about it. Ask David can help, just drop us a line. A2Dominion Resident xxxx Yes. Let's talk. Ask David App: How can we help? Example 2: Ask David App: Hi xxxx, Today’s Tip. March & April is the time of renewing bills. Don't forget to use comparison websites to get the better deals. A2Dominion Resident xxxx: Hi xxxx, I am already doing this and find it very useful as it has saved me money over the last couple months. Kind regards Example 3: Ask David App: Hi xxxx, Today’s Tip. March & April is the time of renewing bills. Don't forget to use comparison websites to get the better deals. A2Dominion Resident xxxx: Thanks, great advice Examples of Ask David Today’s Tip of the day posted Hi xxxx, Todays money Saving tip -Make use of over-ripe fruit by turning them into cakes and desserts. Browning bananas can be mashed up and used to make delicious banana loaf. Hi xxxx, A Money Saving Tip for a weekend challenge - How much can you save by using coupons? Search the internet, magazines, side of boxes for money of coupons, 2 for 1 and see how much you can save over the weekend. But remember… Buy only what you would need. We would love to hear of success stories on Monday! Hi xxxx, 1p Saving Challenge: Easily Save Over £650 in Less than a Year - https://skintdad.co.uk/skint52-week-saving-challenge/ 77 Hi xxxx, Today’s Money Saving Tip - Christmas may seem like a distant memory, but are you still paying off Christmas Debt. For assistance or help send a message to Ask David. Hi xxxx, Today’s Money Saving Tip –have an early spring clean and sell what you don’t need – Reply and Ask David Hi xxxx, Today’s Tip of the Day. Some newspapers offer a £5 cash incentive for when you collect tokens, or rewards that you can exchange. Swap the daily purchase and look on line for the news, and save the money you would have spent. This could equate to the equivalent of around £17 per month if you purchase a paper every day. Hi xxxx, Today’s Tip - How much have you saved from all the tips of the day and challenges set. Reply back and share your progress 78 Appendix 12: Money Savings Stand 79 Appendix 13: 6 Week Fitness Training Course for residents in Ealing and Southall 80