In our last training area, we developed program objectives within each of the four BSC perspectives. We contemplated the way we could collect information on each objective so that it gave us knowledge about where we were located, relative to our own vision and measures of success.
Objectives within the Funder perspective were set to meet the requirements of the ILC team and other donors. Within the Member perspective, we considered what it is that our members most value and want to experience from our peer groups. We then delved internally in the Build perspective and thought about what our program would need to develop and invest in for meeting those needs. Finally, we considered what knowledge we would need to learn now and in the future for continued success in our peer program delivery through the forward thinking Learning perspective.
Every evaluation (or evidence gathering) program needs clearly defined components and implementation plans. For each objective, you should try to formulate a measure that tells you whether the objective has been met. We have also considered this previously and selected indicators for each objective within the tables under each BSC perspective. By keeping a systematic record of all the indicators associated with each objective, you are systematically measuring how well the programme is doing. This is how we know where it is that we are, on our journey toward our vision.
What exactly are indicators? These are variables, or measurable pieces of information, which signify whether your peer program is achieving an objective. In some cases, you will be collecting evidence on your processes (i.e. have you delivered expected activities?). Others will be about outcomes from your activities. Process evidence informs you of how a peer program is implemented. It takes into account the various program inputs in their entirety. This includes our rights-based foundations, elements of good practice, specific goals and objectives selected, as well as, resources available. Process evidence also relates to the specific activities, such as, the facilitator training you provide, the group topics you discuss, and all other elements, right down to the individual peer support interactions and participant reactions.
Some areas you may contemplate collecting evidence on could include:
- Program context, influential aspects of the disability community
- Summarize evidence on who is participating in your groups;
- Evidence relating to how peer supporters/facilitators, are trained;
- specifics about peer supporter and participant interactions;
- Evidence on whether the delivery occurred as designed/planned.
Some ways you might measure these kinds of process objectives consist of:
- Interviews /questionnaires with peer supporters/participants/stakeholders;
- Observation of training sessions and peer support interactions;
- Administrative bookkeeping (such as, how many training sessions were held).
Outcome evidence tells you something about the changes which resulted from those activities and services. We will want to collect information on the impact of the peer support program. This applies both in terms of individual members and, hopefully, in the longer term, overall society attitudes around core concepts such as inclusion and equity. For example, we may want to ask members whether they know more about accessing the NDIS than before joining the group. Do they feel they now have greater autonomy in their life? Do they feel they have more choice and control?
As you plan the evidence you will collect, you need to consider the best available indicator to assess the achievement of each specific objective. If you collect information on that indicator, are you sure that this will tell you if you have achieved that objective?
Broadly, it is possible to collect two main types of information on an indicator. These are information from secondary data, and information that is primary data. Secondary data involves gathering information from sources that have already been compiled in some way. For example, if a peer program objective is responsible budgetary management, a selected indicator may be a variance, (i.e. the difference between budgeted spending and actual spending). This information is likely already calculated for Board or Finance Committee meetings and reporting. If we use it to measure this objective, then we are using secondary data for this information collection. You should reference secondary information; that way, everyone knows where it has been drawn from (and who prepared it, along with its other uses, if appropriate). Documentation provides an ongoing record of activities. These records can take the form of informal feedback from peer group members, reflections through journals of group facilitators or progress reports. The challenge of documentation is that it requires an ongoing commitment to regularly document thoughts and activities throughout the evaluation process. Sometimes this can be overlooked, particularly given the strict time frames of delivering peer support.
Secondary data is the least expensive way of gathering evidence. This is because the only cost is in collating it for project purposes. The evidence was already being collected; meaning there is no additional outlay or investment required. The risk is that it is being developed for another purpose, so it may not be the most accurate or tailored measure possible for that peer program objective. As is often the case, there will naturally be a trade-off between the information’s cost and benefits. This needs to be factored in when planning your overall evaluation, or ‘information gathering’ project.
Primary data is relevant information that comes from the project using purposeful observations and measurements collected. This evidence will form the basis of each thorough investigation of where your peer program is currently located. Evidence collection projects may involve the collection of qualitative and/or quantitative primary information. Quantitative information is collected through measurement and is able to be processed using computational, statistical or other techniques. This contrasts qualitative information, which is gathered using observation or subjective judgment and does not involve measurement (at least immediately). Qualitative information may be processed or quantified where appropriate, or it may be through images or as text (such as quotes by Members, or feedback from Funders). The nature of qualitative data should provide sufficient information for analysis and conclusion. Naturally, the type of information collected should be determined by the objective itself and the indicator selected.
When contemplating collection of information from primary sources, various methods are possible. While we may be able to collect evidence on our peer support via observations, this is generally not used in the peer space. Given our focus here on peer programs, we are going to consider questionnaire data, which is evidence we obtain by asking people. The two key collection methods here are either, from interviews or surveys.