6.1 An Overview of Gathering, Managing and Communicating Information

6.1.1 Knowing the Journey Data Will Take

Data travel. On this journey they are gradually collated and analysed as the data move from field sites or different project staff and partner organisations to be centrally available for management decisions and reports. The journey involves a transformation from data to information and knowledge that is the basis of decisions. Data are the raw material that has no meaning yet. Information involves adding meaning by synthesising and analysing it. Knowledge emerges when the information is related back to a concrete situation in order to establish explanations and lessons for decisions. Many rural development projects have much data lying around, less information, little knowledge and hence very little use of the original data for decision making (see Box 6-1). To avoid this problem, plan not only how you will gather data but also how you will transform the data into valuable knowledge.

Figure 6-1 shows how data travel. Table 6-1 lists questions that need to be considered for each part of the journey. For each performance question and indicator, the journey will be different in terms of the choice of methods, frequency and responsibilities. Irrespective of the journey, be sure that the information you are collecting is helping you answer your performance questions (see Section 5).

Figure 6-1. The journey data take

Table 6-1. Preparing the journey for your data

6.1.2 Considerations When Choosing Your Method

Before choosing your method, be clear about three methodological aspects:

• the difference and overlap between methods for qualitative and quantitative information;
• the implications of working with individual or group-based methods;
• what makes a method participatory � or not.

Several steps need to be followed to select the most appropriate method(s) (see 6.2.2 for more details):

1. Check that you are completely clear about what information you need collected, collated, analysed or fed back, for which you are seeking a method.
2. Check that another group, person or organisation is not already collecting the data. Check, where possible, how the information was collected to see if it is reliable enough for your needs.
3. Be clear about how accurate you need to be.
4. Does the information relate to a specialist area? If so, seek specialist advice or documentation before proceeding with the method selection.
5. Be clear about the task that needs to be accomplished, and whether this concerns qualitative and/or quantitative information. Consider whether a method is needed to collect, collate, analyse, synthesise or disseminate information.
6. Decide the extent to which the data gathering or analysis process is to be participatory, and therefore whether you need to work with individuals, groups or a combination.
7. Decide if your data-collection coverage is to be sampled or comprehensive. If working with a sample, decide on your sample size, clarify the "sampling frame" and select your sample (see D.1).
8. Do you have several methodological options or is there only one? List your method options and make an initial selection. If using a sequence of methods, check that the methods complement each other.
9. List your methods and make an initial selection.
10. When you think you�ve got the right method for the task at hand, consider if it is: feasible, appropriate, valid, reliable, relevant, sensitive, cost-effective and timely.
11. Pre-test your method, with a small number of participants who are similar to those from whom information is going to be sought. Adjust your method based on recommendations from the test run.
12. Determine the frequency of use.

6.1.3 Gathering, Collating and Storing Information

When preparing for data gathering, do not forget to:

• Consider carefully how to select interviewers and facilitators.
• Consider how to distribute the tasks of collection and analysis among different people and what is needed to limit errors.
• Ensure that those using the methods are comfortable with them.
• Ensure clarity of language.
• Prepare the practicalities of each method, such as materials needed.

Avoid error by considering possible causes of sampling errors and non-sampling errors. Non-sampling errors are particularly critical. These can occur due to interviewer bias, inadequacy of methods, processing errors and non-response bias (see 6.3.1).

Check your data from time to time. Spot checks are important at the beginning of any project � if you are using existing data sets � by looking at where data come from, who has collected information and the methods and standards they used. Also check data collection when using a new method or when working with new fieldworkers, new implementing partners, new staff, etc.. Data can be suspicious if you notice overly precise data (like perfect matches between targets and actual realised activities), sudden large changes in data, and data gaps.

For each bit of information, define how it will be recorded. Practise with the people doing the recording before setting out to collect data.

The step of collating (or aggregating) information often gets lost in the gap between data collection and analysis. It requires some attention as it can greatly facilitate analysis if undertaken well and can introduce error if done poorly. Collation is needed when you are scaling up your information from a smaller unit of analysis to a larger one or when information has been collected from different sources with different methods. The collation of qualitative data requires special care and analytical skills.

Qualitative and quantitative data analysis are both critical for making use of M&E data but are also quite distinct processes. The Guide focuses on aspects of qualitative data analysis as statistical procedures fall outside its scope. Refer to Section 8 for many ideas on how to encourage reflective meetings and analytical reporting in addition to the ideas in 6.4.2.

When deciding how to organise the storage of M&E information, consider these four questions(also see 7.5):

1. What information needs to be stored?
2. Who needs access to the information and when?
3. What type of information needs to be stored � hard copies or data that can be computerised and accessed centrally?
4. Regularly assess what information you need to keep and what can be discarded.

6.1.4 Considering Communication of M&E Results

M&E-related findings have many potential audiences: funding agencies, steering committees, cooperating institution, project and implementing partner staff, and primary stakeholders. The main purpose of communicating findings is to ensure accountability and motivate stakeholders to action. Draft M&E findings need to be discussed with implementing partners and primary stakeholders in order to get feedback on accuracy, reach joint conclusions and agree on next steps. Final findings can then be passed to the relevant organisations for accountability and action.

Plan carefully how you will communicate your M&E findings. Reach agreement with project stakeholders on who needs to receive what kinds of M&E information. Remember to include accountability, advocacy and action-oriented audiences and to agree on the information (content and form) they need.

Plan for communication as part of your M&E system from the outset. Do not hope or expect that someone else in the project will communicate M&E findings. As part of this, invest in good communication, not only in producing effective outputs but also in project-based capacities for communication.

A key communication task is to ensure that your findings are correct. Workshops and meetings are critical events to seek feedback and plan action.

When planning to present M&E information for feedback, consider these practical aspects:

• Ensure clarity of message for specific audiences.
• Agree on the frequency for communicating information.
• Ensure timeliness. When do you need to get feedback to still be useful for decision making?
• Consider location. Where will people feel at ease?

Use different media to communicate findings. Written reporting is most known and ranges from formal progress reports, to special studies, to informal briefs in the form of memorandums highlighting a current issue. M&E findings can often be communicated more effectively verbally than by other means. Speaking directly with a target audience provides a quicker and more flexible way to convey your message. Also use visual displays, such as graphs or charts showing trends or maps, to convey summaries of what is happening.

Back to Top

6.2 Deciding Which Methods to Use

6.2.1 What Are Methods?

A method is an established and systematic way of carrying out a particular task. Agronomists have methods for measuring crop yield. Economists have methods for calculating return on investment. Anthropologists have methods for looking at household decision-making patterns. Accountants have methods for budgeting and reporting on project funds. And managers and facilitators have methods for helping groups to make decisions.

M&E makes use of a wide range of methods for gathering, analysing, storing and presenting information. In your M&E activities, you are likely to use established research methods from the biophysical and social sciences, as well as from a growing collection of participatory methods (see Box 6-2). Sometimes the information you require will make it necessary to adapt an existing method or develop an entirely new method.

In carrying out M&E, it is often necessary to combine a series of methods (see Box 6-3). For example, a participatory rural appraisal (PRA) process used to find out how primary stakeholders are benefiting from a project might combine some 15 or more different methods ranging from transect walks to matrix ranking and focus group discussions. Likewise, a household survey or annual project review meeting would combine a series of interviewing, discussion and facilitation methods. The combination of a series of methods in a structured way is often referred to as a methodology. For example, you have a methodology for a workshop or a methodology for a baseline survey.

6.2.2 Types of Methods

Annex D provides a description of 34 different methods commonly used for M&E and, in particular, participatory M&E. They have been grouped as follows:

• sampling methods;
• core M&E methods (such as stakeholder analysis and questionnaires);
• discussion methods for groups (such as brainstorming and role plays);
• methods for spatially-distributed information (such as maps and transects);
• methods for time-based patterns of change (such as diaries and photographs);
• methods for analysing relationships and linkages (such as impact flow diagrams and problem trees);
• methods for ranking and prioritising (such as matrices).

You will probably also need to draw on other specialised methods related to specific technical fields, which are clustered under biophysical measurements (Method 5) and cost-benefit analysis (Method 7) in Annex D.

By calling on specific technical expertise when developing a detailed M&E plan, you can ensure the inclusion of appropriate specialist methods. Before selecting your methods, first consider three important aspects:

• quantitative versus qualitative methods (see Table 6-2);
• individual versus group-based methods (see Table 6-2);
• the extent to which a method can be participatory.

Table 6-2. Examples of multi-purpose M&E methods

Quantitative and Qualitative Methods

Quantitative methods directly measure the status or change of a specific variable, for example, changes in crop yield, kilometres of road built or hours women spend fetching water. Quantitative methods provide direct numerical results.

Qualitative methods gather information by asking people to explain what they have observed, do, believe or feel. The output from qualitative methods is textual descriptions.

Much information in M&E reports tends to be based on numbers. Quantitative data are clear and precise and are often considered to be more scientifically verifiable. You will always need this kind of information. However, for some performance questions you will need to complement it by asking people about their experiences and opinions.

Choosing to use a method to produce or analyse qualitative or quantitative data (see Box 6-4) depends not only on the type of information you are seeking but also on the capacities and resources you have available, how the information will be used and how precise data need to be (see Box 6-5).

Note that the difference between quantitative and qualitative methods is not absolute. Much qualitative information can be quantified. For example, opinions can be clustered into groups and then counted, thereby becoming quantitative. Note, however, that you can never make quantitative information more qualitative. You cannot extract an opinion from a number.

Considering Individual- or Group-Based Methods

Throughout the M&E process � from design, to data collection and analysis � you can choose to use methods to consult with groups or with individuals (see Table 6-3). Working with individuals can give you more detailed information but it will only give an overview after analysing data from a set of individuals. A group-based method will elicit a more collective perspective � with areas of consensus and divergence � while personal details and perspectives are less likely to emerge. Groups ask more of the facilitator and the quality of discussions depends on the size of the group and how comfortable people are with each other and the topic at hand. Annex D includes one cluster of methods that are particularly suited for group discussions. However, many other methods in Annex D can also be used in a group context (see Table 6-2).

The more people involved at any one M&E event, the greater the importance of good facilitation and planning. The facilitator�s skill will largely determine whether a method is used successfully in a group. Good facilitators will provide suggestions, probe, encourage, redirect and also take notes. They also help manage conflicts by encouraging people to listen to and understand each other�s perspectives.

Table 6-3. Pros and cons of working with individuals and groups

What Makes a Method Participatory

Many projects are keen to involve primary stakeholders more in M&E. They commonly consider that collecting data from local people using so-called participatory methods is sufficient. Imagine the following scenario. The M&E staff of a project goes to a group of farms to understand if soil nutrient flows have changed as a result of farmer training on soil conservation. They meet the farmers and ask them to sketch maps showing where nutrients enter the farms, how they are used and how they leave, and in particular showing what has changed after soil conservation measures were adopted. The mapping process lasts about two hours, after which the team goes back to the M&E office with the sketched maps to synthesise and analyse the data for a report to the director. At some point, the report is copied and sent to the village. Can you call this mapping process participatory?

Participation in M&E is often limited to working with primary stakeholders as information sources, rather than as joint users of information and therefore potential analysts and co-designers of methods. If you have selected the method and use it to get information from people, then you are involved not in a participatory process but in an extractive one. This is fine � unless you are aiming for participatory M&E. In which case, you would involve other stakeholders in choosing and using methods.

Many people think there is a set of so-called "participatory" M&E methods, but this is not the case. A method is not inherently participatory or not participatory. Many of the methods useful for M&E can be used in either a participatory or non-participatory way. The participatory impact comes with the way a method is used and who helped select it. The use of a technical method for testing water quality, for example, can become participatory if the community is involved in deciding what aspects of water quality to measure, collecting the data and reviewing the results. On the other hand, if a group is directed to produce a map of the area, there is little discussion, and the map disappears into the project office forever, then this cannot be called participatory mapping. See 2.6 for general considerations for participatory M&E.

To ensure that the selection and use of methods is participatory, consider these questions.

1. In what aspect of the M&E methods is participation important? In selection or design of the method, in applying it for data collection or for analysis?
2. Who should ideally be involved in the task at hand? Who needs to help select, design or use the method? Ideally, those who are to use it for collecting or analysis should be involved in selection/design. This can include staff of implementing partners, project staff, primary stakeholders and consultants.
3. Who wants to be involved in what? Not everyone has the time or inclination to participate. This is not a problem, as full participation is neither practical nor possible. Instead, you need to ask those you would like to involve if they are able and interested.
4. What is needed for effective participation? Self-confidence is needed before effective participation is possible. Therefore you need to create the conditions for people to feel free in helping define methods, in testing and adjusting them, in collecting data, etc. This can include providing training or follow-up mentoring, finding the right time and place, offering childcare support, etc.

6.2.3 Selecting Your Methods

To select the most appropriate methods for the task at hand, the steps below can give some guidance.

1. Be clear about what you need to know. Section 5 discusses the process of deciding what you want to monitor and evaluate. Before you start with method selection, confirm with those involved that everyone is clear on what information needs to be sought.
2. Check that another group, person or organisation is not already collecting the data. Before investing in method selection for data gathering and analysis, find out if the information you are seeking is already available and from where (see Table 6.3). Government agencies, universities and research organisations will often have data that can contribute to the project�s information needs. Start by asking whether there are reporting mechanisms in the villages, local towns, district capitals, etc. for information you might need, such as population, disease incidence, tax collection and so on. The methods employed will be many and varied, ranging from national statistical and census methods to specific research methods. You might find it helpful to make an inventory of existing information collection, as in an IFAD-supported project in Zambia (see Table 6-4).

Check, where possible, how the information was collected to see if it is reliable enough for your needs. In some situations it may be possible to modify data gathering by other agencies to better support the M&E work of the project. However, if you think the data quality cannot be improved or if they are too difficult to access, then you will need to consider collecting the data yourself.

Table 6-4. Part of an inventory of information useful for the District Development Project that is already being compiled in Zambia³

3. Be clear about how accurate you need to be. Higher accuracy is always more desirable than lower accuracy. However, in some cases you may not need precise figures or detailed opinions based on a representative sample, but only a general impression. For example, you can choose to do a series of 50 measurements on farmers� fields to measure exact productivity. But you might only need to know if most farmers are satisfied with their yields, for which discussion with several farmer leaders might be sufficient.
4. Does the information relate to a specialist area? If so, seek specialist advice or detailed documentation before proceeding with the method selection. This is the case, for example, for cost-benefit analysis and geographic information system mapping (see Methods 7 and 19 in Annex D). They require expert input in order to assess if they are worthwhile for the project to use.
5. Be clear about the task that needs to be accomplished and whether it concerns qualitative and/or quantitative information. Consider whether a method is needed to collect, collate, analyse, synthesise or disseminate information. Does the performance question or indicator for which you are seeking a method require quantitative, qualitative or both types of information? Think about whether you need individual or group opinions. Also consider how the people involved prefer and are able to communicate, as this determines the choice of medium: written, oral, visual and/or dramatic. Some methods are based on diagrams, while others focus on written information.
6. Decide the extent to which the data gathering or analysis process is to be participatory and, therefore, whether you need to work with individuals, groups or a combination. Different stakeholders can be involved in data gathering and analysis of information to varying degrees. Be clear about why you are seeking more participation (see Box 6-6). Is it for consistency in processing or for shared analysis? This will affect the choice of method. The extent of participation will also influence the suitability of certain methods. For example, a cost-benefit analysis is not suited for just anyone, but for someone with an economic background. If you are developing an M&E system that micro-credit groups are to implement and manage, then questionnaires will only be suitable if they design this themselves and are confident about analysing the results.

• Decide if your data collection coverage is to be sampled or comprehensive. If working with a sample, decide upon your sample size and then choose an appropriate sampling method. It is often not possible to gather data from the entire population that interests you. Instead, a sample will be needed if the population is too large, time is limited or you face resource and capacity constraints. Select your sample well, as it affects which methods are appropriate and feasible, and will affect the validity of your findings.
• Decide on your sample size. The optimal size of your sample has little to do with the size of the population you are studying. Other factors are more important, such as the available budget, number of subgroups to be analysed and time available. See D.1 for more details.
• Clarify the "sampling frame". This refers to a description of the set of all possible individuals you could sample. See D.1 for more details.
• Select your sample. You have two options (see Methods 1 and 2 in Annex D). Random sampling methods give every individual in a population an equal chance of being selected. Non-random sampling methods involve a more deliberate selection within a population, particularly when a certain kind of opinion or comparison is needed. Or you can combine these two options (see Box 6-7).

8. Do you have several methodological options or is there only one? Armed with all these details about how you hope to find information, ask yourselves if you actually have any options. Sometimes the type of information you are seeking can be found clearly only in one way. For example, knowing how many turtles have laid eggs on breeding beaches will require you to go and look. However, it is more likely that you will have several options.
9. List your method options and make an initial selection. Once you know what the method needs to do, then it is time to list all options and choose. Table 6-5 provides one way to help you organise your thinking for this step.

Table 6-5. Helping you match methods for performance questions and indicators

Selection of your method will depend largely on the type of information needed, the skills of those involved and the degree of precision needed. Also make sure that methods complement each other to provide the information you are seeking and that they allow you to crosscheck information. For example, a forestry resource management plan may involve GIS maps (Method 19), resource mapping (Method 17) and transects (Method 18) to gather information on the forest resources, an analysis of historical trends to understand changes in forest use and ownership, an institutional analysis diagram (Method 27) to help with stakeholder analysis and various discussion methods (Methods 11 to 16) to understand local priorities and dynamics.

Critical in your selection process is ensuring appropriateness. Table 6-6 provides an example of the appropriateness of different soil-erosion assessment methods for different audiences. Especially in the case of participatory monitoring, methods should be selected so they can eventually be incorporated into everyone�s everyday activities, as few people are likely to be remunerated for the effort involved. Methods might need to be created after negotiations about appropriateness (see Box 6-8). Where possible, the information collection, analysis and the use of the results should be undertaken by the same people, who should understand the method(s) and agree that they are appropriate.

Table 6-6. Appropriateness of soil-erosion assessment methods for different stakeholder groups ⁶

10. Use this checklist to see if you have the right method for the task at hand.⁸

• Feasibility. Do you have the right skills and equipment for the method? Can the method realistically help cover the intended questions/indicators? Do you have enough time? Can you cover the geographic area adequately? What is the distance between participants and what are the language requirements? Are sufficient technical support and training provided?
• Appropriateness. Does the method suit the conditions of the project? Does everyone involved agree that the method is appropriate and do they understand it? Is the unit of analysis appropriate for the method?
• Validity. Do the people who are to use the information believe the method is valid, i.e., able to assess the desired indicator with enough accuracy?
• Reliability. Will the method work when needed? Is the error that will occur acceptable? Are you using different methods to verify the information collected, rather than using only one particular method so risking distorted information?
• Relevance. Does the method produce the information required or is it actually assessing something similar but, in fact, basically different? Does the method complement the basic philosophy and approach of the project?
• Sensitivity. Is it able to pick up data variations sufficiently? Can it be adapted to changing conditions without excessive loss of reliability?
• Cost effectiveness. Have sufficient financial resources been allocated? Will the method produce useful information at relatively low cost � or is there a cheaper alternative that provides information that is good enough?
• Timeliness. Is there an acceptable level of delay between information collection, analysis and use? Do the methods use the least amount of time possible outside of everyday work? Have you looked for ways to incorporate the use of the methods into other daily tasks?

11. Pre-test your method. You should pre-test all M&E methods to make sure they are feasible and will give you the desired kind of information. Pre-testing is particularly critical prior to a major data-gathering exercise. It involves a trial run with a small number of participants who are similar to those from whom information is to be sought. Check that the questions are clear and see how long the method takes per person or group. Adjust your method based on the outcome of the test run. You might need to organise additional training if the method seems to require more skills than those possessed by the people who are to use it.
12. Determine the frequency of use. Monitoring implies repeated use of a method to make comparisons, for example, returning to a map (Method 17) every six months to update the information or holding a focus group (Method 12) to see if views have changed. Methods need to be consistently applied at each monitoring moment so that information is not distorted, comparisons are possible and findings are reliable.

Back to Top

6.3 Gathering Data from the Field

6.3.1 Preparing and Planning for Data Collection

After selecting and pre-testing the method � but before starting the data collection � you will need to make the final preparations. Consider what you might need to do to limit common problems in the field.

Consider carefully how to select interviewers and facilitators. Two types of fieldworkers will be needed: interviewers to collect data and facilitators to conduct group-based discussions and analysis. Interviewing and facilitating are two sets of complementary skills. Consider whether the following factors may influence the quality of interviews and discussions: age, gender, background and position in the community, educational level, socio-economic level, personality and attitude, physical health, language, religion and cultural customs. These factors may impair or enhance an interviewer or facilitator�s capacity to understand certain topics or be acceptable to whomever he/she is meeting. Select those people who fit best with the task at hand and the stakeholders with whom they will interact.

Consider how to distribute the tasks of collection and analysis among different people and what is needed to limit errors. The number of people involved in each stage of the data journey will affect the consistency and accuracy of data. The greater the number of people involved, not only the more organisation is needed but also the greater the risk of data inaccuracies and inconsistencies. Plan how you will ensure that fieldworkers achieve consistent quality of data collection/facilitation and how data will be verified (see 6.3.2).

Ensure that those using the methods are comfortable with them. Each method should be pre-tested and practised by individuals who are to apply them. Facilitation techniques need to be mastered by those who will interact with stakeholders to collect and analyse information. This means understanding and practising facilitation techniques but also having the skills to design methods jointly with stakeholders. A training session on methods needs to cover the purpose of each method and of data collection and analysis, improve the specific skills for working with groups and doing good interviewing, and teach ways to record information.

Ensure clarity of language. Ideally, field workers either speak the relevant language or are accompanied by a trusted interpreter. If working through translation, spend time getting the translations right with native speakers and, if possible, train the translators in the selected M&E methods. A list of clear translations needs to be prepared before the fieldwork starts. One way of ensuring that an unusual method, such as matrix scoring (Method 32, Annex D), is translated correctly is by having one native speaker translate it and then asking another person to translate it back to the original language. Then the two versions can be discussed with the data-collectors to be sure they understand and can comment upon the nuances involved.

Prepare each method. Each method will require its own preparations (see Box 6-9). Be sure to organise materials, including sufficient backups of the measuring and recording instruments (pencils or pens for filling in forms or questionnaires, notebooks in which to write, markers for flip charts, batteries for a laptop computer or tape recorder, etc.). Carefully plan the formats needed to record information (see 6.3.3)

6.3.2 Ensuring Reliability of Information

Reliability of information is about consistency. To increase the reliability of information, stop to consider possible causes of inconsistency. Errors creep into the system when, for example, field staff document answers inaccurately, selected respondents are not the best information sources, field staff are unclear about the purpose of information gathering, etc. Two basic types of data errors are "sampling errors" and "non-sampling errors".

A sampling error occurs when you have chosen the wrong sample (see 6.2.3 and D.1, Annex D). It is the difference between an estimate derived from a sample survey and the value that would result if a census of the whole population were taken. For example, if a sample has a response rate of 30%, the sample error estimates how accurately the sample has estimated the 30% of the population that it supposedly represents. Sampling errors arise when the information you have collected does not accurately represent the target population. Casley and Kumar (1988: 81) list the types of households that could be missed when compiling a sample, resulting in data biases: remote or inaccessible households, those with frequently absent members (e.g., migrant labourers), newly created single-person households and ethnic minorities (as they are often marginalised within a village). See D1 for information on how to select a sample. Sampling errors do not occur in a census, for example if you ask all the micro-credit groups the same questions. Because you have involved all of them, you will only have non-sampling errors.

The most common and diverse types of errors are the non-sampling errors. Knowing the possible causes of systematic non-sampling errors can help you limit the error.

• Interviewer bias. An interviewer can unfairly influence the way a respondent answers questions. This may occur if the interviewer or facilitator is too friendly, aloof or prompts the respondent. Fieldworkers need to have adequate capacities but also the right incentives. This can also be caused by a management culture that discourages the reporting of problems such as low levels of implementation (see Section 7 for more on incentives).

• Inadequate methods. Causes include: complicated collection procedures, inappropriate formats, ambiguous questions, mismatch of questions and method, etc.

• Processing errors. These can arise through miscoding, incorrect data entry, incorrect computer programming and inadequate checking.

• Non-response bias. If a significant number of people do not respond to a certain question, then results may be biased because the characteristics of non-respondents may differ from those who have responded. Some questions may be difficult to understand for certain people.

Non-sampling error can occur at any stage of a sample survey or census, and unlike sampling error, it is not generally easy to measure. The non-sampling errors are difficult to measure due to the diversity of sources (the interviewers, respondents, coders, data entry operator, etc.).

Information inaccuracies can have more than one source of error. For example, in a micro-credit project in India, the implementing partners felt that data collected were inaccurate due in part to a burdensome and cumbersome process. The NGOs also questioned the capacity level of local groups to fill out the lengthy monitoring formats accurately. Furthermore, there was very high turnover of grassroots workers, primarily due to very low salaries paid under the programme, so consistency of data collection was bound to suffer. The NGOs feared that if primary data were not accurate, then errors would multiply as the information from the different groups and staff was collated into larger figures, leading to a false picture of the progress and impact.

Avoiding Non-Sampling Errors During Data Collection

Many sources of non-sampling error can be avoided or minimised. Table 6-7 lists some actions you can take to reduce the most common types of errors.

Table 6-7. Common errors during data collection and how to reduce them

Verifying the Data Once You Have Them

Data must, from time to time, be verified. Only by checking whether your data make sense and are valid can you feel safe that you are analysing progress and process based on correct inputs. You do not have to check data all the time. Keep your data verification process efficient by undertaking spot checks at key moments:

• at the beginning of any project � if you are using existing data sets � by looking at where data come from, who has collected information and the methods and standards they used;
• when using a new method;
• when targets and data match perfectly;
• when working with new fieldworkers, new implementing partners, new staff, etc.

When all goes too smoothly with data collection, then probe to see if there really are no problems lurking underneath the surface. Problems are inevitable and their absence may signal that problems are suppressed. Keep an eye out for signs of problematic data and investigate where problems might be occurring.

• Overly accurate data. When the data collected match targets too perfectly the data are probably problematic. In one IFAD-supported project in Asia, large variations emerged in reporting per county. Most counties consider the targets written in the appraisal report as compulsory and strive to achieve them. They only report when achievements are close to 100% of the targets. For instance, in two counties, the 1996 performance records a 100% achievement for practically all activities. In another project, a review in 2001 of the data on physical progress showed that targets and actual figures of implementation were exactly the same, every month, for every parameter. These are clear cases of unreliable data.
• Sudden, large changes in data. In northeast Brazil, an NGO was monitoring the adoption rates of contour ploughing and noticed a huge increase in adoption rates. The NGO knew it had not undertaken much training with farmers on contour ploughing so doubted the data. Focused research was undertaken in several communities to see if the data were accurate. It turned out that the data were, in fact, accurate but that adoption to contour planting had been triggered by a surge in animal traction. Animals cannot plough up and down steep slopes so contour ploughing had become the side effect of increased use of animal traction. ⁹
• Gaps in the data. When certain information has many non-respondents, this may point to a respondent error or an error in the choice of method for that information.

Options for Verifying Data

Every project needs to find its own way to incorporate verification into its data-collection process. In Yemen, the RADP project deals with data verification when management senses a problem with the data collected by component departments and sent via the M&E unit. Management forms a committee from the department concerned and the M&E unit to verify the information. The department concerned may also make a field visit and submit a report directly to the project director and copies to the M&E unit.

Other projects outsource data verification. In the ADIP project in Bangladesh, the reliability and validity of data are crosschecked using additional data-collection exercises. This includes, for example, the evaluation of demonstration plot performance and research activities by consultants. The responsible governmental department verifies M&E data, but project management decides when such verification will happen and who should carry it out. In the APPTDP project (India), the primary data are collected through village liaison workers. Data are then verified by an appointed agricultural/development consultant. Only then are the verified data passed to the central monitoring unit for analysis.

To check data yourself, triangulation is an important principle. This means collecting the same type of information but from different sources and using different methods. This can be as simple as, for example, asking the same questions with different focus groups or comparing the outputs of a map and a transect of the same area.

Verifying quantitative data is often more straightforward, as more agreed standards exist. For example, many types of biophysical measurements indicate how to calculate whether the data are representative. Verifying qualitative data is more difficult, as there are no clear rules. You can use techniques like "key judges" to verify the interpretation of information (see Box 6-10).

6.3.3. Recording Data

Besides knowing how to conduct interviews and facilitate discussions, fieldworkers need to know how to record responses. Data can be recorded in many ways, depending in large part on the data collection method. Some methods require the filling in of forms or tables, others require using a tape recorder, video recorder or camera, writing answers on cards or flip-charts, or taking detailed notes.

For each bit of information, define how it will be recorded. Practise with the people doing the recording before setting out to collect data.

Whichever data-recording method you choose, make sure you are consistent in how you record or it will be difficult to compare and analyse the data. Also consider the information storage implications (see 6.4.4 for more details). Where and how will data be stored so that they are safe and accessible? This will affect how data are recorded. Box 6-11 describes one example of the daily recording of information that can then be fed into reports on the progress of the project.

A good form helps the recorder enter data consistently. It should clearly represent the selected M&E indicators (as words, as diagrams or symbols, or reformulated as a question) and give sufficient space for the collector to fill in the information. Data forms should include space at least for:

• date, location, time and duration of interview or discussion;
• name of enumerator/facilitator;
• name of participants;
• topic(s) being discussed and methods used;
• key findings � either in a predefined format (see Box 6-12) or in terms of key words and descriptions if the data gathering method is open ended.

When diagrams form the basis of discussion (see D.4, D.5, D.6 and D.7), extra care is needed to make additional notes since the diagram itself will never capture all the important opinions and conclusions.

Who designs the recording form is critical, particularly if monitoring is to be carried out by local groups. For example, self-help groups in a credit project in India have developed a coding system to ensure that all their members can participate in their regular self-evaluation process. Since many members are illiterate, the questions are symbolised with pictures and the three levels of evaluation are colour-coded. This is an example of an agreement scale (see Box 6-12).

Back to Top

6.4 Collating, Analysing and Storing Information

Once data have been collected, they need to be organised into a manageable form ready to be analysed efficiently. This involves transcribing data into a systematic format, entering the information obtained from each respondent or group and organising it into one overall format, for instance, into a computer database.

6.4.1 How to Collate Information

Collation of information is needed when:

• you are scaling up your information from a smaller unit of analysis to a larger one, for example, compiling all individual interviews to develop an overview of a micro-credit group or pulling together all village-level information into a district-level analysis;

• information has been collected from different sources with different methods, to lay the basis for making comparisons and finding patterns during analysis.

Collation of information requires an appropriate format. With some methods, this is a very straightforward process. It can simply involve filling in a statistical programme on the computer with numbers that represent measurements or it can entail entering numbers that are pre-identified codes representing specific ideas, following the form, questionnaire or notes used in the data collection process. With statistical data, compilation ensures that the many data are reduced to clearly labelled tables. These tables should integrate the findings according to your performance question. For example, it should show location-specific trends if you are trying to understand how impact varies per community or district.

The collation of qualitative data requires special care and analytical skills (see Box 6-13). Box 6-14 describes the basic steps to order open-ended responses. Section 6.4.2 discusses this in more detail, since data collation and data analysis with qualitative data are overlapping processes.

6.4.2 Why Analyse M&E Information

Analysing M&E findings requires looking closely at the information (ideas, facts, impressions), clarifying and structuring it, understanding connections and identifying core elements, in order to arrive at conclusions that can lead to action. Analysing M&E findings has several functions:

• to refine understanding � by discussing initial information with project stakeholders, more refined insights can emerge;
• to limit biases � ensuring a thorough discussion about information means that this is crosschecked and people can point out when they feel an issue has been represented incorrectly;
• to build a clear picture of a situation/event/process and reach consensus � by discussing data, contradictions and gaps can be identified and can be understood or filled;
• in participatory M&E, joint analysis can strengthen ownership of the conclusions and motivate people to invest more in making changes happen.

Analysis of M&E information and critical reflection are closely related, so please refer to Section 8 for many ideas on how to encourage reflective meetings and analytical reporting.

Consider who needs to be involved in analysis. The question of who is making sense of the data is central to participatory analysis. Often, work that may initially have been very participatory can shift towards analysis only by project staff. Sometimes this is necessary, as some aspects of analysis and synthesis can be excessively tedious or time-consuming for primary stakeholders. Shared analysis can make all the difference between a superficial descriptive report or simplistic feedback session and analysis based on shared understanding that motivates people to action, whether they are villagers, policy makers or technical staff.

Consider how you will undertake analysis. Choosing a method for analysis depends on various factors, including whether it will be a participatory process, the tool you use to collate and analyse the data (e.g., a computer), and the type of information that is being collected. For instance, if it is qualitative information, analysis will involve looking for patterns in descriptions and explanations of patterns (see 6.4.3). For quantitative data, the analysis will follow statistical procedures and show trends in terms of percentages or ratios. In both cases, analysis will involve comparing planned results with actual ones to understand the reasons for differences, to compare differences over geographic ranges or between groups, or simply to monitor changes over time.

Many of the methods in Annex D can be used for data analysis. For instance, if you should choose more participatory processes, see D.3 on discussion methods for more ideas. D.6 on analysing relationships and linkages and D.7 for ranking and prioritising are also useful.

6.4.3 Analysing Quantitative and Qualitative Data¹²

The analysis of quantitative data is often better known in projects than that of qualitative data. Quantitative data analysis often - but not exclusively - encompasses calculations, such as total and average numbers of activities implemented or percentages as compared to plans or targets. More elaborate statistical analysis may also be required, for example with cost-benefit analysis (see Annex D). Discussing the specific procedures of statistical analysis lies beyond the scope of this Guide, so the focus here is on ways to deal with qualitative information.

The analysis of qualitative information is very different and can be more difficult than that of quantitative data for those who are not used to dealing with opinions and non-standard answers. Through content analysis of collected information, conclusions can be formulated for each of the performance questions or indicators. The analysis process involves identifying the categories of responses found in the raw data.

Involve the data collectors in analysis. All M&E data collectors and facilitators � whether they are project staff, implementing partner staff or primary stakeholders � should participate in sessions to analyse qualitative data. Because of the nature of qualitative data, it is critical that those who were present when the data were gathered also participate in analysis. Much happens in open-ended discussions that is observed by facilitators and helps to explain the data.

Collect and analyse qualitative data concurrently. Qualitative data collection is intended to trigger an iterative learning process. This means that information from one discussion or interview will indicate aspects of the topic that you will need to pursue with other questions and methods. So analysis of one set of interview data may indicate changes needed in subsequent interviews or discussions. A second reason for immediate analysis of information is that it is impossible to note everything that is said in open-ended discussions. Additional information, such as about the group dynamics and how they influenced what was said, will not be recorded but are critical to interpreting information. So the sooner the analysis takes place, the easier it is to remember aspects that were not noted.

Structure analysis around each performance question and each category of interviewees. For example, if M&E field staff conducted individual interviews with two farmer leaders and with the village council (VC) in one day, then the two sets of data (farmers and VCs) would be analysed separately. During the analysis, the team may need to refer back to the performance questions to clarify the objectives of the different discussions.

Follow these five steps to analyse the data.

• Re-read the interview questions to the group. This allows everyone to remember the focus of the M&E work.
• The note-taker(s) read aloud the responses for each question. If there is more than one set of notes, each set of notes should be read.
• Discuss the responses and share other comments that may not have been written down, to clarify exactly what the interviewees were saying.
• Cluster the responses and summarise findings. Together, identify the categories of responses in collected information and summarise the findings concisely. The summary should indicate the trends in the information in terms of whether the attitudes or ideas expressed were shared by all interviewees, the majority, half, a minority or only a few. Although you cannot quantify the different types of answers, do report trends.
• Identify unclear or missing information. Determine whether there is missing or unclear information that should be investigated in subsequent M&E work.

6.4.4 Storing M&E Information

Documenting information is critical for M&E, providing the basis for communication, transparency, consensus building and continuity of consultative processes. Stored information serves as the source of institutional memory turned to by newcomers and when verification or comparison with the past is needed. The quantity of information that all projects collect and share calls for information systems to store data and make them accessible to others.

Consider four questions when planning the storage of information (also see 7.5).

1. What information needs to be stored?

Think about what information and how much you need to store. Information storage is needed at two levels: to guide the project strategy and for tracking operations. In principle, everything you decide to monitor and evaluate will need to be stored in some way. Information about progress with implementation, stakeholder reviews, annual project reviews, primary stakeholder databases, changes in the context, causes, unexpected impacts, minutes of meetings� the list quickly becomes overwhelming. Collecting excessive information will also require you to store it (see Box 6-15). Therefore, consider carefully what information needs to pass to whom for decision making and for reporting. Section 5 details how to choose what to monitor and evaluate.

2. Who needs access to the information and when?

How the data are stored depends on who is to have access to the information and how often. Information to guide the project strategy is critical for managers (project staff and implementing partners), steering committees, primary stakeholder representatives and funding agencies. Information on operations is critical for fieldworkers, managers of project components and primary stakeholders.

Consider the skills of the users and the types of communication with which they are comfortable (see Box 6-16).

Only store material where it will be used. This is particularly important with the raw data on paper, such as diagrams. Do not assume that all diagrams need to be copied, distributed and stored at all levels. Only keep them where they are used. This usually means leaving the originals with the stakeholders who produced them.

3. What type of information needs to be stored � hard copies or data that can be computerised and accessed centrally?

The more people who need to use information, the better it is to computerise it. However, not all data gathered at a local level will be entered into a computer. This can be due to local implementing partners and primary stakeholders not having access to computers or electronic networks or lacking the necessary skills, or because the information is diagram-based. Diagrams can be (photo-) copied and distributed to those who will need access in that form, for example, local groups and community-based facilitators. Generally, however, you will only need short reports that summarise the findings from the discussions that occurred as the diagrams were generated and from the diagrams themselves.

4. Regularly assess what information you need to keep and what can be discarded.

A data-storage system will soon get congested and overflow if it is not updated regularly. This is as true for archives of hard copies as it is for computerised data. Computerised data are more easily archived in unobtrusive yet accessible ways. Simply make backups and store them in a safe place away from the hard disk.

For hard copies, making decisions about what to discard is more critical. Make sure that you keep all material you are legally required to store, such as tax and audit-related financial records, for the required time period. This will vary per country. Also make sure that you keep copies of all material you need for making comparisons of change over time. This includes baseline data, summaries of progress with implementation and interim impact information.

Back to Top

6.5 Communicating M&E Findings for Action and Accountability

6.5.1 Why Communicate M&E Findings

M&E-related findings have many potential audiences. When reporting on progress with the AWPB, you will direct yourself to funding agencies, steering committees, cooperating institutions and implementing partners. Primary stakeholders have a right to knowing overall how the project is progressing and they deserve the opportunity to react to initial findings. Funding agencies and managers need information on impact, while all implementing partners need to understand problems in order to find solutions.

Two sets of M&E findings will need to be communicated. First, it is good practice to discuss draft M&E findings with implementing partners and primary stakeholders in order to get feedback on accuracy, reach joint conclusions and agree on next steps. Once the M&E findings are agreed upon, these can be communicated to funding agencies, cooperating institutions, government departments and other projects. This second set of final findings will fulfil accountability needs but can also serve for advocacy purposes.

6.5.2 Planning How to Communicate M&E Findings

Know Your Audiences

Reach agreement with project stakeholders on who needs to receive what kinds of M&E information. Table 6-8 shows the information needs of different audiences for a World Food Programme project in China. It outlines what data and insights the M&E system must produce and for whom. Note that it focuses on communicating for accountability and not on communication for action and decision-making purposes.

When undertaking an audience analysis for your project, remember to:

• include accountability, advocacy and action-oriented audiences;
• define what you expect from the audiences by communicating with them (financial support, commitment to action, etc);
• agree on the information (content and form) they need in order to achieve your purpose.

Table 6-8. Audiences for information on a WFP project in China ¹³ (high/medium/low priority)

• Ensure clarity of message for specific audiences. The interests and concerns of different audiences vary and will require adapted reports, both in terms of content and language. Reports should communicate different levels of detail according to the audience being addressed. For example, strategic and implementation levels of management require different focuses. At the strategic level, you need to provide a general review of the project�s progress and problems. At the implementation level, more detail is required to help facilitate and coordinate day-to-day project management tasks.
• Agree on the frequency for communicating information. This will often fit the timing of decision-making meetings. If you are holding a meeting in order to seek immediate feedback, choose a time when people will be able to come.
• Ensure timeliness. Be sure to present information while there is still momentum, in order to benefit from the feedback. However, if setbacks should take place, be sure to let the audience know and be clear about the delay involved. This issue is not only important for getting feedback, but also for maintaining project credibility.
• Consider location. Box 6-21 shows the importance of thinking about various conduits of information to be sure that how and where you share your findings will be able to reach people, providing them the opportunity to give feedback.

• Make effective use of graphic information to facilitate analysis. Visually presented information is often easier to understand. The better and more quickly your information is understood, the more likely you will get direct, useful feedback. There are many ways to present your information pictorially: through the use of graphs, diagrams, maps, pictures, photographs or videos. Some of these presentation forms will arise naturally as a result of your choice of data-gathering method. For instance, by showing the results of a series of mapping exercises or photographs, people can see at a glance what has been measured, how and how it has changed. Other visual portrayals, such as graphs or pie charts, need to be created from the information obtained through statistical data analysis.
• Keep focused on your task. A feedback session can strand in a general talking event with no clear outcomes. Plan the event carefully around the anticipated outputs � e.g. clarifications, additional insights, conclusions, action steps, etc. Don�t rely on improvisation as your main facilitation strategy. It is always necessary but too much can lead to confusion. Avoid imposing ideas by thinking how people are most likely to share their thoughts on the M&E data. Be sincere in the reporting � include the new insights, otherwise participation will become a farce.

1/ UNDP-World Bank Water and Sanitation Program - South Asia. 1999. Sustainability Monitoring: the VIP Way. A Ground-level Exercise. UNDP-World Bank.

2/ Gosling, L. with Edwards, M. 1995. "Toolkits: A Practical Guide to Assessment, Monitoring, Review and Evaluation". Save the Children Development Manual 5. London: Save the Children.

3/ Johnson, D. with M. W. Kanyinji, R. Lupenga, G.L.E. Nkhata, B. K. Siantumbu and S.M.H. Sithole. 1998. Communication Strategy and Monitoring and Evaluation Framework: Main Report. Zambia: District Development Project, Government of the Republic of Zambia and the United Nations Capital Development Fund.

4/ Westley, K. "Lessons Learned from Different CARE Country Offices in Design, Monitoring and Evaluation". Mimeograph. London: CARE-UK.

5/ IFAD, ANGOC and IIRR 2001, 240, see Further Reading.

6/ Note that "appropriateness" is not defined in this example. Herweg, K., K. Steiner and J. Slaats (eds). 1998. Sustainable Land Management Guidelines for Impact Monitoring: Workbook and Toolkit. Page 65. Berne: Centre for Development and Environment.

7/ Abbot, J. and Guijt, I. 1998. "Changing Views on Change: Participatory Approaches to Monitoring the Environment". SARL Discussion Paper 2. London: IIED. Download

8/ Inspired by: Mikkelsen, B. 1995. Methods for Development Work and Research: A Guide for Practitioners. New Delhi, London, and Thousand Oaks: Sage Publications. and Rugh, J. 1986. Self Evaluation - Ideas for Participatory Evaluation of Rural Community Development Projects. Oklahoma City: World Neighbors.

9/ Sidersky, P. and Guijt, I. 2000. "Experimenting with Participatory Monitoring in Northeast Brazil: The case of AS-PTA's Projeto Paraíba". In: M. Estrella et al. (eds). 2000. Learning from Change: Issues and Experiences in Participatory Monitoring and Evaluation. London: Intermediate Technology Publications.

10/ Abes, R.V. 2000. "Strengthening citizen participation and democratisation in the Philippines: ELF's impact evaluation." In: M. Estrella et al. (eds). 2000. Learning From Change: Issues and Experiences in Participatory Monitoring and Evaluation. London: Intermediate Technology Publications.

11/ Feuerstein, M. T. 1986, 95-102, see Further Reading.

12/ Based on Aubel, J. 2000, see Further Reading.

13/ Hereward, M. and Siping, W. 2000. "Bringing results to the people". Presentation at M&E Workshop Wuhan, 9 - 12 October, 2000. UNICEF Office for China.

14/ Modified from FAO, "Notes on Rapid Appraisal Methodologies for Assessing Impact". Download

Download PDF Version (259 KB)

Back