In the case of sustainability assessment of policy, concept validity seems more appropriate. This validity concerns the degree to which any assessment device measures what it purports to measure. The central question of the validity is whether the inclusion of created or chosen instruments compares the desired concept or not. Two steps may be needed to determine these forms of validity:

1.Conceptualizing different assessment areas to measure the concept.

2.Finding their central dimension.

For example, sustainability assessment of environmental role of forest land use can be assessed on the basis of existing forest area and/or reduction of water flow. Determining the central dimension (usually by factor or cluster analysis), which in this case may be called derived dimension, is also important. Each variable under consideration is then correlated with that derived dimension. If there is no central dimension, several independent measures of the concept could be used. Reliability is also another important concept needing to be determined, which usually deals with the instruments used for measuring a policy. As in this case, there may not be any mechanical instrument for measurement, more specifically a measure’s reliability is the degree to which the same scores would be obtained if it were possible to repeat assessment procedures.

9.2 PARTICIPATION EVALUATION

Participation evaluation is one of the important phases of assessing the social dimension of a policy. Participation evaluation is targeted to assess the potential and performance of policy actors. However, to identify the potential of the actors, participation evaluation also includes the definition and identification of problem and evaluates the constraints and opportunities of the system. The roles and potentials of actors are clarified on the basis of constraints and opportunities the actors enjoy within the policy. Those potentials are used as a background scale for measuring the performance of the actors. Different aspects of participation evaluation are presented in the Rapid Appraisal of Agricultural Knowledge System (RAAKS) model (Fig. 9.1). Using the guideline shown in the RAAKS model, only the participatory aspect of a policy evaluation may be possible. The model can help in building up the information base, but it helps little about the physical resource base. Thus, the policy evaluations need to go beyond the model. There are some limitations

•Marxist model, and

•Corporatist model.

These policy models are not mutually exclusive. In fact more than one model could be used for policy formulation and it is not impossible that a different combination may be needed for a different society and each society may have a different process model. The institutional framework engaged in the policy formulation might have an effect on the policy output and such differences need to be considered during policy evaluation. The best way to achieve a better result is to consider policy implementation as a part of the policy process. Barrett and Fudge (1981) wrote:

it is essential to look at implementation not solely in terms of putting policy into effect but also in terms of observing what actually happens or gets done and seeking to understand how and why.

When a policy is about resources, some negotiation is usually needed between different actors and agents, as each of them tends to maximize their benefit. The political process, by which the policy is mediated, negotiated, and modified during its formulation and legitimization, does not stop when initial policy decisions are made but continues to influence policy through the behavior of those responsible for implementation and those affected by the policy actions for protecting their own interest. This view of implementation takes away the traditional approach on formal organizational hierarchies, communication, and control mechanisms and places more emphasis on:

•the multiplicity of actors and agencies involved and the variety of linkages between them,

•their value systems, interests, relative autonomies, and power bases, and

•the interaction that takes place between the actors and the agencies.

The identification of key stages in policy implementation is a crucial procedure in evaluation and extraordinarily it is one that is neglected in official programs of evaluation. The identification of stages depends on the type and nature of the policy in consideration, but five key elements have been suggested by Winter (1996):

1.Publicity—potential recipients know about the policy scheme through publicity. The evaluation identifies whether there was any bias in publicity or whether some recipients heard about the scheme before the others.

114 Sustainability Assessment

2.Advice/extension—recipients are made involved in the scheme through advice/extension. The evaluation deals with whether sufficient advice was available to recipients or not. Whether the advice was accurate or unduly optimistic or pessimistic. Whether there was any bias in the advice so that some recipients are more or less likely to respond than others.

3.Participation in the scheme—identifies how the recipients are involved in the policy scheme. The evaluation delineates the administrative ease to embark on the scheme, whether any financial problem exists which may exclude the target groups and once started whether the scheme was managed and administered efficiently.

4.Monitoring and adaptation—identifies the attempts for solving subsequent problems. The evaluation tries to define the kind of monitoring action, how and for what objectives the monitoring action was done and whether the monitoring led to any modifications of the policy.

5.End point—determines whether the end point is spatially and temporally logical to the goal of the policy.

However, there is a limit to the ability to evaluate the roles of players through a process model. In the case of sustainability assessment of policies, such evaluation is generally retrospective (not progressive).

9.4 RETROSPECTIVE POLICY EVALUATION

Retrospective evaluation is different from progressive evaluation. In progressive evaluation, goals and process are given, evaluation tries to track whether the process going to achieve the goals but in retrospective evaluation status of achievement is given but there is a need to establish the goal and processes to explain why the achievement was poor or perfect. Other application of social science theory, methods, and techniques to identify and assess the processes and impacts of governmental policies and programs may remain similar. Possible steps of retrospective policy evaluation are presented in Table 9.3.

In assessing the sustainability perspectives of policies, retrospective evaluation is far and away one of the most important aspects of outlining actor relationship. Hoggwood and Gunn (1984) identified five main categories of technique for retrospective evaluation:

1.evaluating the situation before and after implementation,

2.modeling,

3.experimental method,

4.quasi-experimental methods, and

5.retrospective cost benefit analysis.

“Before and after implementation” might appear to offer the most straightforward approaches to assessing the achievements of a policy, but in practice they present considerable difficulties. It is rarely possible, for example, for the policy evaluator to be farsighted enough to establish the full information prior to introducing a policy scheme. Even if information is available, difficult questions may remain whether apparent policy achievements are exclusively the consequence of a particular policy.

In many ways, modeling is a refinement of “before and after” studies. Models attempt to incorporate systematically all those factors that may have influenced a policy outcome. Lack of suitable data and erroneous assumptions about causal relationships provide a formidable challenge to those wishing to evaluate policy through modeling.

As the name implies, the experimental method relies on testing a policy on a particular group of people, ideally retaining a control group with identical characteristics with the experimental group on which the policy is not applied. The experimental method in policy