Prioritizing and Selecting Vital Signs -
What Should be Monitored?
The task of selecting a relatively small set of measurements for a national park
that "represent the overall health or condition of park resources, known or
hypothesized effects of stressors, or elements that have important human values"
is not a trivial task. It is relatively easy to generate a list of potential
monitoring projects or indicators to address a park's most critical data needs,
but the process of paring the list down to a few "vital signs" that best represent
the composition, structure, and function of the larger ecosystem is very challenging.
There is no tried and true method for developing and prioritizing a list of
potential vital signs. Many different approaches for developing and evaluating
potential indicators have been used by various monitoring programs, and there
are numerous sets of criteria for the 'ideal indicator'. A number of these approaches
have been summarized in the document
Example Criteria and Methodologies for Prioritizing Indicators
Most networks of parks are following the basic approach shown in the figure below
to identify and prioritize potential vital signs. The scoping process usually involves
a series of meetings, workshops, brainstorming sessions, questionnaires, literature
reviews, and other information-gathering exercises to identify monitoring questions
and data needs that include (1) focal resources (including ecological processes)
important to each park, (2) agents of change or stressors that are known or
suspected to cause changes in the focal resources over time; and (3) some basic
key properties and processes of ecosystem health (e.g., weather, soil nutrients).
Conceptual models are then developed to help organize and communicate the information
compiled during scoping, and to identify where cause-effect is known between
some of the stressors and response variables. The scoping and conceptual modeling
efforts will result in a list of potential vital signs, which must then be prioritized
using some set of criteria agreed upon by the parks. Many networks have generated
a separate prioritized list for each park, plus a "network list" that combines the
rankings from each park with the number of parks that have identified a particular
vital sign as a high priority. The final step in the process is using the prioritized
lists in combination with other criteria such as efficient use of personnel, cost and
logistical feasibility, partnership opportunities with other programs, and a large
dose of common sense to SELECT the set of vital signs that will be included
in the initial monitoring program. Thus, it is recommended that PRIORITIZATION
and SELECTION of vital signs be treated as two separate steps in the process.
Basic approach to identifying and selecting vital signs for integrated monitoring
of park resources (source: Kurt Jenkins, USGS Olympic Field Station).
The process of prioritizing the list of potential vital signs for a park network
will involve a group decision that involves many different parks, individuals,
and disciplines. A structured group decision-making process should be used to
take all of the information and ideas available, and then produce judgments,
manage conflict, and enable consensus. Several approaches to group decision-making
are summarized below. More detail can be found in the document
Group Decision-Making Processes used
to Prioritize Vital Signs
- BOGSAT (Bunch of guys/gals sitting around a table). Not Recommended.
This is a term coined by Schmoldt and Peterson (2000); it is often used
in workshop 'scoping meeting' settings, and is the most common and often
least effective approach to group decision-making. Group deficiencies related
to the BOGSAT or workshop setting have received the unenviable names of
"social-loafing" and "group-think", and are often the result of member shyness
or alternatively individual dominance, lack of communication skills, social
pressure to conform and personality conflicts or uncooperative individuals.
The consequences are often "an abundance of unfocussed and rambling discussion,
which mixes judgmental and intellective issues, ... and a cost in inefficiencies
of time and effort and the loss of ideas introduced in the wrong context"
(Schmoldt and Peterson, 2000, p.
64). [quoted from Oliver (2002), Ecological Indicators 27:1-15.]
- Delphi. Recommended for brainstorming, but not for prioritization/selection
of indicators. The Delphi approach, especially when used over the internet,
provides an ideal vehicle for rapidly and efficiently drawing together expert
knowledge and opinion on complex issues faced by natural resource managers
(Oliver 2002). "Delphi may be characterized as a method for structuring
a group communication process, so that the process is effective in allowing
a group of individuals, as a whole, to deal with complex problems". It is
based on the premises that: (1) opinions of experts are justified as inputs
to decision-making where absolute answers are unknown; and (2) a consensus
of experts will provide a more accurate response to a question than a single
expert. The objective of the Delphi approach is to generate many ideas,
initially, in the absence of evaluation. It usually involves questionnaires
to which each expert responds anonymously. These are returned to participants
for revision accompanied by feedback which summarizes all responses. This
iterative process may continue until convergence of opinion is reached.
The distinction between Delphi and other GDM processes is that the communication
process operates among panel members dispersed in space and time. Also,
the Delphi approach is commonly applied to large groups (30-100 individuals)
that do not function well in a face-to-face environment. (see references
in the document link above).
- Analytic Hierarchy Process. Recommended, as long as some 'tweaking'
is allowed with the final 'short list'. The analytic hierarchy process (AHP)
is a decision-making framework that uses a hierarchical structure to describe
a problem and paired comparisons to rank decision alternatives with respect
to importance (or preference or likelihood). This technique has been applied
to a wide variety of decision problems. Schmoldt et al. (1994) describe
its use for inventory and monitoring program planning and give an example.
Workshop facilitators and specialized software is
available. See papers by Schmoldt
and Peterson 2000 and Schmoldt
and Peterson 1997).
- Hybrid (simplified version of AHP). Recommended. In this approach,
the park network uses some combination of BOGSAT, questionnaires, DELPHI,
and scoping workshops to brainstorm and produce a list of monitoring questions
and potential indicators. A smaller group of individuals (e.g., technical
committee and Board of Directors) then establishes a set of criteria and
subcriteria (with numerical weights) for ranking the monitoring questions
or indicators, and uses a database in a group setting to prioritize the
potential indicators. The 'short list' might then be adjusted based on expert
opinion and 'common sense' judgment calls to produce the final set of recommended
indicators. An example database in MS Access is available from Steven Fancy.
Recommended Approach for Prioritizing and Selecting Vital Signs
The current, recommended approach for prioritizing vital signs involves the use of an
MS Access database during a workshop setting as a means of focusing discussion and
obtaining advice from experts. This approach was successfully implemented by the
Mojave Desert Network during a May 2004 workshop, and is summarized in the following
Summary of Vital Signs Prioritization Process
for the Mojave Desert Network [1.7 MB .pdf Presentation by Kris Heister,
Mojave Desert Network, at 2005 Annual I&M Meeting]. More recently, the Northern
Great Plains Network has used this database approach during their September 2005
workshop. The basic steps involved in this recommended approach are as follows:
- During Phase 1 of the monitoring planning and design effort, the network coordinator
leads a process involving a series of meetings, workshops, brainstorming sessions,
questionnaires, literature reviews, and other information-gathering exercises to identify
key monitoring questions and data needs. Initial conceptual models are developed to help
organize existing information and knowledge and to promote communication across
disciplines. This process eventually leads to a relatively long list of potential
vital signs for all parks in the network.
- The list of potential vital signs is loaded into the vital signs prioritization
database, which is organized around the
Ecological Monitoring Framework.. For each potential vital sign, the database
includes a justification statement about why the vital sign needs to be monitored, an
initial set of monitoring questions and objectives, which parks the vital sign applies
to, and other draft information.
- Vital signs are consistently ranked for each park using a set of criteria that are
applied equally across all parks and disciplines. The 3 criteria used by the Mojave
and other networks are Management Significance, Ecological Significance, and Legal
Mandate. The ranking for Management Significance for each park is done prior to the
workshop by the park managers, and the ranking for Legal Mandate is usually done by
the network coordinator prior to the workshop, and then reviewed by the park managers.
The ranking for Ecological Significance is usually done during the workshop by the
subject-matter experts.
- During the prioritization workshop, subject-matter experts, park managers and others are
divided into subject workgroups (e.g., Air and Geology; Water Resources; Plants; Animals)
and are asked to review, revise, and improve the information entered into the database
(e.g., justification, monitoring questions and objectives) for vital signs assigned to
their workgroup.
- As a result of applying the ranking criteria, prioritized lists are generated for each
park and one list is generated for the network overall. Each workgroup is then asked to
work with the highest-ranking vital signs assigned to their workgroup, and to develop a
set of specific measurable objectives for each vital sign, and to identify existing
protocols and partnership opportunities as well as other information.
- The workshop results are documented in a report, which is then sent out for peer
review to a larger audience. After incorporating the comments of this peer review, the
results of the workshops are used by the network Board of Directors and/or Technical
Committee to guide the next step, which is the SELECTION of the initial set of vital
signs to be monitored by the network parks.
