1. Questions

The very first step in creating a sampling design is to establish what questions, goals, or information requirements are being addressed with the planned data collection (Table 1, #1). For some efforts, these may already exist, as with the standards laid out in the Clean Water Act or the Sage-Grouse Habitat Assessment Framework (HAF)⁹ in which the questions are laid out precisely. For example, in the HAF, the formal monitoring objectives for the question of “is this suitable sage-grouse habitat?” are determined according to a form that lists the data required and interpretations of the ranges of values to reach a conclusion.⁹ In many cases, however, the process is less well defined and can make specifying formal monitoring objectives challenging. For example, a common question such as “Were habitat improvement treatments effective?” requires a number of subsequent clarifying questions to develop monitoring objectives, including “what kind of habitat,” “what indicators define that habitat,” and “what constitutes effective?” Note the need for clarifying questions is not a weakness or a sign of poorly written management questions, but rather a part of the process.

2. Objectives

Without clear and explicit articulation of monitoring objectives about what needs to be known, there is no guarantee that a sampling design will produce data that can address the question, goal, or requirement that initially spurred the sampling effort.¹⁰ A clear monitoring objective for the goal to evaluate the effectiveness of habitat improvement treatments might be, “At least 75% of greater sage-grouse habitat in treatment areas must have between 15% and 25% foliar cover from sagebrush.” This clearly states what is meant by “effective” and how that effectiveness is being measured as part of the monitoring effort. Specific monitoring objectives may also be subject to important contextual factors that can limit what is possible (e.g., site ecological potential), species-specific habitat requirements (e.g., breeding versus brood-rearing), and potential risk factors (e.g., large amounts of bare ground and risk of accelerated soil erosion). Ideally, monitoring objectives are supported by reference materials like ecological site descriptions and scientific literature as well as professional experience and judgment. Often, specifying quantitative monitoring objectives is the most challenging aspect of sampling design, especially given what is achievable with available resources like staffing and funding and the timeframe for sampling.

Given that most monitoring efforts are limited by available resources, monitoring data are increasingly collected to address multiple resource questions or management goals. Creating specific monitoring objectives is especially challenging and requires even more thoughtful consideration throughout the sampling design and monitoring implementation process if the intent is to use the data for multiple questions or goals. For example, although broad-scale monitoring projects like NRI have primary goals of monitoring overall rangeland health and trends, the data can have valuable applications for other purposes such as species distribution modeling or remote sensing applications. Another example is the use of Bureau of Land Management's (BLM) Assessment, Inventory, and Monitoring strategy (AIM) under which data are often collected for Land Use Plan evaluation and also used for more local applications like grazing permit assessments for allotments.¹¹ Thus, considering secondary uses of data during the design stage can extend the utility of a sampling design for a broader set of end users and purposes.

The framing of monitoring objectives in a quantitative format can help with creating benchmarks for evaluating monitoring data. Benchmarks are defined categories such as “meeting criteria” and “not meeting criteria,” which are related to management decisions and can be applied to data to make comparisons between different sampling locations.^12,13 For example, if the management objectives for percent of foliar cover account for ecological potential, then “meeting criteria” for arid shrubland would be a lower percent foliar cover than in a wetter and more productive riparian area. By applying benchmarks to separate values from sampling locations into “yes/no” categories, they are harmonized and can be directly compared across lands with differing ecological potential. The process of creating benchmarks can also be useful for identifying priorities both for sampling design and broader management objectives. Patterns related to data needs and distribution of resources may emerge to guide allocation of sampling effort to meet project goals. For example, if when determining objectives for sage-grouse monitoring, a significant number of suitable habitat criteria relate to breeding habitat, in particular, then sampling design steps focused on breeding habitat areas and seasons would be appropriate while placing less emphasis on other seasonal habitats.

3. Reporting units

This stage of sampling design is for determining at what level (i.e., what areas or span of time) results should be summarized, commonly referred to as reporting units (Table 1, #3). The levels do not need to be spatially exclusive and can be based on any relevant information including administrative boundaries, soil maps, ecological potential units, management history, or other management-relevant units. Reporting units differ from sampling strata (see “Stratification and Allocation” below) in that they are summary units and not used for dividing an area into regions for sampling efficiency. During the design process, reporting units do not need to be exhaustive, and additional reporting units can be identified and potentially reported on after the design is completed (provided sufficient samples exist in each reporting unit). However, it is helpful to identify all possible reporting units during the design phase to build the design to ensure sufficient sampling effort in each reporting unit. For sage-grouse habitat, knowing results will be needed for each watershed means the sampling design must contain adequate sampling in each watershed, which might not be the case if reporting units were not explicitly considered.

Tracking information about monitoring objectives and reporting units is an important but complicated task, and benchmarking frameworks like the BLM's AIM Terrestrial and AIM Lotic Benchmark Tools (AIMBT)^14,15 can provide valuable structure for a project (Fig. 1). The AIMBT consists of Microsoft Excel workbooks using a standardized format for the description of benchmarks including the applicable data, the qualifying range of values, and the associated reporting units. This structure serves as documentation for later reference and guides the building of appropriate benchmarks. The AIMBT is machine-readable and supported by software tools (i.e., the aim.analysis R package¹⁶), which makes applying benchmarks to data at a later time as part of the data analysis easier to automate.

The AIMBT can and should be used in conjunction with other decision tools including policy or technical references directed at monitored resources. For example, in sage-grouse habitat monitoring, many of the objectives and benchmarks are already defined in the HAF. Not all resources have existing quantitative policy guides or requirements, so it also can be useful to reference Ecological Site Descriptions (ESDs) through the Ecosystem Dynamics Interpretive Tool (EDIT), a web interface to a system designed to provide characterization of the ecological potential and behavior of a site using state-and-transition models, and the supporting resource management knowledge¹⁷(Fig. 1).

4. Data collection methods

Selecting data collection methods is determined by monitoring objectives and is an important aspect of sampling design (Table 1, #4). Many methods are used for collecting environmental data, but likely only a narrow set will effectively answer any given question or satisfy an objective. Major factors to consider include whether the objectives of the sampling design include requirements for data that are quantitative, qualitative, or both and the required level of precision.¹⁸ Methods differ in sensitivity to variability, and the selection of a method influences the number of sampling locations (i.e., the level of sampling effort or sample size) required to produce useful data. For example, measurements of gaps in foliar cover described in the Monitoring Manual for Grassland, Shrubland, and Savanna Ecosystems (MMGSSE)¹ has been shown to have higher interobserver variability than point-intercept estimates of foliar cover in comparisons made using data available in the BLM Terrestrial AIM Database.¹⁹ The higher observer variability of the canopy gap results may require either a reduced level of acceptable precision or a greater number of sampling locations to produce a high-confidence estimate than for point-intercept estimated foliar cover. If the data will be used in conjunction with other data sets it is important that the methods selected measure the same or compatible variables. Regardless of the requirements, there are certainly documented and robust methods of gathering the needed monitoring data, and a best practice is to use existing documented methods from peer-reviewed sources, which are frequently organized in manuals and technical references^1,18 (Fig. 1). Standardized methods also make comparisons to other published or publicly available data easy, which can help contextualize observations. Further, peer-reviewed methods are defensible in court, should the sampling effort become involved in litigation. Accordingly, modifications to standardized methods should not be made without careful consideration because they can compromise compatibility with existing data, affect accuracy or precision of indicator estimates, and expose monitoring programs to legal challenges. As an example, when evaluating the condition of sage-grouse habitat, many of the most useful methods (e.g., line-point intercept for estimating foliar cover) are described in publications specifically about sage-grouse habitat.⁹

Figure 1.

Example tools and to which steps in sampling design creation they apply. Tools can inform multiple steps and each step can take advantage of multiple tools. This is not an exhaustive list of tools and—depending on the nature of a specific sampling design—some of the listed tools could be useful for unmarked steps (e.g., Ecological Site Descriptions could potentially be used to understand landscape variability as it applies to the required effort).

The primary tools used to determine data collection methods are manuals, technical references, and policy sources (Fig. 1). These types of documents (e.g., MMGSSE¹, the sage-grouse HAF,⁹ and Interpreting Indicators of Rangeland Health²⁰), contain well-documented, field-tested, and peer-reviewed methods. By comparing available methods with the monitoring objectives and associated benchmarks, a small set of methods can be selected to meet the data needs. This avoids the pitfalls of creating new methods, which may not collect data with the accuracy or precision needed to be defensible in litigation. Novel methods may be appropriate, but should be considered and tested against existing methods to understand their biases and limitations.

If monitoring questions and objectives are not fully addressed with available methods, we suggest adding your new or customized protocols while keeping the core established methods intact. If selecting line-point intercept as the method for measuring foliar cover for sage-grouse habitat, then the method is described in detail in MMGSSE, which includes the full written methodology, a data recording format, and calibration protocols to ensure the data are consistent and comparable to existing data.¹

5. Sample frame

A sample frame provides a sampling design with constraints, including the spatial and environmental extent of the sampling effort. Sample frames are often spatial (i.e., study or project area) but also can be focused on a species (i.e., a population or herd) and include a temporal component. Here, we focus on spatial sample frames, which are defined to include the locations or areas relevant to the monitoring objectives and the questions intended to be answered^6,21 (Table 1, #3). Note that in trend monitoring, repeated measures are important and the sample frame needs to define the frequency of remeasurement. The sample frame is the area (or length for linear features) within which data will be collected and to which the results of data analyses are applicable. The sample frame can be determined using the monitoring questions and objectives of the sampling effort. In other words, the sample frame must include all parts of the landscape to which the sampling effort applies and no parts of the landscape that cannot or should not be sampled. For example, if the goal is to monitor sage-grouse habitat on public lands, then the sample frame must cover at least the extent of the potential estimated habitat, but would not include nonhabitat areas or areas not to be sampled, such as privately owned land. In specific applications, like oil or gas well-pad reclamation monitoring, a sample frame for reference sampling locations may be a combination of spatial (e.g., within 2 km of the pad) and environmental criteria (e.g., having the same soil and topography as the pad) that allow objective reclamation standards to be established specific to an individual well pad.⁵

The sample frame for a sampling effort can be selected and tested with multiple tools.The reporting units defined in populating the AIMBT and policy requirements are a good starting point for defining and organizing the sample frame with all reporting units entirely falling within the sample frame. Mapping tools, like geographic information systems (GIS) software, are useful for visualizing options to make decisions.

6. Design type

Once the monitoring objectives, data collection methods, and sample frame for a sampling effort are set, the next decision point is whether the sampling design will use probabilistic or nonprobabilistic methods of selecting sampling locations. Both approaches fill particular needs and have their applications, but each also comes with restrictions that determine which is more appropriate for given monitoring objectives.

Probabilistic sampling designs allow for results within a greater sample frame area to be inferred from the sampling locations. Probabilistic (i.e., random) designs establish a set of locations that might be sampled and assigns each site a probability of being selected before using those probabilities to pick a random subset that will be sampled. For probabilistic designs, the method of “how” to select the subset must be determined. Because these sampling locations are typically not deliberately selected for and are therefore not representative of a single, specific resource use, the data collected can be reused in multiple analyses within the sampling frame.

Most statistical methods of inference (e.g., extrapolating from individual measured sampling locations to the whole sample frame) depend on the assumption of random sampling.²¹ The statistical analyses possible with probabilistic sampling offer significant advantages over nonprobabilistic sampling. Statistical analyses using monitoring data can estimate the properties of the sample frame because a random subset of locations is representative of (i.e., behaves as a stand-in for) the whole set of locations. A common application of monitoring data is to calculate the central tendency values (e.g., mean or median) of relevant indicators within the reporting units. For example, if all sage-grouse habitats in a sample frame had a chance of being sampled, the vegetation data collected from a sample of locations can also be used to estimate the percent cover of perennial grass across the entire sample frame. Further, data derived via probabilistic sampling designs can estimate variability within a sampling frame that may spur other concerns or the need for stratification to organize sampling into more uniform units. Data collected probabilistically can also be cautiously reused in multiple analyses unrelated to the original purpose of the data collection if the design fits the new monitoring objectives.²² As an example, data collected to answer questions about rangeland health may also be used in analyses focusing on the evaluation of grazing permits.¹¹ However, there are constraints on reusing data, however, which depend on the details of a sampling design. In particular, the extent of the sample frame—and its overlap with other sample frames if multiple data sets are combined—sets limits on inference extent.

For a probabilistic design, the next decision is how to select sampling locations from all possible locations in the sample frame. Two common probabilistic methods for sampling landscapes are simple random and spatially balanced random (Fig. 2). A simple random design is straightforward, selecting one random location at a time from the available pool where all locations have an equal probability. One somewhat counterintuitive and often undesirable artifact of simple random sampling is the natural clustering of points.⁶ Spatially balanced random sampling algorithms were developed to maintain the same statistical properties of simple random sampling but to overcome this natural clustering by spacing sample locations—for example Generalized Random Tessellation Stratified²³—to select locations that are randomly distributed and also spatially balanced across the sample frame^24,25 (Fig. 2). There are also random sampling designs in which sample probabilities can be unequal (e.g., an accessibility weighted cost approach), but unequal probabilities make analysis and interpretation more complicated.²¹

Nonprobabilistic sampling (i.e., key area or purposeful sampling) relies on the deliberate selection of specific sampling locations, which are either targeted at a specific location of interest or assumed to be representative of the entire sample frame. However, assumptions about how widely key area data can be generalized into unsampled areas is subjective and statistical interpretations are limited to the sampled sites, but these approaches are straightforward and have been used historically by land management organizations.⁷ Key areas can be an appropriate option for monitoring a specific area or permitted activity, particularly when time and funding are limited, but can be challenging to defend if conclusions are extrapolated beyond sample locations and the areas they were selected to represent. Data collected from nonprobabilistic designs cannot be extended statistically beyond the sampled locations, which limits the utility of the data collected. In some cases, retrospective demonstrations of how representative a set of key area samples are relative to a broader sample frame can be used to justify inference across a broader area. For example, exhaustive environmental datasets like digital elevation models can show that a set of samples covers a similar distribution of elevation values to those throughout a sample frame. Using these post hoc assessments depends on the assumption that the environmental dataset is closely related to variation in the collected data (e.g., elevation being related to foliar cover), but these assumptions are often challenged.

For key area sampling, there are tools to select appropriate reference locations. In addition to best professional judgment and existing data, there is the Automated Reference Toolset (ART).⁵ ART uses soil and topography information in an area and selects locations that closely match to act as references.^4,5 Because the inputs are spatial rasters, there is potential for remotely sensed products like vegetation functional group mapping to be used with ART.^5,26,27 This is particularly useful in evaluating recovery from disturbances where having undisturbed, matching reference plots provides context for assessing how the recovery process is unfolding.

Figure 2.

A comparison between two approaches to drawing a random sampling design with 100 points. The upper map shows two Sagebrush Focal Areas (SFAs) in northern Nevada with points distributed in a stratified, spatially balanced, random design and the interface from the Balanced Design Tool used to draw the points.²⁸ In comparison the lower map, a simple random design drawn in ArcGIS, has less even point densities across the sample frame, particularly in the Northern Central Nevada SFA (in purple). The combination of stratification and spatial balance has effectively prevented spatial clustering in the upper design, potentially increasing efficiency.

There are software tools available for random designs. The Balanced Design Tool (BDT)²⁸ is a web interface that allows the rapid drawing of spatially balanced, random sampling designs from a sample frame quickly and easily. The BDT can quickly test a sampling design by creating an interactive map of the results and offering control over features like stratification and how sampling locations are distributed (see Stratification and allocation below). The R Package sample.design has the same features as the BDT with finer control over designs, using functions from the package spsurvey to draw spatially balanced designs.^29,30 Sample.design, however, requires the ability to write R code, and the BDT provides a graphical interface. In both cases, the designs are random and reproducible because the outputs include the code and shapefiles needed to recreate the design in the future.

7. Stratification and allocation

Stratification divides a sample frame into smaller areas (i.e., strata) and allocates sampling effort per stratum²¹ (Table 1, #7). When done correctly, stratification accomplishes three main goals for sampling design: 1) reduction in overall variance estimates by separating differing areas within the sample frame, 2) representing small yet important areas, and 3) allowing for disproportionate sampling in strata to accommodate specific monitoring objectives or to allocate more samples to strata with higher heterogeneity to better characterize those areas. Stratification is commonly used in rangeland monitoring to deal with nonuniform variability in the sample frame. If a sample design's strata are defined homogeneously (i.e., variability within strata is less than variability among strata), then the resulting increase in statistical power requires smaller sample sizes to meet monitoring objectives.

Stratification can be used to allocate samples disproportionately across the sampling frame. For example, stratification can ensure that important yet less spatially prevalent areas (e.g., riparian areas) are sampled. In a simple random or spatially balanced random design the likelihood of sampling a rare, but important area, is low because it makes up a small proportion of the sample frame. Alternatively, more sample sites could be put in strata that were expected to be sensitive to change (e.g., more human impact) and less effort put into sampling areas expected to be more stable (e.g., remote areas). Strata can be applied to a sampling design before or after determining the required effort (see Estimating required effort below), depending on monitoring objectives. Stratification should not be undertaken lightly because it introduces significant complications for analyses, including adjusting for differing variation among strata and the loss of inference area when combining data with other stratified designs.

One common goal of sampling design is to best represent the environmental variability in the sample frame, which may require stratification with multiple criteria. For example, a manager might know from experience that vegetation communities vary with elevation, which influences the characteristics of the sage-grouse habitat. Standard approaches to stratification require sampling locations be allocated to each stratum combination (e.g., high-elevation shrubland, low-elevation shrubland, and high-elevation grassland), which quickly becomes intractable.

Several recently developed sampling approaches allocate samples across heterogeneous landscapes on the basis of matching allocation patterns to the patterns in multiple environmental variables, which can approximate complex variability across a landscape.^31,32 A conditioned Latin Hypercube (cLHS) is one of those approaches used commonly in soil science, which takes into account all environmental variables of concern to condition the probability of selection for each part of the landscape while maintaining correlation structures. For example, cLHS has been used in soil mapping along with raster data of topography, expected soil type, and difficulty of access to produce sampling designs that take all three into account and selects optimal sampling locations to provide coverage while maintaining probabilistic inference.^32,33 Although the samples from a cLHS sample draw are not explicitly weighted with respect to area sampled, central tendency and variability estimates can still be calculated as long as the environmental variables represent the entire sample frame. However, samples from a cLHS draw are designed to represent the environmental variability in a sample frame and not reduce variance (as is often the case with stratified designs). Managers interested in minimizing sample variance estimates will want to include factors or variables in their analysis (e.g., soil map unit, elevation) that may explain among sample variability in the measure of interest to increase their ability to detect change or differences.

Stratification tools include the Benchmark Tools, the BDT,and sample.design.^28,29 The benchmarks in a completed Benchmark Tool often take into account ecological potential or management units, both of which can suggest natural groupings of ESDs or management units to create homogeneous strata. ESD information can be found through EDIT and tied to soil maps to create strata.¹⁷ The BDT and sample.design are both capable of handling stratified designs, the former in a point-and-click interface and the latter in R code. The ability to see the outcome of a proposed design immediately in the BDT through an interactive map makes it easier to iterate through possible stratification and allocation schemes to find one that works well (Box 1).

8. Estimating required effort

Once stratification and allocation decisions have been made, sampling effort (i.e., number of sample locations) needs to be allocated. Because sampling is always constrained by time, funding, and available labor, making decisions about where to apply available resources to maximize data value is important (Table 1, #8). An initial estimate of the amount of data needed must be made before any allocation occurs. Heterogeneity of the sample frame, heterogeneity within potential strata, variability of the data collection methods, and other factors have a direct influence on the minimum amount of data necessary to meet the objectives of a sampling effort. Determining what constitutes a “sufficient” sample requires significant thought and is helped by existing data, which can be used to estimate variability in the sample frame. For the example of needing to detect sagebrush cover within a 15% to 25% cover range, one would need to estimate the expected standard deviation of the cover values and decide errors can be tolerated.³⁴ Type I errors—so called “false alarms”—would lead to concluding sagebrush cover was in the desired range when it was not. Type II errors would lead to missing a desirable outcome or concluding cover was not within the desired range when it was. From convention, Type I error rate is typically set at 5% (i.e., α = 0.05) and Type II error rate at 10% to 20% (corresponding to statistical power from 80% to 90%). However, the consequences of each type of error should drive the selecting of acceptable error rates.^35,36

This approach to sample allocation brings up a paradox of sampling design in which one needs to estimate the variability of an area they need to sample before sampling it. In general, detection of smaller changes, lower Type 1 error, and higher power require more samples to achieve. In more variable landscapes, such as those with different soils and multiple intermixed ecological potentials, it may not be appropriate to assume that the variation is similar across a sampling frame. Such cases require more complicated consideration of sample sizes and use of strata to parse out variation at the outset. These situations often result in stratified designs where power analysis may be specific to the different strata used (e.g., soil types). There are also situations in which logistics, not sufficiency, are the primary limiting factor; available funding, labor, and time may set a hard limit on how much sampling can be done, but often adjusting data collection methods and using landscape stratifying variables can ensure monitoring objectives are met.³⁷ Although these conditions are not ideal, a realistic expectation is to conduct as much sampling as can be afforded in the available time.