---
title: "Part 3: Large Downloaded GBIF Tables"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Part 3: Large Downloaded GBIF Tables}
  %\VignetteEngine{knitr::rmarkdown_notangle}
  %\VignetteEncoding{UTF-8}
---

## Transparent setup

```{r setup, include = TRUE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7,
  fig.height = 5,
  purl = FALSE
)

has_data_table <- requireNamespace("data.table", quietly = TRUE)
ext_file <- function(...) {
  path <- system.file("extdata", ..., package = "gbif.range")
  if (nzchar(path)) {
    return(path)
  }
  normalizePath(file.path("..", "inst", "extdata", ...), mustWork = TRUE)
}

library(gbif.range)
```

## Scope

This vignette documents the disk-based workflow for large downloaded GBIF tables. It is intended for situations where occurrences have already been exported from GBIF and the full table is too large or too inconvenient to keep in memory.

The workflow is built around three functions:

- `split_gbif_by_species()` to split a large GBIF table into one file per species or GBIF key,
- `species_csvs_to_ranges()` to read those files sequentially and build one range per species,
- `read_range_rds()` to read the saved range outputs back from disk.

This workflow is meant for the point where direct credential-free GBIF retrieval via `rgbif` (Chamberlain et al. 2022) is no longer the most practical option. For many single-species or moderate-volume tasks, `get_gbif()` is enough. For very large downloaded exports, working from files on disk is usually the better choice.

## Why the workflow is split into two steps

`get_range()` is intentionally a single-species function. That is scientifically sensible, because range inference depends on one focal set of occurrences and one associated taxon concept at a time.

For large multi-species downloads, the package therefore separates file handling from range inference:

1. first create one species file per GBIF key,
2. then process those files one by one.

This design keeps peak memory use low and makes intermediate files easy to inspect, clean, archive, or parallelize outside R if needed.

It also keeps the biological interpretation clean. Each file still corresponds to one focal GBIF taxon key, and each range is then inferred with the same `get_range()` machinery described in Part 1: `vignette("gbif-retrieval-and-taxonomy", package = "gbif.range")`.

## Example data

The package includes a small GBIF-style offline example under `inst/extdata`. The file is much smaller than a real GBIF export, but it exercises the same workflow.

```{r input-file, eval = has_data_table}
gbif_file <- ext_file("occ_example_4sps.csv")
utils::head(utils::read.delim(gbif_file, sep = "\t", stringsAsFactors = FALSE))
```

## Step 1: split the downloaded table by species

The first step is to stream the input file in chunks and write one species file per GBIF key.

```{r split-table, eval = has_data_table, warning = FALSE}
batch_root <- file.path(tempdir(), "gbif-range-batch-vignette")
split_dir <- file.path(batch_root, "split")
occ_dir <- file.path(batch_root, "occ_min")
range_dir <- file.path(batch_root, "ranges")

# Start from a clean temporary workspace so the vignette is reproducible.
unlink(batch_root, recursive = TRUE)

split_summary <- split_gbif_by_species(
  input_file = gbif_file,
  outdir = split_dir,
  chunk_size = 100,
  sep_in = "\t",
  sep_out = "\t",
  overwrite = TRUE,
  verbose = FALSE
)

knitr::kable(
  transform(split_summary,
    species_file = basename(species_file)
  )[, c("species_name", "n_records", "species_file")],
  align = "l"
)
```

A few implementation details are worth highlighting.

The splitter reads only the requested columns, not the full source table. It keeps the file extension as `.csv` for convenience, but the files remain tab-delimited by default. This preserves GBIF-style exports while still producing species-specific files that are easy to browse or reuse.

This is also the right place to simplify the table before any range-building starts. In a production workflow, this step can be used to retain only the columns needed for the later analysis and to create a species-level file archive that is easier to inspect than one monolithic GBIF export.

## Step 2: build ranges sequentially from disk

For this vignette, the ecoregion layer is a single enclosing polygon built from the extent of the example records. This keeps the example fully offline and lightweight while still exercising the file-based range workflow.

In ordinary use, you would typically replace this with `ecoreg = "eco_terra"`
or with a spatial object returned by `read_ecoreg("eco_terra")` (Olson et al.
2001; The Nature Conservancy 2009). More generally, any polygon object with a
named character column works as `ecoreg`, and `make_ecoreg()` accepts any
spatially structured raster as input — not only climate layers.
See Part 2: `vignette("ecoregion-constrained-range-inference", package = "gbif.range")`
for full details on both.

```{r batch-range-build, eval = has_data_table, warning = FALSE}
occ_example <- utils::read.delim(gbif_file, sep = "\t", stringsAsFactors = FALSE)

eco_batch <- sf::st_sf(
  data.frame(ECO_NAME = "batch_demo"),
  geometry = sf::st_sfc(
    sf::st_polygon(list(matrix(
      c(
        min(occ_example$decimalLongitude) - 5, min(occ_example$decimalLatitude) - 5,
        max(occ_example$decimalLongitude) + 5, min(occ_example$decimalLatitude) - 5,
        max(occ_example$decimalLongitude) + 5, max(occ_example$decimalLatitude) + 5,
        min(occ_example$decimalLongitude) - 5, max(occ_example$decimalLatitude) + 5,
        min(occ_example$decimalLongitude) - 5, min(occ_example$decimalLatitude) - 5
      ),
      ncol = 2,
      byrow = TRUE
    ))),
    crs = 4326
  )
)

range_summary <- species_csvs_to_ranges(
  species_dir = split_dir,
  ecoreg = eco_batch,
  ecoreg_name = "ECO_NAME",
  outdir = range_dir,
  occ_outdir = occ_dir,
  occ_save_as = "tsv",
  range_save_as = "rds",
  sep_in = "\t",
  overwrite = TRUE,
  degrees_outlier = 30,
  clust_pts_outlier = 2,
  buff_width_point = 1,
  buff_incrmt_pts_line = 0.1,
  buff_width_polygon = 1,
  format = "SpatVector",
  verbose = FALSE
)

knitr::kable(
  transform(range_summary,
    occ_file   = sub(".*speciesKey_", "", basename(occ_file)),
    range_file = sub(".*speciesKey_", "", basename(range_file))
  )[, c("species_name", "n_points", "occ_file", "range_file")],
  align = "l"
)
```

Internally, `species_csvs_to_ranges()` reduces each species file to the minimal structure expected by `get_range()`: one focal species plus decimal longitude and latitude. It can also save those minimal occurrence tables to disk, which is useful if you want to inspect exactly what was passed to `get_range()` after deduplication.

Once the occurrences are split to species level, it becomes straightforward to rerun the exact same batch with different ecoregions, different range arguments, or stricter deduplication while keeping the raw downloaded export unchanged.

### The built-in ecoregion shortcut

The same workflow can resolve packaged ecoregions internally:

```{r ecoreg-shortcut, eval = FALSE}
# Written to its own directory and stored under its own name, so that running
# this chunk by hand does not overwrite the offline results built above.
range_summary_builtin <- species_csvs_to_ranges(
  species_dir = split_dir,
  ecoreg = "eco_terra",
  ecoreg_name = "ECO_NAME",
  outdir = file.path(batch_root, "ranges_eco_terra"),
  overwrite = TRUE
)
```

This is the most convenient option when your downloaded table and your intended range product both target a broad terrestrial workflow.

It is also the most natural way to move from a large terrestrial GBIF export to a batch of ecoregion-constrained range maps without having to pre-load the ecoregion object yourself.

### A production-style terrestrial batch

The offline vignette uses a simple polygon because it must build quickly and without network access. A real downloaded terrestrial workflow would normally look more like this:

```{r production-batch, eval = FALSE}
# Non runable example 1
split_summary <- split_gbif_by_species(
  input_file = "gbif_download.tsv",
  outdir = "species_occurrences",
  chunk_size = 1e5,
  sep_in = "\t",
  sep_out = "\t",
  overwrite = TRUE
)

# Non runable example 2
range_summary <- species_csvs_to_ranges(
  species_dir = "species_occurrences",
  ecoreg = "eco_terra",
  ecoreg_name = "ECO_NAME",
  outdir = "species_ranges",
  occ_outdir = "species_occurrences_min",
  occ_save_as = "tsv",
  range_save_as = "rds",
  overwrite = TRUE
)
```

This is the clearest batch version of the main terrestrial workflow: one GBIF export in, one species file per taxon, one saved range per taxon.

### Inspect one species file before building ranges

One advantage of the split-first design is that you can inspect or clean individual species files before generating ranges. This is often useful when testing a new workflow on a few taxa before launching a full batch.

```{r inspect-split-file, eval = has_data_table}
first_species_file <- split_summary$species_file[1]
utils::head(utils::read.delim(first_species_file, sep = "\t", stringsAsFactors = FALSE))
```

That inspection step is simple, but it is often the moment where issues such as missing coordinates, duplicated points, or unexpected names become obvious.

## Step 3: read and inspect the saved ranges

If the range outputs are saved as `.rds`, `read_range_rds()` restores them as a
`getRange` object with the same `init.args` and `rangeOutput` fields returned by
`get_range()`, so they can be passed straight on to the other package functions.
Note that `init.args$ecoreg` is not serialized and must be re-supplied before a
restored range can be used with `cv_range()`.

```{r read-ranges, eval = has_data_table}
# Read the first saved range to inspect its structure.
first_range <- read_range_rds(range_summary$range_file[1])
class(first_range)
class(first_range$rangeOutput)
nrow(first_range$rangeOutput)
```

The example below overlays every saved range in the batch summary. This is a useful diagnostic pattern when checking whether a batch run produced geometries with the expected extent and shape.

```{r plot-ranges, eval = has_data_table}
range_colors <- c(
  grDevices::rgb(0.10, 0.40, 0.75, 0.30),
  grDevices::rgb(0.85, 0.35, 0.10, 0.30),
  grDevices::rgb(0.20, 0.65, 0.30, 0.30),
  grDevices::rgb(0.70, 0.65, 0.10, 0.30)
)

# Read every saved range once, then dissolve each to a single outline
ranges <- lapply(range_summary$range_file, function(f) {
  merge_range(read_range_rds(f))
})

# Extent spanning the whole batch
combined_ext <- terra::ext(ranges[[1]])
for (r in ranges[-1]) {
  combined_ext <- terra::ext(
    min(terra::xmin(combined_ext), terra::xmin(r)),
    max(terra::xmax(combined_ext), terra::xmax(r)),
    min(terra::ymin(combined_ext), terra::ymin(r)),
    max(terra::ymax(combined_ext), terra::ymax(r))
  )
}

# Draw the first range with the shared extent, then overlay the rest
terra::plot(
  ranges[[1]],
  ext = combined_ext,
  col = range_colors[1],
  main = "Batch-generated ranges"
)
for (i in seq_along(ranges)[-1]) {
  terra::plot(ranges[[i]], add = TRUE, col = range_colors[i])
}
```

The outlines above are smooth and blocky because `ecoreg` here is a single
rectangular polygon, chosen to keep the vignette offline: each range is just the
buffered hull of a point cluster clipped to that rectangle. Running the
`range_summary_builtin` call shown above and plotting it instead of
`range_summary` swaps that rectangle for `eco_terra`, which constrains the same
hulls to real ecoregion boundaries and produces much finer outlines following
coastlines and habitat limits.

## Practical advice

The disk-based workflow is most useful in three situations.

First, when a direct in-memory call to `get_gbif()` is not appropriate because the project already relies on downloaded GBIF exports.

Second, when a large multi-species table needs to be checked, cleaned, or archived at the species level before range inference.

Third, when the same species-specific occurrence files need to be processed repeatedly with different `get_range()` settings, ecoregions, or evaluation schemes.

The third point matters scientifically, not just computationally. It makes parameter sensitivity analyses much cleaner, because you can rerun the same archived species files under alternative ecoregion layers or buffering choices without changing the underlying occurrence input.

## Take-home message

The disk-based workflow turns `gbif.range` into a practical bridge between very large GBIF exports and species-level range inference. Instead of treating file splitting and range building as an external preprocessing step, gbif.range provides a coherent, testable, and documented path from a downloaded GBIF table to saved species ranges on disk.

## References

Chamberlain, S., Oldoni, D., & Waller, J. (2022). rgbif: interface to the global biodiversity information facility API. https://doi.org/10.5281/zenodo.6023735

Olson, D. M., Dinerstein, E., Wikramanayake, E. D., Burgess, N. D., Powell, G. V. N., Underwood, E. C., … Kassem, K. R. (2001). Terrestrial ecoregions of the world: a new map of life on Earth. *BioScience*, 51(11), 933–938. https://doi.org/10.1641/0006-3568(2001)051[0933:TEOTWA]2.0.CO;2

The Nature Conservancy (2009). Global Ecoregions, Major Habitat Types, Biogeographical Realms and The Nature Conservancy Terrestrial Assessment Units. Cambridge (UK): The Nature Conservancy. https://geospatial.tnc.org/datasets/b1636d640ede4d6ca8f5e369f2dc368b/about