---
title: "Discovering Tampa Bay data"
description: "Search the checked catalog and live portals, then interpret and filter the results."
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Discovering Tampa Bay data}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", eval = FALSE)
```

`tampaBayOpenData` searches a bundled catalog of 52 checked layers or the
live ArcGIS portals of six Tampa Bay publishers. Maintainers have validated
the configured endpoints for checked layers. A `discovered` layer comes from
a portal index and has not received those checks.

Live examples below are displayed but not run while the site is built. The
checked-catalog example runs offline.

## Start offline

```{r checked-catalog, eval=TRUE}
library(tampaBayOpenData)

tampaBayOpenData::tbod_list_portals()[, c("id", "publisher", "jurisdiction")]
tampaBayOpenData::tbod_list_datasets(source = "checked")[,
  c("id", "title", "jurisdiction", "validation_status")]
tampaBayOpenData::tbod_search_datasets(
  "park", source = "checked", spatial = TRUE,
  jurisdiction = c("stpete", "clearwater")
)
tampaBayOpenData::tbod_dataset_info("construction-permits")[
  c("title", "scope_note", "verified")]
```

`source = "checked"` reads the registry without HTTP. IDs such as
`construction-permits` and `stpete-parks` are package names, not ArcGIS item
IDs. `tbod_dataset_info(id)` gives the scope, terms, source URL, and endpoint
verification date; that date says nothing about record freshness. The
[coverage article](coverage.html) has counts by publisher, and the [selection
policy](https://github.com/Jaclenga/tampaBayOpenData/blob/main/docs/checked-catalog-policy.md)
explains which layers enter the catalog.

## Match and rank search results

`tbod_search_datasets()` searches checked IDs, titles, descriptions, publisher
names, categories, and tags. It ignores case, but treats punctuation literally.
All words in the query must occur somewhere in a checked row; they can occur in
different fields. If no checked row matches literally, the search tries a
limited spelling fallback for alphanumeric query words; words shorter than
four characters have no typo tolerance.
The fallback searches only the checked catalog; it does not alter a live portal
query. An empty or whitespace-only query lists datasets.

Exact title or ID matches come first. Title or ID prefixes follow, then
substrings, then rows whose title or ID contains the query words separately.
Matches in tags, categories, or publisher come next; description-only matches
follow those. Within each group, each word scores title, ID, tags or
categories, publisher, and description in that order; title, ID, and source
URL then break ties consistently. These are ordering rules,
not measures of dataset quality or relevance to a specific analysis.

```{r checked-search-ranking}
tbod_search_datasets("permit", source = "checked", limit = 5)[,
  c("id", "title", "publisher")]
tbod_search_datasets("permti", source = "checked", limit = 5)[,
  c("id", "title", "publisher")]
tbod_search_datasets("", source = "checked", limit = 5)[,
  c("id", "title", "publisher")]
```

`limit` is a nonnegative whole number or `Inf` (the default). It selects the
first rows **after** source discovery, checked overlay, filtering, and ranking.
Even `limit = 0` still performs a selected live scan and retains its scan
diagnostics. `max_items` separately caps the number of service items inspected
per live portal. A result limit does not make an incomplete scan complete.

## Discover live portal layers

```{r live-search}
found <- tbod_search_datasets("housing", source = "live", portals = "city")
found[, c("id", "title", "publisher", "validation_status", "modified")]

# A one-row result carries the source needed for inspection or retrieval.
if (nrow(found)) {
  selected <- found[1, ]
  selected[, c("item_id", "service_url", "layer_id", "portal")]
  tbod_dataset_info(selected)
}
```

By default, `tbod_list_datasets()` and `tbod_search_datasets()` use `source = "all"`:
they search all six portals and include the checked catalog.
`source = "live"` returns only layers found through portal indexes, including
checked layers when a live endpoint matches a checked entry. If a source is not
indexed or an index lags, it may not appear in live results; a compatible public
ArcGIS layer can still be retrieved by URL with `tbod_get_arcgis_layer()`.
Portal indexes can match fields that are not returned in discovery rows.
`tbod_search_datasets()` ranks indexed hits using available title, ID, and
metadata, but keeps all portal hits even when a returned row has no local text
match. A layer found
both live and in the checked catalog appears once with `checked` status. With
`source = "all"`, a checked match can still appear when its portal index does
not return it.

Use `portals = "city"` for Tampa, `"tbrpc"` for the regional planning council,
or `"stpete"`, `"clearwater"`, `"hillsborough"`, or `"pinellas"` for the
other publishers. A vector selects several. `tbod_list_portals()` shows their
publisher names and source locations offline. `jurisdiction` defaults to
`"all"` for catalog functions and can be a code or a vector of codes. A
jurisdiction or publisher filter narrows the organizations scanned and filters
checked entries. `portals` only selects live organizations; it does not filter
checked entries when `source = "all"`. Retrieval and download functions default
to `jurisdiction = NULL`: a checked ID resolves across the bundled catalog when
it is unique. If two jurisdictions use the same ID, supply `jurisdiction` to
select one.

When a live endpoint matches a checked entry, the result keeps its package
ID, scope, and terms, combines checked and portal tags and categories, and uses
the portal item's modification time. New IDs have the form
`arcgis:<item-id>:<layer-id>` and can be passed to `tbod_get_dataset()` later.
The item or service can still change or disappear. Retrieval checks query
responses and requires `Query` when the layer advertises a capabilities list.

## Search beyond the bundled publishers

`tbod_discover()` accepts a public ArcGIS sharing REST root, FeatureServer or
MapServer service root, or a layer URL. Portal queries search public service
items; service queries filter layer names. Both return rows accepted by
`tbod_get_dataset()`. Portal results have `discovered` status and stable
`arcgis:` IDs; service results have `not_checked` status and use the layer URL
as their ID. Queryable MapServer layers are supported.

```{r custom-discovery}
# Replace this example address with your portal's public sharing REST root.
portal <- "https://example.maps.arcgis.com/sharing/rest"
other <- tbod_discover(portal, query = "parks", max_items = 10)
if (nrow(other)) {
  data <- tbod_get_dataset(other[1, ], limit = 100)
  # A stable ArcGIS ID can also be resolved later through its own portal.
  same_layer <- tbod_get_dataset(other$id[[1]], portal = portal, limit = 100)
}
```

Custom sources default to an `"unspecified"` jurisdiction. Set `publisher`
and `jurisdiction` in `tbod_discover()` when you know them; these labels then
travel with the discovery row and its retrieval provenance. The `portal`
argument on retrieval applies only to a stable `arcgis:<item-id>:<layer-id>`
ID. When retrieving a direct layer URL, `tbod_get_dataset()` does not accept
a `jurisdiction` override.

## Filter results

Both catalog functions accept these optional filters:

| Filter | Match |
| --- | --- |
| `jurisdiction` | One or more codes from `tbod_list_portals()`; default `"all"`. |
| `publisher` | Exact publisher names from `tbod_list_portals()`, ignoring case. |
| `validation_status` | `"checked"` or `"discovered"`. |
| `spatial` | `TRUE` for geometry layers, `FALSE` for tables; unknown geometry is excluded when filtering. |
| `topic` | Literal substring in tags and categories, ignoring case. |
| `category` | Exact category path or final category label, ignoring case. |
| `modified_after`, `modified_before` | Inclusive bounds on the catalog's `modified` timestamp. |

Character filters accept vectors: values within one filter are alternatives;
separate filters and the search text must all match. Date bounds accept a
`Date` or strict `"YYYY-MM-DD"` string for a whole UTC day, or a `POSIXct`
value for an exact instant. Rows with unknown modification times are excluded
only when a date bound is supplied.

```{r county-search}
county_transport <- tbod_search_datasets(
  "road", source = "live", jurisdiction = "pinellas", spatial = TRUE,
  topic = "transportation", modified_after = as.Date("2025-01-01"),
  max_items = 10
)
county_transport[, c("id", "title", "publisher", "modified", "modified_source")]
```

`modified_source = "portal_item"` identifies a live ArcGIS item timestamp;
`"checked_snapshot"` identifies an upstream timestamp saved in the checked
registry. Neither establishes when every record was last updated. The
separate checked `verified` date records an endpoint check, not feature
freshness.

## Understand scan limits and failures

`max_items` caps **portal service items per organization**, not the number of
expanded layers. The default scans at most 25 items per organization. Use
`max_items = Inf` explicitly to follow all available search pages, with a
suitable `total_timeout`. A capped listing can miss matching items beyond the
scan limit. `jurisdiction` and `publisher` narrow organizations before scanning;
other filters apply after discovery and the checked overlay. Consequently,
an item matching `topic` or `modified_after` may be missed by a capped scan.
`publisher` matches a configured name exactly, without regard to case; a name
with no match returns no rows. Use `tbod_list_portals()` to inspect names.

```{r scan-metadata}
results <- tbod_list_datasets(source = "live", portals = "stpete", max_items = 10)
attr(results, "discovery_complete")
attr(results, "discovery_truncated_portals")
attr(results, "discovery_scanned_items")
attr(results, "discovery_issues")
attr(results, "discovery_failed_portals")
```

Check `discovery_complete` before treating results as a complete scan of the
selected public portal indexes. It does not establish a complete inventory of
publisher services. Portal indexes can lag or omit items, private services are
outside the search, and the package includes only public Feature Service and
Map Service items with supported, queryable layers. The other attributes show
scan progress, skipped items, and failed portals. A time budget can stop an
operation; capped scans or item failures can leave returned results incomplete.
A combined search retains rows from working publishers or from earlier pages
of a failed scan. If all selected portals fail before returning rows,
`source = "live"` errors; `source = "all"` warns and returns checked rows.

Continue with [retrieval and downloads](retrieval.html) or read about
[coverage and source limits](coverage.html).