Buoy Barn

Lightweight ERDDAP frontend API for NERACOOS buoy information, recent readings, and for generating point forecasts from different model sources.

If you are setting it up for the first time see Initial Configuration down below.

Otherwise you can use make up to start the server, which you can then access the admin at localhost:8090/admin/ and api at localhost:8090/api/.

You can also launch it quickly in your current Kubernetes cluster. See Kubernetes below

There are three ways to refresh data.

• First, you can refresh the values for a specific dataset. This is designed to work with the ERDDAP subscription service, allowing the ERDDAP server to notify Buoy Barn when there is new data, which is more efficient than refreshing on a schedule.

  • First navagate to datasets/ in the API, then find the slug of the dataset that should be refreshed. Append that value to datasets/ and then you can add refresh/ after that, so /api/datasets/<SLUG>/refresh/ will trigger the dataset to update the associated timeseries data.

  • Then go to the erddap server that contains the dataset, say Coastwatch, and find the subscription service (it should be on the home page). From the add page, you can then enter the dataset ID, your email, and the URL that you just found previously to subscribe to updates.

• Second, you can refresh all timeseries from a specific server. This is for timeseries that do not use the ERDDAP subscription service.

• For the fastest refreshing of datasets, Buoy Barn can subscribe to the MQTT service for an ERDDAP server.

  • For this, add the MQTT broker, port, user, and password to the ERDDAP server configuration in the admin.

  • Then run manage.py erddap_mqtt <server_name>. This will connect to the server, subscribe to change/# to get all dataset updates, then for every message check the topic change/<dataset_id> and trigger a refresh of that dataset in the background.

  • Not all messages to a dataset topic are necessarily new data, but they generally mean a change to how ERDDAP understands a dataset, so it's worth attempting a refresh. Because this can be a bit noisy, it checks that there isn't another refresh already scheduled to manage the load on the ERDDAP server.

If you ctrl-c out of the logs (or close the window), you can get the logs back with make logs.

The Docker containers are launched in the background with make up, so they won't dissapear if you close the window or logs. Therefore when you are done, ctrl-c out of the logs and run make down to shut down the server and database.

See forecasts/Readme.md for details about how the forecasts API works and how to add new forecasts.

To add or edit a platform (buoy) go to admin/deployments/platform. From there you can add or edit a platform.

See below for adding a timeseries to a platform.

If you wish to keep track of deployments for particular platforms, you can manage them at admin/deployments/platform.

Platforms and their TimeSeries can be imported from an ERDDAP dataset's metadata:

• For a new platform, use Import from ERDDAP on the platform list.
• To add a dataset to an existing platform (or to bring its timeseries up to date with ERDDAP), use Import dataset from ERDDAP on that platform's page.

Choose the ERDDAP server and enter the Dataset ID (for example N01_sbe37_all). If the dataset has more than one depth or station, add constraints to pick one out, as a JSON object of the ERDDAP constraint (variable and operator) and value, such as {"depth=": 50.0} or {"station=": "44027"}. You can find them from the dataset's ERDDAP page, by choosing a value in the optional constraint dropdowns. Time doesn't need to be constrained.

Preview shows what would change without saving anything:

• The platform location, from the dataset's latitude/longitude (or site_latitude/site_longitude) global attributes, the position of the selected station, or the middle of the geospatial_* bounds. For an existing platform, a location more than 500 m from its current one is highlighted.
• Platform fields ERDDAP has values for: station_name (from platform_name/long_name/title), mooring_site_desc (from mooring_site_desc/summary), ndbc_site_id (from ndbc_site_id/wmo_platform_code), and a guessed platform_type. Each field, and the location, has its own checkbox next to the current and ERDDAP values. Blank fields and a missing location are ticked to be filled in, while values that are already set are only changed if you tick them, so importing another dataset to a platform won't rename it.
• New timeseries, existing timeseries that differ from ERDDAP (with what would change), and ones that already match.
• Variables that can't be imported, because no existing DataType matches their standard_name, long_name or short_name. DataTypes are never created by the import, add one in the admin and import again.

Import selected (or Update selected for an existing platform) creates the ticked timeseries, updates the ticked existing ones and platform fields, and queues the dataset to be refreshed.

Sync with ERDDAP on a platform's page re-checks every ERDDAP dataset (and set of constraints, such as each depth) that the platform's timeseries already use, and shows them together:

• Existing timeseries that differ from ERDDAP (for example new QC flags or changed tidal datums) are ticked to be updated.
• Variables in those datasets without a timeseries yet are listed, but not ticked, as the existing setup is assumed to be deliberate.
• Each platform field, and the location, can be kept as it is or set to the value from any of the datasets, as they can disagree. Only blank fields, or a missing location, that every dataset agrees on are filled in by default.
• Datasets that can't be loaded from ERDDAP are listed, and the rest are still synced.

Update selected saves the ticked changes and queues the changed datasets to be refreshed.

• QARTOD/QC: every numeric QC flag variable (with intent = data_quality, a quality standard name, or qc/qartod in its name) a data variable lists in ancillary_variables that has flag_values and flag_meanings becomes a constraint keeping values whose meaning is pass, good, quality_good or not_evaluated. For NERACOOS style flags that's salinity_qc=0, and for IOOS QARTOD flags (1 pass, 2 not evaluated, 3 suspect, 4 fail, 9 missing) it's <test variable><=2. Rollup flags (standard_name = "aggregate_quality_flag") are also used even when they aren't listed in ancillary_variables: they apply to the variable whose name they start with (<variable>_qartod_rollup), or to every variable if there is only one in the dataset.
• Depth: a depth= constraint sets the TimeSeries depth. Without one, it comes from the variable's sensor_depth, a depth variable whose actual_range is a single value, or equal geospatial_vertical_min/max, so single depth datasets like A01_ocean_001m don't need a constraint.
• Tidal datums: a tidal_datum_offsets_meters variable attribute with a JSON object of offsets, such as {"mhhw": 1.469, "mllw": -1.573}, sets the TimeSeries datum_<name>_meters fields.

The same import can be run from a shell (make shell), which creates any new timeseries for an existing platform:

>>> from deployments.utils.erddap_loader import add_timeseries
>>> from deployments.models import Platform

>>> n01 = Platform.objects.get(name='N01')

>>> add_timeseries(n01, 'http://www.neracoos.org/erddap', 'N01_accelerometer_all', {'depth=': 50.0})

Tests can be run with make test. Django will initialize a new database, run migrations, run tests in transactions, and then delete the database. Tests that attempt to access external servers (for getting ERDDAP data) should be mocked out using VCR to make sure they are more repeatable despite network, server, or dataset issues.

Automated tests are run after code is pushed to GitHub by Semaphore, and test status is indicated on pull requests.

Codacy automatically checks for code quality and test coverage.

Tests coverage can be run with make coverage. After the tests are run, and the report displayed in the command line, it will also generate an html report. This report can be viewed by opening app/htmlcov/index.html which will also attempt to automatically open in the default browser.

To start the database, you'll need to configure it with a username and password. You'll also need to set a secret key that Django will use to protect sessions.

Redis also runs as a cache and a task queue, and the uri to the cache instance should be included.

To do so create a secret.env file in ./docker-data/ that looks something like this.

POSTGRES_PASSWORD=secret_string
POSTGRES_USER=a_user_name
SECRET_KEY=a_really_long_random_string_that_no_one_should_no_and_should_probably_be_gibberish
REDIS_CACHE=rediss://cache:6379/0
DJANGO_DEBUG=True
DJANGO_MANAGEPY_MIGRATE=off
SENTRY_DSN=some_long_string_from_sentry
CELERY_BROKER_URL=rediss://cache:6379/1

An additional environment variable is currently configured in docker-compose.yaml as it is unlikely to need to be changed.

There are additionally two environment variables for changing the timeout when querying ERDDAP servers. RETRIEVE_FORECAST_TIMEOUT_SECONDS and RETRIEVE_DATAFRAME_TIMEOUT_SECONDS.

SENTRY_TRACES_SAMPLE_RATE can be used to set what percentage of requests are performance traced and sent to Sentry. Defaults to 0 if not set.

PROXY_CACHE_SECONDS can be set to override the default caching time for CORS proxy requests to ERDDAP servers.

Metrics are exported over OTLP to an OpenTelemetry collector, and are a no-op unless OTEL_EXPORTER_OTLP_ENDPOINT is set. docker compose up starts a local collector that logs what it receives, so docker compose logs -f otel-collector shows the instrumentation working.

• docs/observability.md — what is exported, how it relates to Sentry, and how to work on it locally.
• docs/observability-deployment.md — Prometheus metric names, verification queries, and the few things the neracoos-aws-cd overlay still needs on top of this repo's kustomize base.

Then you can use make up to start the database and Django server. Docker will take a few min to download and build the containers for the database and server.

Once the database and server are started, you'll need to run make migrate to get the database structure in place.

To create an administrator account run make user and follow the prompts.

Once you have an account setup, you'll be able to login to the admin at localhost:8090/admin/.

You can use Django fixtures to quickly save models from the database and reload them. make dump will save most of the deployment related models to /app/dump.json, which you can load with make load.

• app/
  • account/ Django user account app.
  • buoy_barn/ Primary Django application.
  • deployments/ Database models and API.
  • forecasts/ Forecast models and API.
  • utils/
    • wait-for-it.sh Shell script that can wait until specified services are avaliable before finishing. Helps make up launch Django more reliably.
  • Dockerfile Django server build steps
  • manage.py Django management script
  • requirements.txt Python package requirements
• docker-data/
  • postgres/ Database storage
  • secret.env Environment variables to manage database connection and similar configuration.
• .gitignore Files that should be ignored by git
• .prospector.yaml Linting configuration
• docker-compose.yaml Database and server docker container configuration
• Makefile Common commands made easier to remember.
• Readme.md You are here.

• build Builds containers in parallel
• up Builds and starts Docker containers before following the logs.
• mqtt Enables MQTT subscriber for a NERACOOS dev server.
• down Stops and removed Docker containers.
• stop Stops Docker containers.
• logs Follow logs for currently running Docker containers.
• migrations Attempt to auto-detect and create new migrations for any Django models that have new fields.
• migrate Run database migrations
• prune Clear out old Docker layers and containers systemwide (not just this project)
• fixtures Dump Deployments models to /app/deployments/fixtures/
• load Load Django fixtures in /app/deployments/fixtures/*.yaml
• user Create a new Django admin/super user.
• shell Launch a Python shell from the Django manage.py
• test Run unit tests.
• requirements-compile Take the high level requirements files generate a pinned requirements.txt file.
• requirements-tree See the dependency tree of the requirements and any issues.
• lint Run prospector (and associated linting tools).

Django may let you know that your database is not up to date with all the migrations.

System check identified 1 issue (0 silenced).

You have 1 unapplied migration(s). Your project may not work properly until you apply the migrations for app(s): deployments.

Run make migrate to run any outstanding migrations.

Often the Django server can be ready before the database is due to the database catching up and making sure everything is committed before allowing access.

In order to try to avoid that wait-for-it.sh is run before python manage.py runserver that tries to wait for the database to become avaliable.

This takes care of things most of the time, but sometimes the database will look avaliable to wait-for-it.sh (it depends on if a port is open, and Postgres opens the port at a specific stage in recovery to report status).

If this does occur (look for psycopg in the errors), just open and save app/manage.py to get the server to try to reload. It works to hit save in most other .py files under the app/ directory as well.

Releases are prepared with nox sessions defined in noxfile.py, either locally or via a GitHub Actions workflow. Both paths do the same thing: bump the version, fetch GitHub's auto-generated release notes, and write them into Changelog.md.

Requires the gh CLI to be authenticated. GitButler (but) is used to create the release branch/commit if it's installed; otherwise plain git is used instead.

1. Make sure your working tree is clean, then run:

   nox -s release_prep -- patch   # or: minor, major, or an explicit version like 0.11.0

2. The session resolves the new version, pulls the real GitHub-generated release notes for it (gh api .../releases/generate-notes), and shows you the exact Changelog.md entry it's about to write. Confirm to proceed (or pass --yes to skip the prompt), and it will:

   • Insert that entry into Changelog.md.
   • Bump the version in app/pyproject.toml.
   • Run prek so app/uv.lock gets regenerated to match.
   • Create a release-{version} branch and commit titled Release - {version} with those changes.

3. Review the commit, then open the PR yourself — but pr new release-{version} -F <message file> (path printed by the session), or git push + gh pr create if you're not using GitButler.

4. Once CI passes and the PR is merged into main, run:

   nox -s release_draft -- 0.10.3

   This reads the now-merged Changelog.md entry and creates a draft GitHub release (gh release create --draft) from it — running this does not ship the release. Open the printed URL and click Publish release on GitHub yourself: that manual click is what actually fires the image build/push/Sentry-release/notify flow (a bot/token-authored gh release create can't fire it directly — see below). If you abandon a draft instead of publishing it, its tag was already created on main, so delete both the draft and the tag (git push origin --delete vX.Y.Z) before re-running for that version.

Two manually-triggered workflows mirror the local nox sessions, for cutting a release without any local setup:

• 1. Release Prep (.github/workflows/release-prep.yml) runs nox -s release_prep on a runner. Trigger it from the Actions tab (or gh workflow run release-prep.yml -f version=patch); it pushes the release branch and opens a draft PR automatically. Review and merge the draft PR as usual.
• 2. Release Draft (.github/workflows/release-draft.yml) runs nox -s release_draft on a runner once the release PR is merged into main. Trigger it from the Actions tab (or gh workflow run release-draft.yml -f version=0.10.3) to create a draft GitHub release from the merged Changelog.md entry — check the run's job summary for the draft's URL. This does not ship the release: GitHub deliberately doesn't fire release: published for actions taken by a workflow's own token, so someone still has to open that URL and click Publish release themselves to trigger the image build/push and downstream deploy notification.

(Workflow names are numbered so they sort correctly in the Actions tab's alphabetically-ordered sidebar.)

As we are moving towards using Kubernetes, you can launch Buoy Barn in a Kubernetes cluster. See the /k8s/ directory to see the configs.