This folder contains a set of helpers written in Go and Bash, which are used to automate the creation of relevant environment variable related files for the ocis admin documentation. It is usually a good idea to start using them before a new ocis stable branch is deployed which is the docs prerelease related to ocis master. The helpers must run on a newly created ocis stable version at a minimum.

A copy task will copy all relevant data to a subfolder inside the modules directory which is build relevant for Antora.

The subfolders in services (see Folder Structure) are the source for this process. Some content in these folders need mandatory manual updating before the copy task can be issued. This is due to the fact that only a human can decide on the desired outcome. Although many things are fully automated, these few steps cant.

• Table of Contents
• Prerequisites
• Process Flow
• Programs and Scripts
• Tasks
  • services
  • rogue
  • extended
  • deltas
  • copy
  • cleanup_s and cleanup_h
• Folder Structure
• Notes to the extended_vars.yaml File
  • Tasks After any File Changes
  • Recovery
• Helper Usage
  • Version Dependent Tasks
    • For the Docs Prerelease Version
    • For any Other Version
• Envvar Delta File Creation
• Development

• You must have go installed.
• Bash must be available.
• You must have cloned the ocis repo on the same lavel as this repo.

When creating the ocis admin docs for a version, Antora requires several files to be present that are generated by the helpers. Therefore, documentation must ensure that these files are present and up to date. Otherwise, errors will be thrown due to missing files, or the data will not be in sync with reality.

The steps to keep everything in sync are:

• Regularly use the services and rogue task from the prerelease against ocis master.
  Note that the prerelease is a regular version folder. The question if you have switched to the relevant stable version in ocis applies then to ocis/master.
• Always use all helpers for a new ocis version.
• The copy task will copy the generated files related to a version to a defined location where Antora can process them.

The helpers are (in order of usage):

1. identify_envvar_placeholder_names.sh
2. ocis_helpers (or go run . if you want to run from source)

ad 1)
This Bash script checks for any non-SemVer usage of environment variables relating to introduction and removal versions. These version placeholders, which are usually text strings identifying a release name, are used when the ocis Dev Team does not know when the relevant environment variable will be added or finally be removed. Running this script makes it easier to identify these environment variables and set the correct version in the ocis repo. IMPORTANT: This script must be run and any necessary fixes made before the other helpers get started. Otherwise, incorrect data will appear in the documentation.

ad 2)
This Go program generates all the envvar related files required for the documentation. It is highly recommended to run them one by one. The generated files are located in the services subfolder unless otherwise defined.

The deltas task of this program generates the relevant added, deprecated and removed envvar adoc tables. To run this task, you must first change the versions to be compared in the delta_config.yaml file of the version processed to ensure the output is correct. A major prerequisite is, that each version to be compared must have an up-to-date env_vars.yaml file located in the persistent_files subfolder of that version and that all changes are already merged into the relevant ocis version branches.

Files with output

1. services: generate service envvar tables
2. rogue: create/update the extended_vars.yaml file
3. extended: generate extended_configvars table
4. deltas: generate the added, deprecated and removed envvar tables
5. copy: copy the subfolders of /services/ to /modules/admin/examples/ocis_helpers/ except 'persistent_files'

Cleanup only

6. cleanup_s: cleanup folders in /services/ except 'persistent_files'. run services, extended, deltas to recreate
7. cleanup_h: remove the folder /modules/admin/examples/ocis_helpers/. folder contents is build relevant for antora

The services task reads all environment variables present in the code in the ocis repository's services folder. Therefore, it is crucial to switch to the relevant ocis branch first and parameterise the helper for the target version in question. See section Helper Usage for more details.

There are four outcomes of this task:

• For each service an adoc (<service_name>_configvars.adoc) file that renders to a table containing each envvar used for the particular service.
• For all global envvars (starting with OCIS_) a file named global_configvars.adoc.
• For each service a yaml file (<service-name>-config-example.yaml) for deployments that use yaml configuration. Note that there is no extra file covering global envvars.
• A file named env_vars.yaml that is a collection of envvars with additional information which is used to create delta files.

In file env_vars.yaml, for a global envvar (starting with OCIS_), the name key lists the global envvar followed by all service envvars using it, separated by ;. The occurrences are collected from the ocis code, a global envvar that is no longer part of the code keeps the name it had.

Use the -d flag (debug) to additionally print all global envvars (starting with OCIS_) where the service envvars referencing them do not agree on the same defaultValue. For each of those global envvars, all different default values found are printed together with the service envvars using them. The same report is printed a second time for the description key, listing all different descriptions found per global envvar. The flag does not change the content written.

The rogue task identifies all environment variables that are not defined in a service. These environment variables are queried on ocis startup, before any other actions or settings are made. The relevant identification method is the use of the Go function go.GetEnv. Due to the nature of this process, the result must be checked and fixed manually, because if the code location shifts, the process cannot identify its validity. Any changes to this file must be checked for validity. Therefore, it is crucial to switch to the respective ocis branch first and parameterise the helper for the corresponding target version. See section Helper Usage for more details. There is a section Notes to the extended_vars.yaml File with an in-depth description for how to manage changes.

There is one outcome of this task:

• The file named extended_vars.yaml collects all the rogue environment variables found, along with additional information, which is then used to create the extended_configvars.adoc file.

The extended taks generates from the extended_vars.yaml an adoc file.

There is one outcome of this task:

• A file named extended_configvars.adoc that represents all rogue envvars.

The deltas task generates a set of output files representing the changes to the former version for each target version defined. This task requires the services task to have been run, and for all referenced env_vars.yaml files to be up to date. There is a section Notes to the extended_vars.yaml File with an in-depth description for how to manage changes Envvar Delta File Creation.

Use the -d flag (debug) to check for introduction versions that are found additionally but are out of scope.

There are three outcomes of this task:

• A file named <version_from>-<version-to>-added.adoc that represents the added envvars to this version.
• A file named <version_from>-<version-to>-deprecated.adoc that represents the deprecated envvars in this version.
• A file named <version_from>-<version-to>-removed.adoc that represents the removed envvars from this version.

The copy task moves all non-permanent results to their final destination for use with Antora. Please note that using the copy task alone ensures that the files are tracked and included in any docs build.

These tasks cleanup (remove) any files that were created by the helpers in either the services (except persistent files) and the output folder. Note that the latter is relevant when developing, as Go template files are temporarily saved there.

The folders are managed using constants that can be redefined if necessary. Please note that if the name of the services folder is changed, other helpers may also need to be adapted. The intention behind using constants is to make development easier if folder names need to be changed. Changing without a good reason is not adviced.

1. ocis_helpers/output folder
2. content/ocis/<version>/services folder (source for 3)
3. content/ocis/<version>/modules/admin/examples/ocis_helpers folder (Antora build relevant)

ad 1)
This folder is temporarily generated for all go-template files that are created on the fly. These go programs will then be executed and generate the required output which is saved into the services folder. If not defined by a flag when using ocis_helpers, the folder is deleted once the program has finished successfully. This folder is completely ignored using .gitignore.

ad 2)
The services folder has a sub structure that reflects the purpose of the generated files.

services
  adoc              (1)
  env_var_deltas    (2)
  extended          (3)
  persistent_files  (4)
  yaml              (5)

Please note that all folders are ignored using .gitignore except the persistent_files. folder. All other data can be recrated at any time.

• In (1), you will find all the .adoc files representing the environment variables used in the services. This also includes an additional file that collects all global environment variables starting with OCIS_. Therefore, the "service name" global is a reserved service name and must not be used to name a service.

• In (2), you will find the added, deprecated and removed .adoc files for that specific version.

• In (3), you will find the rendered extended adoc file based on extended_vars.yaml.

• In (4), you will find three types of files.

  • First, there is the env_vars.yaml file that represents all service dependent envvars like in a database. When running the deltas task to generate the envvars that have been added/deprecated/removed, this file is being used to compare between versions.
    Please note that this file is created “chaotically” and currently has no sort order. Although each recreation appears to involve many changes compared to the previous version, only a few of these are real. Note that the generated tables follow the order of this file.
  • Secondly, there is the extended_vars.yaml file. These environment variables are not related to or defined by any service. They are used to parameterise the ocis code on startup.
  • Third, there is the delta_config.yaml file. It defines which versions the deltas task compares and which introduction versions are excluded from gathering. See Envvar Delta File Creation.

• In (5), you will find yaml configuration examples for envvars related to a service.

ad 3)
The copy task will copy the contents of (2) to (3) except the persistent_files folder. This folder is excluded from the copy process because it contains important files that are unique to the version and rarely change. These files are required by the helpers.

The extended_vars.yaml file is updated when issuing ocis_helpers <version> rogue.

This file is very sensitive, so you need to make sure you know what you're doing when you change its contents. Do not experiment. A backup of this file has been created. If anything goes wrong, either simply discard the changes or delete the modified file and run the relevant helper again. It will then be recreated from the backup. Ontop, the changes from the execution will be applied.

• These envvars are defined by parsing for go.GetEnv in the ocis code.
  Because there is no standardisation in how developers implement this, a lot of knowledge is required to understand the implementation, intention and the interpretation of the changes in the updated file. Note that test ocis code is usually in the /tests folder, which is excluded from gathering. However, there may be a requirement for tests in a service directly. This creates an additional entry that needs to be flagged for exclusion.

• The YAML file is never overwritten except the foundincode key, but rather extended by environment variable data found in the code.
  Therefore, you must check for validity and manipulate any updates generated manually before generating the adoc file from it. The resulting adoc file is renderd as it is. No more validation is made.

• Please note that new folders are occasionally added to the ocis repository that do not need to be included in the gathering.
  These folders can be excluded via the exclude_paths constant in the extractor_extended_env.go code. As a rule of thumb, any successful change to the YAML file should be reflected in the extended_vars.yaml.do_not_delete file to make it easier to recover the file in case of accidental deletion or corruption. This recration is done automatically if the file is missing in the envvars folder!

• A change is triggered in three cases on folders parsed:

  • go.GetEnv code is identified
  • The line number of an existing go.GetEnv has changed because of code changes
  • A formerly added block is no longer "valid" because the rawname key is no longer present

Example block:

- rawname: _registryEnv
  path: ../../ocis/ocis-pkg/registry/registry.go:88
  foundincode: true
  name: MICRO_REGISTRY
  type: string
  default_value: nats-js-kv
  description: 'The Go micro registry type to use. Supported types are: ''memory''
    and ''nats-js-kv'' (default). Only change on supervision of ownCloud Support.'
  do_ignore: false

The above snippet shows the structure, as well as the tasks required for manual manipulation:

• The do_ignore key is very important.
  If this key is set to true, the follow-up process will add it to the extended .adoc file that is generated. It will then show up in the docs.
  If there are multiple similar blocks with the same rawname key, only one should be set to true and configured. The others must not be removed, but set to false to exclude them from further processing.

• The foundincode key indicates whether rawname (a block) has been removed from the ocis code.
  If the result is false, check for a configured block with the same rawname where just the path key differs. If one is found, update from the found block the line number respectively the foundincode key. Then remove the block with foundincode false.

• While rawname defines the variable used internally in the code,
  name defines the environment variable that is used to configure ocis. Somtimes, but not always, it is directly defined via go.GetEnv.

  • Please note that a developer can use a variable for the name which requires in-depth investigation what the environment variable name is.
  • If the name is "strange", check with the development team whether this environment variable should be excluded from the documentation.

• The type, default_value and description fields cannot be populated automatically. You must check with the development team which values to provide.

• The path key indicates the file and line number where this environment varaible has been found.

  • If this environment variable's line number has changed within the same file, a new block is added because the helper cannot distinguish the cause. Therefore, a manual check of the ocis code is required to identify the valid block. Once the correct line number has been identified, simply update the line number of the block with populated values and remove the added block.
  • If the environment variable has been found in multiple files, set do_ignore to true in all duplicate blocks except the one containing populated values. Do not delete the blocks — they will be regenerated on the next run with doingnore false!

• After fixes have been applied, you must re-run the rogue helper to check that the fixes were successful. You will know if it was successful if formerly added blocks are not added again.

Once all changes have been completed, the following tasks needs to be carried out:

• Run ocis_helpers <version> extended to generate the adoc files for extended environment variables. Check the result.
• Copy the contents of the updated extended_vars.yaml file to ocis_helpers/extended_vars.yaml.do_not_delete
  This will make life easier when the file is accidentally corrupted or deleted for that version, as it will allow you to recover it.

If the extended_vars.yaml file has been corrupted or accidentally deleted, it can be recovered. Simply discard the changes or delete the corrupted version. The Go code will then check for its absence and recover it on the next helper run using the extended task. Any changes will then be applied to the recovered file. IMPORTANT: Both files MUST be present in a version branch!

• Please note that the ocis_helper directory is part of the repo and generally available for all ocis versions to be processed.
• IMPORTANT: The version folders that you are working on must be the same in both the documentation and the ocis repository. DO NOT MIX VERSION BRANCHES.
  Examples:

  docs      ocis
  
  8.3       master       OK (8.3 in docs as prerelease)
  8.2       stable-8.2   OK
  8.1       stable-8.1   OK
  8.3       stable-8.2   WRONG
  

First switch to the relevant branches in both repos. The versions must match!

As there may be changes to Go dependencies during development or between versions, updating the Go dependencies may be necessary. When running ocis_helpers <version> <task>, you will be notified by the following message: go: updates to go.mod needed; to update it. Run go mod tidy to update the dependencies for this version. These changes are tracked and must be committed and merged for that version. This is one of the reasons why versions must match.

• In ocis/master, dependency changes may happen from time to time.
• In ocis/stable-<version>, dependency changes can happen when there are backports from master for whatever reason. Usually, there will be no dependency change for this version post creating the branch, but this is not guaranteed.

IMPORTANT: Do not delete tools.go and always check the result of go mod tidy.
go mod tidy only keeps dependencies that are imported by files of this module. The ocis packages, however, are imported by the intermediate go code which is generated into the output folder that lies outside of this module. Without tools.go, which lists those imports for go mod tidy to see, the requirement gets dropped and every task using the intermediate code fails with:

module github.com/owncloud/ocis/v2 provides package
github.com/owncloud/ocis/v2/services/<service>/pkg/config/defaults and is replaced
but not required; to add it: go get github.com/owncloud/ocis/v2

• After running go mod tidy, go.mod must still contain a require github.com/owncloud/ocis/v2 <version> line. If it does not, the tidy run has stripped it, see above.
• If the ocis version processed adds a new service, add the .../pkg/config/defaults package of that service to the import list in tools.go. Else go mod tidy may drop dependencies that only the new service needs.

The way in which the helpers are used differs slightly depending on which version they are used for.

IMPORTANT

• Before using any ocis helpers, create a docs working branch for the version to which you want to apply the changes.
• Switch to the relevant branch in the ocis repo and pull for any updates.

A docs ocis prerelease is a folder that is named like a regular version but is marked for Antora to be a prerelease. It relates to the content in ocis/master.

The following tasks need to be issued:

• Regularly run ocis_helpers <version> services, especially when there are changes to environment variables in the ocis repo.

  • This will create all service dependent adoc and yaml files for environment variables. Note that the adoc files may still contain a placeholder for the introduction version.

• Before a new stable-x.y is created:

  • Use the identify_envvar_placeholder_names.sh script to check if any environment variables in the ocis repo need to be updated to regular SemVer for the upcoming stable release.
    • Please note that the version to be set must be in the format <major>.<minor>.0.
      The patch release is always set to 0 (zero)! Any issues found must be updated in the ocis repo immediately - before the stable branch is created.
    • Please note, that before you can continue, the changes in ocis must be merged and backported to the stable branch if required!
  • ocis_helpers <version> services
  • ocis_helpers <version> rogue and
    ocis_helpers <version> extended
    These tasks will catch all envvars that are not defined in a service and create the adoc file from it.
    Careful follow the Notes to the extended_vars.yaml File to complete this task.

• Please note that any environment variables that have been added, deprecated or removed are not part of the master branch. If data is shown in the documentation, it probably refers to the most recent stable version branch.

• Run ocis_helpers <version> copy to copy all generated data to its final destination.

• Before merging the changes, a docs build must be issued to ensure that all changes are picked up by the Antora build process. Do any fixes if an error occurs.

• The version must be checked to see if the identify_envvar_placeholder_names.sh script has missed anything or developers have added changes that are not covered.

  • To do so, re-run the script. If it finds any envvar with a non-compliant SemVer set, proceed as described in the master branch description above.
  • Please note, that before you can continue, the changes in ocis must be merged and backported to the master branch if required!

• Continue to redo the additional steps from the Before block above. This ensures nothing gets missed or overseen.

  • The services tasks is important to be issued and can be run at any time if for example a new envvar is changed in ocis master and backported to this version branch. By doing so, the env_vars.yaml file will be updated which is the basis for added/deprecated and removed envvars.
  • The rogue task can be used, but this is not essential. Extended envvars should not be changed after a stable branch has been created from the master branch. If a change is recognised, follow the above description to complete it.

• See the next section on how to generate added, deprecated and removed (delta) envvar adoc files.

  • IMPORTANT
    • The process requires that all env_vars.yaml files compared are up-to-date.
    • To generate delta files, the process reads the env_vars.yaml file of both versions compared from the local content/ocis/<version>/services folders.

• Run ocis_helpers <version> copy to copy all generated data to its final destination.

• Before merging the changes, a docs build must be issued to ensure that all changes are picked up by the Antora build process.

Please note that whenever a change is applied to the ocis stable branch that affects environment variables, the procedure needs to be redone!

The delta file creation relies on a file named service_names.yaml located in this folder. There you find a list of keys that represent the first part of a service dependent envvar such as ACTIVITYLOG_ (note the trailing underscore) which could then match any envvar such as ACTIVITYLOG_STORE_ENABLE_TLS. This key is queried and if matched, the path and name values are used to pre-polulate the xref: in the delta file. path and name are used as follows: xref:{s-path}/<path>.adoc[<name>]. If the key cant be identified, placeholders are used and you are notified to fix it. This is usually ONLY necessary if a new service is introduced.

To create the added, deprecated and removed environment variables adoc files for a version, the following prerequisites must be met and steps must be taken:

• All versions compared must have an actual env_vars.yaml file present in the respective ocis docs version. If a file is out-of-date, you must run the services task for that version, for details see above.

• Check the versions set in the delta_config.yaml file of the version you are processing and adapt if required.
  The file is located in content/ocis/<version>/services/persistent_files/. Note that the versionNew key must match the version you pass on the command line, else the task aborts.

  • Any changes of the yaml config must be part of the commit. Which eases a possible post-merge re-run.

• Run ocis_helpers <version> deltas.

  • This will create the required files in the services/env_var_deltas/ subfolder.
  • Use the -d flag (dryrun) beforehand to check if additional exclusion from backported introduction versions need to be applied. If the TO version is e.g. 8.1.0 but you also find 8.2.0 entries, you need to exclude them additionally in extraExcludePattern (not in defaultExcludePattern !).
  • Use the -v flag (verbose) to print the merged exclude patterns.
  • Note that if placeholders are found that have not been catched with the bash script or if there are patch releases for introduction versions other than 0 (zero), you must fix them in the ocis branches (plural) first and recreate + merge the changed env_vars.yaml file before restarting.

• Adapt the content of the output:

  • Check for any occurrences of xxx and yyy respectively you are notified on the command line.
    Consider that xref mandatory requires a resolvable path which will latest pop up on error when building the docs.
  • Group all envvars that belong to the same service and remove the xref directive for all but the first. This makes reading much easier.

• Before merging the changes, run the copy task and mandatory do a docs build to ensure that all changes are correctly rendered.

Note that on any consecutive script run, any manual changes made are overwritten. You need to re-apply them!

When developing ocis_helpers, run go build to create a new executable file and merge it to this repo.