Site Configuration Modules
==========================

A build farm is normally run via the `installers', `site', or
`snapshot-site' target of the Racket repository's top-level
makefile. Each of those targets uses `installers', which expects a
`CONFIG=...' argument to specify a configuration module file (or uses
"build/site.rkt" as the default).

A site configuration module starts `#lang distro-build/config' and
uses keywords to specify various options for the configuration. This
format is described is detail further below; for now, it's enough to
know that there are various options, each of which is associated with
a keyword.

The machine where `make installers' is run is the server machine.
The server machine first prepares packages for installation on
clients.  The site configuration's top-level entry is consulted for
a `#:pkgs' and/or `#:doc-search' option, which overrides any `PKGS'
and/or `DOC_SEARCH' configuration from the makefile.

The site configuration file otherwise describes and configures
client machines hierarchically, where configuration options
propagate down the hierarchy when they are not overridden more
locally.

Each client is normally built by running commands via `ssh', where the
client's host, `#:host' (with and optional `#:port' and/or `#:user')
indicate the `ssh' target. Each client machine must be set up with a
public-key authentication, because a direct `ssh' is expected to work
without a password prompt. An exception is when the host is
"localhost" and user is #f, in which case a shell is used directly
instead of `ssh'. When `ssh` is used, -R is also used to create a
tunnel back to the server, and the client by default uses that tunnel
for all communication, and the server by default accepts only
connections via "localhost".

On the client machine, all work is performed at a specified
directory as specified by `#:dir'. The directory defaults to
"build/plt" (Unix, Mac OS X) or "build\\plt" (Windows), except when
the host is "localhost" and the client is #f, in which case the
current directory (i.e., the server's directory) is used.

Normally, the client directory is a git clone:

  - If the directory exists already on a client machine (and the
    machine is not configured for "clean" mode), then if the directory
    contains a ".git" subdirectory, it is assumed to be a git clone
    and updated with `git pull'. The `git pull' operation can be
    disabled by specifying `#:pull?' as #f, and it defaults to #f
    in the case that `#:dir' is not specified, the host is
    "localhost", and the user is #f.

  - If the directory does not exist, a git repository is
    cloned. The repository can be specified with `#:repo'. By
    default, the server is used as the source git repository (so
    that the server and client are in sync), which means that the
    server's directory must be a git clone.

Note that neither `ssh' nor `git' turn out to be needed when the host
is "localhost", the user is #f, and the directory is not specified
(which corresponds to the defaults in all cases).

If a build fails for a machine, building continues on other
machines.  Success for a given machine means that its installer
ends up in "build/installers" (and failure for a machine means no
installer) as recorded in the "table.rktd" file.

To use the `site' makefile target, the configuration file must at
least provide a `#:dist-base-url' value, which is a URL at which the
site will be made available. To use the `snapshot-site' makefile
target, then `#:site-dest' will need to be specified, normally as a
path that ends with the value produced by `(current-stamp)'.

Hint: When developing a configuration file, use an empty set of
packages to a configuration that works as quickly as possible. Then,
change the list of packages to the ones that you actually want in the
installers.

Machine Requirements
--------------------

Each Unix or Mac OS X client needs the following available:

  * SSH server with public-key authentication (except "localhost")
  * git (unless the working directory is ready)
  * gcc, make, etc.

Each Windows client needs the following:

  * SSH server with public-key authentication, providing either a
    Windows command line (like freeSSHd) or bash with access to
    cmd.exe (like Cygwin's opensshd)
  * git (unless the working directory is ready)
  * Microsoft Visual Studio 9.0 (2008), installed in the
    default folder:
     C:\Program Files\Microsoft Visual Studio 9.0       (32-bit host)
     C:\Program Files (x86)\Microsoft Visual Studio 9.0 (64-bit host)
  * Nullsoft Scriptable Install System (NSIS) verstion 2.x, installed
    in the default folder:
     C:\Program Files\NSIS\makensis.exe
     or  C:\Program Files (x86)\NSIS\makensis.exe
    or installed so that `makensis' in your PATH.

Site Configuration
-------------------

A site configuration module is normally written in the
`distro-build/config' language. The configuration describes individual
machines, and groups them with `parallel' or `sequential' to indicate
whether the machine's builds should run sequentially or in parallel.
Options specified at `parallel' or `sequential' are propagated to each
machine in the group.

Site-configuration keywords (where <string*> means no spaces, etc.):

  #:host <string*> --- defaults to "localhost"

  #:name <string> --- defaults to host; this string is recorded as a
    description of the installer and can be used in a generated table of
    installer links; see also "Names and Download Pages" below

  #:port <integer> --- SSH port for the client; defaults to 22

  #:user <string*/false> --- SSH user for the client; defaults to #f,
    which means the current user

  #:dir <path-string> --- defaults to "build/plt" or "build\\plt", or
    to the current directory if the host is "localhost" and the user
    is #f

  #:server <string*> --- the address of the server as accessed by the
    client; when ssh remote tunneling works, then "localhost" should
    work to reach the server; defaults to the `SERVER' makefile
    variable, which in turn defaults to "localhost"

  #:server-port <integer> --- the port of the server as accessed by
    the client, and also the port started on clients to tunnel back to
    the server; defaults to the `SERVER_PORT' makefile variable, which
    in turn defaults to 9440

  #:server-hosts <list-of-string*> --- addresses that determine the
    interfaces on which the server listens; an empty list means all of
    the server's interfaces, while `(list "localhost")' listens only
    on the loopback device; defaults to the `SERVER_HOSTS` makefile
    variable split on commas, which in turn defaults to `(list
    "localhost")'

  #:repo <string> --- the git repository for Racket; defaults to
    "http://<server>:<server-port>/.git"

  #:pkgs '(<string*> ...) --- packages to install; defaults to the
    `PKGS' makefile variable

  #:dist-base-url <string> --- a URL that is used to construct a
    default for `#:doc-search' and `#:dist-catalogs', where the
    constructed values are consistent with converting a build server's
    content into a download site; since URLs are constructed via
    relative paths, this URL normally should end with a slash

  #:doc-search <string> --- URL to install as the configuration for
    remote documentation searches in generated installers; "" is
    replaced with the PLT default; defaults to #:dist-base-url (if
    present) extended with "doc/local-redirect/index.html", or the
    `DOC_SEARCH' makefile variable

  #:install-name <string> --- string used as the name of the
    installation for package operations in the `user' package scope,
    where "" keeps the name as the Racket version; the default is
    "snapshot" if the value of `#:release?' is false, "" otherwise.

  #:build-stamp <string> --- a string representing a build stamp,
     recorded in installes; the default is from the `BUILD_STAMP'
     makefile variable or generated if the value of `#:release?' is
     false, "" otherwise.

  #:dist-name <string> --- the distribution name; defaults to the
    `DIST_NAME' makefile variable

  #:dist-base <string*> --- the distribution's installater name prefix;
    defaults to the `DIST_BASE' makefile variable

  #:dist-dir <string*> --- the distribution's installation directory;
    defaults to the `DIST_DIR' makefile variable

  #:dist-suffix <string*> --- a suffix for the installer's name,
    usually used for an OS variant; defaults to the `DIST_SUFFIX'
    makefile variable

  #:dist-catalogs '(<string> ...) --- catalog URLs to install as the
    initial catalog configuration in generated installed, where "" is
    replaced with the PLT default catalogs; defaults to
    `#:dist-base-url' (if present) extended with "catalogs" in a list
    followed by ""

  #:readme <string-or-procedure> --- the content of a "README" file
    to include in installers, or a function that takes a hash table
    for a configuration and returns a string; the default is the
    `make-readme' function from `distro-build/readme' (see below)

  #:max-vm <real> --- max number of VMs allowed to run with this
    machine, counting the machine; defaults to 1

  #:vbox <string> --- Virtual Box machine name (as shown, for example,
    in the Virtual Box GUI); if provided, the virtual machine is
    started and stopped on the server as needed

  #:platform <symbol> --- 'unix, 'macosx, 'windows, or 'windows/bash
    (which means 'windows though an SSH server providing `bash', such
    as Cygwin's); defaults to `(system-type)'

  #:configure '(<string> ...) --- arguments to `configure'

  #:bits <integer> --- 32 or 64, affects Visual Studio path

  #:vc <string*> --- "x86" or "x64" to select the Visual C build mode;
    default depends on `#:bits'

  #:sign-identity <string> --- provides an identity to be passed to
    `codesign` for code signing on Mac OS X (for all executables in a
    distribution), where an empty string disables signing; the default
    is ""

  #:j <integer> --- parallelism for `make' on Unix and Mac OS X and
    for `raco setup' on all platforms; defaults to 1

  #:timeout <number> --- numbers of seconds to wait before declaring
    failure; defaults to 30 minutes

  #:clean? <boolean> --- if true, then the build process on the client
    machine starts by removing <dir>; set this to #f for a shared repo
    checkout; the default is determined by the `CLEAN_MODE' makefile
    variable, unless `#:host' is "localhost", `#:user' is #f, and
    `#:dir' is not specified, in which case the default is #f

  #:pull? <boolean> --- if true, then the build process on the client
    machine starts by a `git pull' in `#:dir'; set to #f, for example,
    for a repo checkout that is shared with server; the default is #t,
    unless `#:host' is "localhost", `#:user' is #f, and `#:dir' is not
    specified, in which case the default is #f

  #:release? <boolean> --- if true, then create release-mode
    installers; the default is determined by the `RELEASE_MODE'
    makefile variable

  #:source? <boolean> --- determines the default value for
    `#:source-runtime?' and `#:source-pkgs'

  #:source-runtime? <boolean> --- if true, then create an archive that
    contains the run-time system in source form (possibly with built
    packages), instead of a platform-specific installer; a #t value
    works best when used with a Unix client machine, since Unix
    clients typically have no native-library packages; the default is
    the value of `#:source?'

  #:source-pkgs? --- if true, then packages are included in the
    installer/archive only in source form; a true value works best
    when the `#:source-runtime?' value is also #t; the default is the
    value of `#:source?'

  #:mac-pkg? --- if true, creates a ".pkg" for Mac OS X (in
    single-file format) instead of a ".dmg"; the default is #f

  #:max-snapshots <number> --- number of snapshots to keep, used by
    the `snapshot-site' makefile target

  #:pause-before <nonnegative-number> --- a pause in seconds to
    wait before starting a machine, which may help a virtual machine
    avoid confusion from being stopped and started too quickly; the
    default is 0

  #:pause-after <nonnegative-number> --- a pause in seconds to
    wait after stopping a machine;  the default is 0

  #:custom <hash-table> --- a hash table mapping arbitrary keywords to
    arbitrary values; when a value for `#:custom' is overriden in a
    nested configuration, the new table is merged with the overriden
    one; use such a table for additional configuration entries other
    than the built-in ones, where additional entires may be useful to
    a `#:readme' procedure

Top keywords (recognized only in the configuration top-level):

  #:site-dest <path-string> --- destination for completed build, used
    by the `site' and `snapshot-site' makefile targets; the default is
    "build/site"

  #:pdf-doc? <boolean> --- whether to build PDF documentation when
    assembling a site; the default is #f

  #:email-to <listof-of-string> --- a list of addresses to receive
     e-mail reporting build results; mail is sent via `sendmail'
     unless `#:smtp-...' configuration is supplied

  #:email-from <string> --- address used as the sender of e-mailed
    reports; the first string in `#:email-to' is used by default

  #:smtp-server <string*>
  #:smtp-port <string*>
  #:smtp-connect <'plain, 'ssl, or 'tls>
  #:smtp-user <string-or-#f>
  #:smtp-password <string-or-#f>
    --- configuration for sending e-mail through SMTP instead of
    `sendmail'; the `#:smtp-port' default (25, 465, or 587) is picked
    based on `#:smtp-connect', which in turn defaults to 'plain;
    supply non-#f `#:smtp-user' and `#:smtp-password' when
    authentication is required by the server

  #:site-help <hash-table> --- hash table of extra "help" information
    for entries on a web page created by the `site' and
    `snapshot-site' makefile targets; the hash keys are strings for
    row labels in the download table (after splitting on "|" and
    removing "{...}"), and the values are X-expressions for the help
    content

More precisely, the `distro-build/config' language is like
`racket/base' except that the module body must have exactly one
expression (plus any number of definitions, etc.) that produces a
site-configuration value. The value is exported as `site-config'
from the module. Any module can act as a site-configuration module
a long as it exports `site-config' as a site-configuration value.

The `distro-build/config' language also adds the following functions
to `racket/base':

 (machine <opt-kw> <opt-val> ... ...) -> site-config?
   Produces a site configuration based on the given keyword-based
   options. The support keyword arguments are described above.

 (sequential <opt-kw> <opt-val> ... ... config ...)
   -> site-config?
    config : site-config?
   Produces a site configuration that runs each `config'
   sequentially. The support keyword arguments are described above.

 (parallel <opt-kw> <opt-val> ... ... config ...)
   -> site-config?
    config : site-config?
   Produces a site configuration that runs each `config' in
   parallel. The support keyword arguments are described above.

 (site-config? v) -> boolean?
 (site-config-tag config) -> (or/c 'machine 'sequential 'parallel)
    config : site-config?
 (site-config-options config) -> (hash/c keyword? any/c)
    config : site-config?
 (site-config-content config) -> (listof site-config?)
    config : site-config?
  Site configuation inspection

 (current-mode) -> string?
 (current-mode s) -> void?
    s : string?
  A parameter whose value is the user's requested mode for this
  configuration, normally as provided via the makefile's
  `CONFIG_MODE' variable. The default mode is "default". The
  interpretation of modes is completely up to the
  site configuration file.

 (current-stamp) -> string?
  Returns a string to identify the current build, normally a
  combination of the date and a git commit hash.

READMEs
-------

The `distro-build/readme' library provides functions for constructing
a README file's content. Each function takes a hash table mapping
configuration keywords to values.

 (make-readme config) -> string
    config : hash?
  Produces basic "README" content, using information about the
  distribution and the Racket license. The content is constructed
  using `config' keywords such as `#:name', `#:platform',
  `#:dist-name', and `#:dist-catalogs', and sometimes `current-stamp'.

 (make-macosx-notes config) -> string
    config : hash?
  Produces "README" content to tell Mac OS X users how to install a
  distirbution folder. This function is used by `make-readme' when
  `#:platform' in `config' is 'macosx.

Names and Download Pages
------------------------

The `#:name' for an installer is used in an HTML table of download
links by the `site' or `snapshot-site' targets. The names are first
sorted. Then, for the purposes of building the table, a "|" separated
by any number of spaces within a name is treated as a hierarchical
delimiter, while anything within "{" and "}" in a hierarchical level
is stripped from the displayed name along with surrounding spaces (so
that it can affect sorting without being displayed).

For example, the names

  "Racket | {2} Linux | 32-bit"
  "Racket | {2} Linux | 64-bit"
  "Racket | {1} Windows | 32-bit"
  "Racket | {1} Windows | 64-bit"
  "Racket | {3} Source"

are shown (actually or conceptually) as

  Racket
   Windows
     [32-bit]
     [64-bit]
   Linux
     [32-bit]
     [64-bit]
   [Source]

where the square-bracketed entries are hyperlinks.

Examples
--------

** Single Installer **

The simplest possible configuration file is

  #lang distro-build/config
  (machine)

In fact, this configuration file is created automatically as
"build/site.rkt" (if the file does not exist already) and used as the
default configuration. With this configuration,

  make installers

creates an installer in "build/installers" for the platform that is
used to create the installer.

** Installer Web Page ***

To make a web page that serves both a minimal installer and packages,
create a "site.rkt" file with

 #lang distro-build/config

 (sequential
  ;; The packages that will be available:
  #:pkgs '("main-distribution")
  ;; FIXME: the URL where the installer and packages will be:
  #:dist-base-url "http://my-server.domain/snapshot/"
  (machine
   ;; FIXME: the way the installer is described on the web page:
   #:name "Minimal Racket | My Platform" 
   ;; The packages in this installer:
   #:pkgs '()))

then

 make site CONFIG=site.rkt

creates a "build/site" directory that you can move to your web server's
"snapshot" directory, so that "build/site/index.html" is the main
page, and so on.

** Accumulated Shapshots Web Page **

To make a web site that provides some number (5, by default) of
snapshots, use `(current-stamp)' when constructing the
`#:dist-base-url' value. Also, use `(current-stamp)' as the directory
for assembling the "site":

 #lang distro-build/config
 (sequential
  ;; The packages that will be available:
  #:pkgs '("gui-lib")
  ;; FIXME: the URL where the installer and packages will be:
  #:dist-base-url (string-append "http://my-server.domain/snapshots/"
                                 (current-stamp) "/")
  ;; The local directory where a snapshot is written
  #:site-dest (build-path "build/site" (current-stamp))
  (machine
   ;; FIXME: the way the installer is described on the web page:
   #:name "Minimal Racket | My Platform" 
   ;; The packages in this installer:
   #:pkgs '()))

Then,

  make snapshot-site CONFIG=site.rkt

creates a "build/site" directory that you can move to your web
server's "snapshots" directory, so that "build/site/index.html" is the
main page that initially points to "build/site/<stamp>/index.html",
and so on. To make a newer snapshot, update the git repository, leave
"build/site" in place, and run

  make snapshot-site CONFIG=site.rkt

again. The new installers will go into a new <stamp> subdirectory, and
the main "index.html" file will be rewritten to point to them.
 
** Multiple Platforms **

A configuration module that drives multiple clients to build
installers might look like this:

    #lang distro-build/config
   
    (sequential
     #:pkgs '("drracket")
     #:server-hosts '() ; Insecure? See below.
     (machine
      #:desc "Linux (32-bit, Precise Pangolin)"
      #:name "Ubuntu 32"
      #:vbox "Ubuntu 12.04"
      #:host "192.168.56.102")
     (machine
      #:desc "Windows (64-bit)"
      #:name "Windows 64"
      #:host "10.0.0.7"
      #:server "10.0.0.1"
      #:dir "c:\\Users\\racket\\build\\plt"
      #:platform 'windows
      #:bits 64))

The configuration describes using the hosts "192.168.56.1" and
"10.0.0.7" for Linux and Windows builds, respectively, which are run
one at a time. 

The Linux machine runs in VirtualBox on the server machine (in a
virtual machine named "Ubuntu 12.04"). It contacts the server still as
"localhost", and that works because the ssh connection to the Linux
machine creates a tunnel (at the same port as the server's, wjich
defaults to 9440).

The Windows machine uses freeSSHd (not a `bash'-based SSH server like
Cygwin) and communicates back to the server as "10.0.0.1" instead of
using an SSH tunnel. To make that work, `#:server-hosts' is specified
as the empty list to make the server listen on all interfaces (instead
of just "localhost") --- which is possibly less secure than the
default restriction that allows build-server connections only via
"localhost".

With this configuration file in "site.rkt",

  make installers CONFIG=site.rkt

produces two installers, both in "build/installers", and a hash table
in "table.rktd" that maps "Linux (32-bit, Precise Pangolin)" to the
Linux installer and "Windows (64-bit)" to the Windows installer.
