PKG_COMP(8)

NAME
pkg_comp - builds pkgsrc packages in a chroot environment

SYNOPSIS
pkg_comp [-c config_name] [-o variable=value] auto [-f] [pkg1 .. pkgN]
pkg_comp [-c config_name] [-o variable=value] bootstrap
pkg_comp [-c config_name] [-o variable=value] build pkg1 [.. pkgN]
pkg_comp [-c config_name] [-o variable=value] config
pkg_comp [-c config_name] [-o variable=value] fetch
pkg_comp [-c config_name] [-o variable=value] sandbox-<command> [arg1 .. argN]

DESCRIPTION
pkg_comp is a utility to build pkgsrc packages in a fully automated and self-contained manner. The process is made unattended by relying on a configuration file that prespecifies the location of all build components, the environment in which to build the packages (aka the layout of the chroot sandbox), and the desired goals of the build. If you want to plug pkg_comp into cron(8), please take a look to pkg_comp4cron(8) now.

pkg_comp can be seen as a wrapper over cvs(1) and git(1), the pbulk infrastructure provided by pkgsrc, and sandboxctl(8). pkg_comp provides the following additional features:

•   Automatically fetch or update the pkgsrc tree before performing a build.

•   Preconfigure the way binary packages are built (file system layout, package options, etc.) by storing all relevant details in a configuration file.

•   Preconfigure a clean environment (chroot sandbox) in which the packages are built to avoid system-wide side-effects and contamination.

•   Trivially set up periodic pkgsrc rebuilds by adding a single line to your crontab(5).

As you can see in the SYNOPSIS section, pkg_comp provides a subcommand-interface: the tool has several modes of operation, and the particular mode to use for a given run is selected by the first non-option argument in the command line.

The following options apply to all commands:

-c config_name
    Specifies the configuration file to use. The format of the file is described in pkg_comp.conf(5).

    If config_name includes any directory separator (aka, one or more slashes) or the ‘.conf’ suffix, then this specifies the path of the configuration file to load.

    If config_name is a plain name without any directory components nor extension, then this specifies the name of the configuration. In this case, pkg_comp will use /usr/pkg/etc/pkg_comp/<config_name>.conf, which must exist; otherwise, the tool exits with an error.

-o variable=value
    Applies an override to the loaded configuration.

    The variable part of the argument must be any of the recognized configuration variables described in pkg_comp.conf(5). The value, if not empty, specifies the value to set the configuration variable to. If value is empty, then the configuration variable is unset.

  The auto command
The auto command provides the main functionality of pkg_comp as it orchestrates all other subcommands in an unattended manner. This command allows you to trivially rebuild all the packages you are interested in at will, even as a periodic cron(8) job; see pkg_comp4cron(8) if you are interested in doing this.

The auto command performs the following steps:

1.  If UPDATE_SOURCES is true, the fetch command is invoked first. See The fetch command for more details.

2.  If the sandbox does not exist yet, creates the sandbox by invoking the sandbox-create command.

3.  Bootstraps pkgsrc and pbulk within the sandbox by invoking the bootstrap command. See The bootstrap command for more details.

4.  Builds the set of packages given either in the AUTO_PACKAGES configuration variable or on the command line by passing those to the build command. See The build command for more details.

5.  If the sandbox was created by this command, destroys the sandbox by invoking the sandbox-destroy command.

The following options apply only to the auto command:

-f  Enables fast mode, which skips updating the source tree. This is a shorthand for these generic flags: -o UPDATE_SOURCES=false.

  The bootstrap command
The bootstrap command prepares the pkgsrc infrastructure within the sandbox to be ready to build packages. This involves two steps:

1.  Bootstraps pkgsrc and installs pbulk under the sandboxes' /pkg_comp/pbulk tree. This installation is used internally by pkg_comp thus is not configurable.

2.  Bootstraps pkgsrc in the location configured by the user via the LOCALBASE et. al. configuration variables.

On completion, the packages directory configured by the PACKAGES configuration variable will contain the binary bootstrap kit and any bootstrap-related binary packages for the target environment, and PBULK_PACKAGES will contain the same artifacts but for the pbulk environment used internally by pkg_comp.

Any existing binaries within the PBULK_PACKAGES tree will be reused on subsequent bootstrap operations if found. If you want to start builds afresh (and you should if you experience strange build failures), you will have to remove the previous binary packages.

  The build command
The build command triggers the build of a set of packages within the sandbox via pbulk. The sandbox must have already been created with the sandbox-create command, but it need not have been bootstrapped yet; create takes care of bootstrapping if necessary.

The packages to build can be provided either as bare names, such as ‘tmux’, or as category/package pairs, such as ‘misc/tmux’.

On completion, the binary packages are left in the directory specified by the PACKAGES configuration variable, which can later be fed to other tools such as pkg_add(8) or pkgin(8) to install the resulting binary packages.

Note that, due to the way pbulk works, this command will build the packages provided on the command line and all other packages previously built within the sandbox. This is necessary to keep fully-populated pkg_summary(5) databases. However, because pbulk reuses existing binary packages and only rebuilds updated packages, any packages not given on the command line should "build" quickly.

In case of build problems, detailed logs for each package that failed to build will be located in the directory pointed at by PBULK_LOG. Of special interest is the SUMMARY subdirectory, which contains summary reports of the whole operation in various different formats. Examples are: report.html and report.txt, both of which summarize how many packages were built and include details of all that failed.

  The config command
The config command dumps the loaded configuration to the standard output. The format of the output is not a script, so it cannot be fed back into pkg_comp. The purpose of this command is to aid in debugging the configuration of the tool before performing any builds, particularly when the configuration files use shell logic to determine the value of any variables.

The output contains two separate "paragraphs": the first lists all options specific to pkg_comp, including those given on the command line and in pkg_comp.conf(5); and the second lists all options passed to sandboxctl(8) as specified in the configuration file pointed at by SANDBOX_CONFFILE.

  The fetch command
The fetch command downloads or updates the pkgsrc tree.

If the tree does not exist yet in the location specified by PKGSRCDIR, this performs an initial checkout using the version control system of choice; otherwise, this updates the local copy, preserving any changes that may exist. Note that local changes that conflict with remote changes will cause the fetch operation to fail.

The CVS_ROOT, CVS_TAG, FETCH_VCS, GIT_BRANCH and GIT_URL variables are used to determine where to get the sources from and whether a particular branch is desired.

  Sandbox manipulation commands
The ‘sandbox-<command>’ family of commands are simple wrappers around sandboxctl(8) using the sandbox configuration specified in the SANDBOX_CONFFILE configuration variable.

FILES

/usr/pkg/etc/pkg_comp/
    Directory containing all system-wide configuration files.

/usr/pkg/etc/pkg_comp/default.conf
    Default configuration file to load if the -c flag is not provided.

/usr/pkg/etc/pkg_comp/extra.mk.conf
    Sample contents for possible free-form mk.conf(5) extensions. This file is explicitly referenced from default.conf by the EXTRA_MKCONF configuration variable.

/usr/pkg/etc/pkg_comp/sandbox.conf
    Sample contents for the sandboxctl(8) configuration file to use in the build of packages. This file is explicitly referenced from default.conf by the SANDBOX_CONFFILE configuration variable.

EXAMPLES
The following examples assume that you have already configured pkg_comp.conf(5) and its corresponding sandbox.conf(5) file.

To build a collection of packages from scratch, on a system in which you may not yet have pkgsrc nor an existent sandbox:

    # pkg_comp auto bash emacs sudo tmux

To manually control the lifecycle of the sandbox, the updates to the pkgsrc tree, and to build the same set of packages, you would do:

    # pkg_comp sandbox-create
    # pkg_comp fetch
    # pkg_comp build bash emacs sudo tmux
    # pkg_comp sandbox-destroy

After the commands above, and under the default configuration, you will find the resulting binary packages under /usr/pkgsrc/packages/All. All you need to do to use them via pkg_add(8) is:

    # PKG_PATH=file:///usr/pkgsrc/packages/All; export PKG_PATH
    # pkg_add sudo

SEE ALSO
cvs(1), pkg_comp.conf(5), hier(7), pkg_comp4cron(8), sandboxctl(8)

HISTORY
pkg_comp 1.x first appeared in pkgsrc on September 6th, 2002 as the pkgtools/pkg_comp package and was later moved to pkgtools/pkg_comp1 on February 12th, 2017, to make room for the 2.x series. The old code was moved aside, instead of just being replaced, because the 1.x series are completely different to the 2.x series and the configuration cannot be trivially migrated.

pkg_comp 2.x, documented here, first appeared on February 17th, 2017 as a full rewrite of the original tool. The rewrite had been in the works for years, seeing multiple iterations. The first was sometime in 2006, when I attempted to rewrite pkg_comp in the language I had just learned: Haskell. The second was sometime in 2010 or so, when I tried to rewrite the tool in Python soon after I joined Google as an SRE. The third, which is what you witness here together with the separate sandboxctl(8) tool, started in 2013 but was put on hold pretty much until 2017 due to me spending more time with family. I was finally able to make the time to complete this rewrite by shifting my schedule to start the day at 5am.

The main reasons for the 2.x rewrite, which involve a complete departure from the codebase of 1.x and its configuration interface, were the desire to make pkg_comp work across platforms, to integrate with pkgsrc's bootstrap and pbulk technologies, and to make it testable to ensure its reliability.

See /usr/pkg/share/doc/pkg_comp/NEWS for more details on the changes across releases.

AUTHORS
The pkg_comp utility was developed by Julio Merino <jmmv@google.com>.

BUGS
The following are known limitations in the current pkg_comp implementation.

•   pkg_comp unconditionally configures a pkgsrc bootstrap kit for the target location given by the user in LOCALBASE. This is intentional for simplicity and consistency across platforms but means that the pkg_add(8) family of tools and make(1) provided by the base system are ignored.

•   Even though pkgsrc supports unprivileged builds, pkg_comp currently requires root. The reason is that pkg_comp relies on a sandbox created by sandboxctl(8) to build packages, and entering a sandbox with chroot(8) requires root privileges. It would be nice to lift this restriction so that users could rebuild personal sets of binary packages under their home directory. One option would be to make the use of a sandbox optional, with the understanding that not using a sandbox would leave any current set of installed packages non-functional during the build.

•   This rewrite of pkg_comp still lacks some interesting features from the 1.x line. For example, support for kver(3) or the vulnerabilities database.