OpenBSD Ports: Working with Ports

OpenBSD Ports - Working with Ports [Handbook Index]

Introduction
Fetching the Ports Tree
Configuration of the Ports System
Searching the Ports Tree
Straightforward Installation: A Simple Example
Cleaning Up After a Build
Uninstalling a Port's Package
Using Flavors and Subpackages
Using dpb to Build Multiple Ports
Security Updates (-stable)
Package Signatures
Reporting Problems
Debug Packages, Debuggers and Backtraces
Helping Us

Introduction

The ports tree is meant for advanced users. Everyone is encouraged to use the pre-compiled binary packages. If you have questions about the ports tree, it is assumed that you have read the manual pages and this FAQ, and that you are able to work with it.

The ports tree is a set of Makefiles, one for each third party application, that controls:

Where and how to fetch the source of the software
Which other software it depends upon
How to alter the sources (if necessary)
How to configure and build it
How to test it (optional)
How to install it

Apart from the Makefile, each port also contains at least the following:

PLIST - instructions for package creation once the application has been built
DESCR - description of the application
distinfo - distribution file checksums and size

All this information is kept in a directory hierarchy under /usr/ports. This hierarchy contains three special subdirectories:

distfiles/ - where software distribution sets are stored after downloading
infrastructure/ - all the necessary scripts and makefiles for the ports infrastructure
packages/ - binary packages built by the ports system

The other subdirectories all form different application categories, which contain the subdirectories of the actual ports. Complex ports may be organized to an even deeper level, for example if they have a core part and a set of extensions, or a stable and a snapshot version of the application. Every port directory must contain a pkg/ subdirectory with packing list(s) and description file(s). There may also be patches/ and files/ subdirectories, for source patches and additional files, respectively.

When a user issues make(1) in the subdirectory of a specific port, the system will recursively walk its dependency tree, check whether the required dependencies are installed, build and install any missing dependencies, and then continue the build of the desired port. All of the building happens inside the working directory that the port creates. Normally this is under ${WRKOBJDIR}, defaulting to /usr/ports/pobj, but you may override this (see configuration of the ports system).

The ports tree is tied to OpenBSD's flavors. Do not check out a -current ports tree and expect it to work on a -release or -stable system. If you follow -current, you need both a -current base system and a -current ports tree. Because no intrusive changes are made in -stable, it is possible to use a -stable ports tree on a -release system and vice versa.

Another common failure is a missing X11 installation. Even if the port you try to build has no direct dependency on X11, a subpackage of it or its dependencies may require X11 headers and libraries. Building ports on systems without X11 is not supported.

More information about the ports system may be found in these man pages:

ports(7) - describes the different stages (make targets) of port installation, the use of flavors and subpackages and some other options.
bsd.port.mk(5) - in-depth information about all the make targets, variables, the fake (installation directory) framework, etc.

Fetching the Ports Tree

Once you have decided which flavor of the ports tree you want, you can get it from different sources. The table below gives an overview of where you can find the different flavors, and in which form. An '✓' marks availability and '✗' means it is not available through that specific source.

Source	Form	Flavor
		-release	-stable	snapshots	-current
Mirrors	.tar.gz	✓	✗	✓	✗
AnonCVS	cvs checkout	✓	✓	✗	✓

Look for a file named ports.tar.gz on the mirrors.

$ cd /tmp
$ ftp https://cdn.openbsd.org/pub/OpenBSD/$(uname -r)/{ports.tar.gz,SHA256.sig}
$ signify -Cp /etc/signify/openbsd-$(uname -r | cut -c 1,3)-base.pub -x SHA256.sig ports.tar.gz

You want to untar this file in the /usr directory, which will create /usr/ports and all the directories under it.

# cd /usr
# tar xzf /tmp/ports.tar.gz

The snapshots available on the mirrors are generated daily from the -current ports tree. You will find the snapshots in the /pub/OpenBSD/snapshots/ directory. If you are using a snapshot of the ports tree, you should have installed a matching snapshot of OpenBSD. Make sure you keep your ports tree and your OpenBSD system in sync!

For more information about obtaining the ports tree via CVS, read the AnonCVS page, which contains a list of available servers and a number of examples.

Configuration of the Ports System

This section introduces some additional global settings for building ports. You can skip it, but then you will be required to perform many of the make(1) statements in later examples as root.

Because the OpenBSD project does not have the resources to fully review the source code of all software in the ports tree, you can configure the ports system to take a few safety precautions. The ports infrastructure is able to perform all building as a regular user, and perform only those steps that require superuser privileges as root (for example, the install make target). Details on how to configure privilege separation are in the bsd.port.mk(5) man page.

It is possible to use a read-only ports tree by separating directories that are written to during port building:

The working directory of ports. This is controlled by the WRKOBJDIR variable, which specifies the directory which will contain the working directories.
The directory containing distribution files. This is controlled by the DISTDIR variable.
The directory containing newly built binary packages. This is controlled by the PACKAGE_REPOSITORY variable.

For example, you could add the following lines to /etc/mk.conf:

WRKOBJDIR=/usr/obj/ports
DISTDIR=/usr/distfiles
PACKAGE_REPOSITORY=/usr/packages

If desired, you can also change the ownership of these directories to your local username and group, so that the ports system can create the underlying working directories as a regular user. Again, ports can be built as a user, but must be installed by root or with doas(1).

Searching the Ports Tree

Once you have the ports tree and the portslist package in place on your system, it becomes very easy to search for software. Just use make search key="searchkey" as shown in this example:

$ cd /usr/ports
$ doas pkg_add portslist
$ make search key=rsnapshot
Port:   rsnapshot-1.4.2
Path:   net/rsnapshot
Info:   remote filesystem snapshot utility
Maint:  Antoine Jacoutot <ajacoutot@openbsd.org>
Index:  net sysutils
L-deps:
B-deps: devel/autoconf/2.69 devel/automake/1.15 devel/metaauto net/rsync
R-deps: devel/p5-Lchown net/rsync
Archs:  any

The search result gives a nice overview of each application that is found: the port name, the path to the port, a one-line description, the port's maintainer, keywords related to the port, library/build/runtime dependencies, and architectures on which the port is known to work.

This mechanism, however, is a very basic one, which just runs awk(1) on the ports index file. A port called "sqlports" allows very fine-grained searching using SQL. It is an SQLite database, but just about any database format can be created using the ports infrastructure. The sqlports port includes the script used to generate the database, which could be used as a basis to generate databases in different formats.

Just install the sqlports package to get started. A sample session could look like this:

$ sqlite3 /usr/local/share/sqlports
sqlite> SELECT FULLPKGNAME,COMMENT FROM Ports WHERE COMMENT LIKE '%statistics%';
pg_statsinfo-3.3.0p0|monitor PostgreSQL activity & statistics
kactivities-stats-5.51.0|statistics for the KDE Activity concept
p5-Devel-Cover-Report-Clover-0.35|backend for Clover reporting of coverage statistics
mailgraph-1.14p1|RRDtool frontend for Postfix statistics
R-3.5.1p1|powerful math/statistics/graphics language
p5-Math-VecStat-0.08p0|provides basic statistics on numerical vectors
py-probstat-0.912p8|probability and statistics utilities for Python
py-statistics-1.0.3.5|port of Python 3.4 statistics module to Python 2
darkstat-3.0.719p1|network statistics gatherer with graphs
pfstat-2.5p2|packet filter statistics visualization
rtg-0.7.4p12|SNMP statistics monitoring system
slurm-0.4.3|network traffic monitor and statistics
tcpstat-1.5p0|report network interface statistics
libstatgrab-0.91p3|system statistics gathering library
p5-Unix-Statgrab-0.109p0|interface to libstatgrab system statistics library
diffstat-1.62|accumulates and displays statistics from a diff file
fragistics-1.7.0p2|Quake 3 statistics program
drupal7-google_analytics-1.2p2|add the Google Analytics web statistics to your site
dstat-0.5p0|dwm status bar statistics
kdf-4.14.3p4|KDE storage device statistics
libsysstat-0.4.1|library used to query system info and statistics
qdirstat-1.4p1|Qt-based directory statistics
sqlite>

The above is still a very basic search. With SQL, just about anything can be searched for, including dependencies, configure flags, shared libraries, etc.

Straightforward Installation: A Simple Example

For clarity's sake, let's consider a simple example: rsnapshot. This application has several build dependencies: devel/autoconf/2.69 devel/automake/1.15 devel/metaauto net/rsync The following commands assume you've configured PORTS_PRIVSEP as detailed in the bsd.port.mk manual.

$ cd /usr/ports/net/rsnapshot
$ make install
===>  Checking files for rsnapshot-1.4.2
>> Fetch https://github.com/rsnapshot/rsnapshot/archive/1.4.2/rsnapshot-1.4.2.tar.gz
100% |**************************************************|   365 KB    00:02
>> (SHA256) rsnapshot-1.4.2.tar.gz: OK
===> rsnapshot-1.4.2 depends on: metaauto-* - not found
===>  Verifying install for metaauto-* in devel/metaauto
===>  Checking files for metaauto-1.0p1
[...]
===>  Installing metaauto-1.0p1 from /build/source/ports/packages/amd64/all/
metaauto-1.0p1: ok
===> Returning to build of rsnapshot-1.4.2
===> rsnapshot-1.4.2 depends on: metaauto-* -> metaauto-1.0p1
===> rsnapshot-1.4.2 depends on: autoconf-2.69 - not found
[...]
===> Returning to build of rsnapshot-1.4.2
===>  Extracting for rsnapshot-1.4.2
===>  Patching for rsnapshot-1.4.2
[...]
===>  Compiler link: clang -> /usr/bin/clang
===>  Compiler link: clang++ -> /usr/bin/clang++
===>  Compiler link: cc -> /usr/bin/cc
===>  Compiler link: c++ -> /usr/bin/c++
===>  Generating configure for rsnapshot-1.4.2
===>  Configuring for rsnapshot-1.4.2
[...]
===>  Building for rsnapshot-1.4.2
[...]
===>  Faking installation for rsnapshot-1.4.2
[...]
===>  Building package for rsnapshot-1.4.2
[...]
Link to /build/source/ports/packages/amd64/all/rsnapshot-1.4.2.tgz
Link to /build/source/ports/packages/amd64/ftp/rsnapshot-1.4.2.tgz
Link to /build/source/ports/packages/amd64/cdrom/rsnapshot-1.4.2.tgz
===> rsnapshot-1.4.2 depends on: p5-Lchown-* - not found
[...]
===>  Installing rsnapshot-1.4.2 from /build/source/ports/packages/amd64/all/
rsnapshot-1.4.2: ok

As you can see, the ports system is doing many things automatically. It will fetch, extract, and patch the source code, configure and build (compile) the source, install the files into a fake directory, create a package (corresponding to the packing list) and install this package onto your system (usually under /usr/local/). And it does this recursively for all dependencies of the port. Notice the "===> Verifying install for ..." and "===> Returning to build of ..." lines in the above output, indicating the walk through the dependency tree.

If a previous version of the application you want to install was already installed on your system, you can use make update instead of make install. This will call pkg_add(1) with the -r flag.

Large applications will require a lot of system resources to build. If you get "out of memory" type of errors when building such a port, this usually has one of two causes:

Your resource limits are too restrictive. Adjust them with ksh's ulimit or csh's limit command.
You simply don't have enough RAM.

Cleaning Up After a Build

You probably want to clean the port's default working directory after you have built the package and installed it.

$ make clean
===>  Cleaning for rsnapshot-1.4.2

By default, workdir dependencies are automatically cleaned. You can remove installed packages that were only needed for building with

# pkg_delete -a
p5-Module-Build-0.4224: ok
autoconf-2.69p2:automake-1.15.1: ok
autoconf-2.69p2: ok
metaauto-1.0p1: ok
Read shared items: ok

If you wish to remove the source distribution set(s) of the port, you would use:

$ make clean=dist
===>  Cleaning for rsnapshot-1.4.2
===>  Dist cleaning for rsnapshot-1.4.2

In case you have been compiling multiple flavors of the same port, you can clear the working directories of all these flavors at once using:

$ make clean=flavors

You can also clean everything up after each build by changing BULK from auto to Yes.

$ make package BULK=Yes

Uninstalling a Port's Package

It is very easy to uninstall a port:

$ make uninstall
===> Deinstalling for rsnapshot-1.4.2
rsnapshot-1.4.2: ok
Read shared items: ok

This will call pkg_delete(1) to have the corresponding package removed from your system. If you would like to get rid of the packages you just built, you can do so as follows:

$ make clean=packages
===>  Cleaning for rsnapshot-1.4.2
rm -f /usr/ports/packages/amd64/all/rsnapshot-1.4.2.tgz

Using Flavors and Subpackages

Please read the ports(7) man page, which gives a good overview of this topic. There are two mechanisms to control the packaging of software according to different needs.

The first mechanism is called flavors. A flavor usually indicates a certain set of compilation options. For instance, some applications have a "no_x11" flavor which can be used on systems without X. Some shells have a "static" flavor that will build a statically linked version. There are ports which have different flavors for building them with different graphical toolkits. Other examples include support for different database formats, different networking options (SSL, IPv6, ...), different paper sizes, etc.

Summary: You will most likely use flavors when a package has not been made available for the flavor you are looking for. In this case, you can specify the desired flavor and build the port yourself.

Most port flavors have their own working directory during compilation, and every flavor will be packaged into a correspondingly-named package to avoid any confusion. To see the different flavors of a certain port, you would change to its subdirectory and issue:

$ make show=FLAVORS

You can also look at the port's DESCR files, which explain the available flavors in more detail.

The second mechanism is called subpackages. A porter may decide to create subpackages for different pieces of the same application if they can be logically separated. You will often see subpackages for the client part and the server part of a program. Sometimes extensive documentation is bundled in a separate subpackage because it takes up a lot of disk space. Extra functionality that pulls in heavy dependencies will often be packaged separately. The porter will also decide which subpackage is the main subpackage, to be installed as a default. Other examples are extensive test suites which come with the software, separate modules with support for different things, etc.

Summary: Some ports are split into several packages. make install will only install the main subpackage.

To list the different packages built by a port, use:

$ make show=PKGNAMES

make install will only install the main subpackage. To install them all, use:

$ make install-all

To list the different subpackages available for a port, use:

$ make show=MULTI_PACKAGES

It is possible to select which subpackage(s) to install from within the ports tree. After some tests, this procedure will just call pkg_add(1) to install the desired subpackage(s).

# env SUBPACKAGE="-server" make install

The subpackages mechanism only handles packages. It does not modify any configuration options before building the port. You must use flavors for that purpose.

Using dpb to Build Multiple Ports

When you need to build more than one or two ports at a time, you can use the /usr/ports/infrastructure/bin/dpb tool. dpb(1) takes a list of ports to build and automatically builds them all in an optimal order, making use of as much parallelism as possible. It can also use multiple machines to perform the building, and produces detailed logs of the build process for troubleshooting, placed in /usr/ports/logs by default.

# /usr/ports/infrastructure/bin/dpb -P ~/localports

This command will read the list of pkgpaths in ~/localports and build all the packages. It can also install the packages after they have been built. The ~/localports file might look something like this:

net/rsync
www/mozilla-firefox
editors/vim

If you do not provide a list of ports to build on the command line or via -P or -I, dpb(1) will build all the ports in the ports tree. If run as root, dpb will automatically drop privileges to dedicated users for fetching distfiles and building ports. This is the recommended way to use it, but it can also run as a regular user. Additionally, the proot(1) utility can be used to further isolate building operations.

Security Updates (-stable)

When serious bugs or security flaws are discovered in third party software, they are fixed in the -stable branch of the ports tree. In contrast to the base system, the -stable ports tree only gets security backports for the latest release.

This means that all you need to do is make sure you check out the correct branch of the ports tree, and build the desired software from it. You can keep your tree up to date with CVS and subscribe to the ports-changes mailing list to receive security announcements related to software in the ports tree.

Package Signatures

Signatures are a good way to make sure packages are legitimate and not corrupted. OpenBSD offers official signed packages using signify(1). No extra effort is needed on the user's part to ensure packages haven't been tampered with.

If you want to build your own signed packages, you'll first need to create keys for signing.

# signify -Gns /etc/signify/my-pkg.sec -p /etc/signify/my-pkg.pub

Note the names: keys for signing packages should end in pkg.

Existing packages may be signed with pkg_sign(1) after being built.

# cd /usr/ports/packages/$(uname -p)
# pkg_sign -s signify2 -s /etc/signify/my-pkg.sec -o signed -S all

In order to install the package on another machine, the public key my-pkg.pub should be put into the /etc/signify directory on that machine.

Reporting Problems

If you have trouble with an existing port, please send an email to the port maintainer. To see the maintainer of the port, type, for example:

$ cd /usr/ports/archivers/unzip
$ make show=MAINTAINER

Alternatively, if there is no maintainer, or you can't reach them, send an email to the ports@openbsd.org mailing list.

In any case, please provide:

Your OpenBSD version, including any patches you may have applied. The kernel version is given by: sysctl -n kern.version
The version of your ports tree: if the file /usr/ports/CVS/Tag exists, provide its contents. If this file is absent, you are using the -current ports tree.
A complete description of the problem. Don't be afraid to provide details. Mention all the steps you followed before the problem occurred. Is the problem reproducible? The more information you provide, the more likely you will get help.
If the application crashes, try to provide a backtrace.

For ports which do not build correctly, a complete build transcript is almost always required. You can use the portslogger script, found in /usr/ports/infrastructure/bin, for this. A sample run might look like this:

$ mkdir ~/portlog
$ cd /usr/ports/archivers/unzip
$ make 2>&1 | /usr/ports/infrastructure/bin/portslogger ~/portlog

After this, you should have a logfile of the build in your ~/portlog directory that you can send to the port maintainer. Also, make sure you are not using any special options in your build, for example in /etc/mk.conf.

Alternatively, you can:

Use script(1) to create a complete build transcript.
Attach the output of pkg_info(1) if it seems even remotely relevant.
gcc(1) internal compiler errors ask you to report the bug to the gcc mailing list. It does save time if you follow their direction, and provide at least the various files produced by gcc -save-temps.

Debug Packages, Debuggers and Backtraces

If a program crashes, obtaining a backtrace will often help point to where the problem lies. Normally this is done with GDB; an old version of this is included in the base OS, but it often doesn't work well with current compilers. For package debugging, install the newer version from ports which provides the egdb binary:

# pkg_add gdb

For some ports on some machine architectures, debug packages are available (for example, if the main port is neomutt-20201127, the debug package will be debug-neomutt-20201127). These contain debug symbols which have been separated into a different file; GDB knows how to load it automatically. The debug package must match the main package. If you are using snapshots, you may need to reinstall to ensure that they are from the same build. Install debug packages with pkg_add as usual:

# pkg_add debug-neomutt

If a debug package is not available, you will likely be able to see function names (but not exact lines) from a coredump; sometimes this is good enough to point at a problem.

A port can be built with debug symbols on an ad-hoc basis (useful on an architecture where debug packages are not normally built) by building the port with make DEBUG=-g. You will need to clean the existing build and packages first (and may sometimes have problems building unless you uninstall first). If the package of interest is a library or depended on by other ports, you may have to uninstall the whole chain to do this. It may also be useful to disable optimization (add -O0 to DEBUG).

$ cd /usr/ports/math/moo
$ make clean; make DEBUG="-g" repackage; make reinstall

If you have a core dump file, point GDB at it and obtain a backtrace:

$ egdb neomutt neomutt.core
GNU gdb (GDB) 7.12.1
Copyright (C) 2017 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-openbsd6.8".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/local/bin/neomutt...Reading symbols from /usr/local/bin/.debug/neomutt.dbg...done.
done.
[New process 577437]
Core was generated by `neomutt'.
Program terminated with signal SIGABRT, Aborted.
#0  _thread_sys_poll () at /tmp/-:3
3       /tmp/-: No such file or directory.
(gdb) bt
#0  _thread_sys_poll () at /tmp/-:3
<...>
#9  0x00000b3f1d5ec847 in main (argc=<optimized out>, argv=0x7f7ffffc8a08, envp=<optimized out>) at ../neomutt-20201127/main.c:1236
(gdb) q

For setuid/setgid programs, core dumps are disabled by default, see sysctl(8) for a way to allow them. Alternatively you can load the program with the debugger, set arguments as needed, and run:

$ egdb neomutt
<...>
(gdb) set args -d 2
(gdb) run
<...>

You will be returned to the debugger when certain signals are received (in most cases, at the point where it would crash).

To enable debug packages in the normal bulk build for a typical port, add "DEBUG_PACKAGES=${BUILD_PACKAGES}" to the Makefile. This will set DEBUG during the build, and during the packaging stage will automatically separate the debug symbols from the main binaries. It will also try to compress the symbols with dwz; this step sometimes fails; if so, it can be overridden with "DWZ=:". If you are regularly doing ports development work, setting "INSTALL_DEBUG_PACKAGES=Yes" in /etc/mk.conf will install newly built debug packages automatically.

Helping Us

There are many ways you can help. They are listed below, by increasing order of difficulty.

Report problems as you experience them.
You can systematically test ports and report breakages, or suggest improvements. Just have a look at the Port Testing Guide. You must run a -current system for creating and testing ports.
Test the updates to ports which are posted to the ports mailing list.
Send updates or patches to a port's maintainer, or to the ports mailing list if the port has no maintainer. Generally this is highly appreciated, unless your patches will cause developers to waste time rather than save time.
Create new ports.

Hardware donations can assist testing ports on the various platforms OpenBSD runs on.