==============================
FreeBSD Implementation details
==============================


Disk and ZFS layout
-------------------

The implementation uses ZFS. :ref:`ceph-volume zfs <ceph-volume-zfs>` creates
the layout on an empty whole disk, without partitioning it first:

* One ZFS pool per OSD, named ``ceph-osd-<id>``. The pool is not mounted
  (``-m none``).
* Inside the pool a zvol ``osd-block-<osd fsid>`` serves as the raw block device
  for BlueStore. It shows up as
  ``/dev/zvol/ceph-osd-<id>/osd-block-<osd fsid>``. By default the zvol reserves
  its space and takes 95% of the pool.
* An optional separate RocksDB or WAL device gets a pool of its own, named
  ``ceph-osd-<id>-db`` or ``ceph-osd-<id>-wal``, with a zvol ``osd-db-<osd fsid>``
  or ``osd-wal-<osd fsid>`` inside it.
* Pools and zvols are tagged with ZFS user properties ``ceph:*``, for example
  ``ceph:osd_id``, ``ceph:osd_fsid`` and ``ceph:cluster_fsid``. This is how
  ``ceph-volume zfs list`` and ``ceph-volume zfs activate`` find the OSDs again.
* The OSD directory ``/var/lib/ceph/osd/ceph-<id>`` is not on ZFS. It lives on
  tmpfs and is rebuilt at every boot.

The result can be inspected with::

  zpool list
  zfs list -t volume

Cache and log devices can be added to the pool afterwards. They are transparent
to Ceph. Assuming that ``ada2`` is an SSD and ``ceph-osd-1`` is the pool of the
OSD::

  gpart create -s GPT ada2
  gpart add -t freebsd-zfs -l osd-1-log -s 1G ada2
  zpool add ceph-osd-1 log gpt/osd-1-log
  gpart add -t freebsd-zfs -l osd-1-cache -s 10G ada2
  zpool add ceph-osd-1 cache gpt/osd-1-cache

    ceph:osd_id       = 0
    ceph:osd_fsid     = <osd_fsid>
    ceph:managed_by   = ceph-volume-zfs
    ceph:type         = block
    ceph:zvol_name    = osd-block-<osd_fsid>

Block zvol
~~~~~~~~~~

A zvol inside the pool, named ``osd-block-<osd_fsid>``, sized to 95%
of the pool's available space (leaving headroom for ZFS metadata)::

    zfs create -V <size> ceph-osd-0/osd-block-<osd_fsid>

Exposed as ``/dev/zvol/ceph-osd-0/osd-block-<osd_fsid>`` and used as
the BlueStore block device. By default thick-provisioned (full
``refreservation``); pass ``--thin`` for sparse zvols.

The zvol carries these ZFS user properties (set via ``zfs set``)::

    ceph:osd_id             = 0
    ceph:osd_fsid           = <osd_fsid>
    ceph:cluster_name       = ceph
    ceph:cluster_fsid       = <cluster_fsid>
    ceph:type               = block
    ceph:objectstore        = bluestore
    ceph:crush_device_class = <class or empty>
    ceph:block_device       = /dev/zvol/ceph-osd-0/osd-block-<osd_fsid>

Optional db/wal zvols
~~~~~~~~~~~~~~~~~~~~~

If ``--block.db`` or ``--block.wal`` is given, additional zvols are
created **before** the block zvol so sizing uses genuinely remaining
space. Two forms are accepted:

* ``--block.db same-pool`` — db zvol carved from the same pool as the
  block device (requires ``--block-db-size``). Convenient but provides
  no I/O separation.
* ``--block.db /dev/ada2`` — dedicated pool ``ceph-osd-0-db`` on a
  separate (typically faster) device.

The db/wal zvols carry the same ``ceph:*`` properties as the block
zvol, plus cross-links so any zvol names the others::

    ceph:type         = db   (or wal)
    ceph:block_device = /dev/zvol/ceph-osd-0/osd-block-<osd_fsid>

The block zvol in turn gets::

    ceph:db_device    = /dev/zvol/<db-pool>/osd-db-<osd_fsid>
    ceph:wal_device   = /dev/zvol/<wal-pool>/osd-wal-<osd_fsid>

OSD data directory
~~~~~~~~~~~~~~~~~~

The OSD data directory (``/var/lib/ceph/osd/ceph-<id>``) is a tmpfs
mount rebuilt at activate time from the zvol's BlueStore labels::

    mount -t tmpfs tmpfs /var/lib/ceph/osd/ceph-0
    ceph-bluestore-tool prime-osd-dir \
        --path /var/lib/ceph/osd/ceph-0 \
        --dev /dev/zvol/ceph-osd-0/osd-block-<osd_fsid>

The durable state lives on the zvol itself; losing the tmpfs on reboot
is harmless and expected.


Automated OSD management (ceph-volume zfs)
------------------------------------------

``ceph-volume zfs`` is the FreeBSD-native OSD lifecycle tool. It uses ZFS
pools and zvols as the underlying block storage, replacing the Linux LVM/raw
backend with FreeBSD-native equivalents.

Each OSD consists of:

* A ZFS pool named ``ceph-osd-<id>`` created with ``-m none`` (no root dataset
  mountpoint) on a dedicated disk or partition.
* A zvol named ``osd-block-<osd_fsid>`` inside that pool, used as the
  BlueStore block device.
* Optional db and wal zvols (``osd-db-<osd_fsid>`` / ``osd-wal-<osd_fsid>``)
  either in the same pool (``--block.db same-pool``) or on a dedicated device.

All pools and zvols are tagged with ZFS user properties for discovery without
requiring a running Ceph cluster.

Subcommands::

    ceph-volume zfs inventory          # list available disks
    ceph-volume zfs prepare --data /dev/ada1
    ceph-volume zfs list               # list Ceph OSDs
    ceph-volume zfs activate --all     # activate all OSDs at boot
    ceph-volume zfs zap /dev/ada1      # dry-run wipe
    ceph-volume zfs zap --force /dev/ada1

Global flags ``-v``/``--verbose``, ``-j``/``--json``, ``--format``,
``--debug``, and ``-n``/``--dry-run`` are accepted before or after the
subcommand.

Thick-provisioned zvols are the default (prevents pool oversubscription).
Pass ``--thin`` to use sparse zvols instead.


Configuration
-------------

As per FreeBSD default, extra software installs under ``/usr/local/``.
The default Ceph configuration path is ``/usr/local/etc/ceph/ceph.conf``.
Create a symlink so the standard ``/etc/ceph`` path works::

    ln -s /usr/local/etc/ceph /etc/ceph

A sample configuration file is at
``/usr/local/share/doc/ceph/sample.ceph.conf``.


Build system
------------

FreeBSD builds use clang (the system compiler). Several GNU-ld-specific
linker flags are automatically disabled when building with clang or lld:

* ``--copy-dt-needed-entries`` is skipped (guarded by ``CMAKE_CXX_COMPILER_ID``).
* ``--no-undefined-version`` is the lld default; stale version-script entries
  are pruned from ``librados.map``.
* ``HAVE_ASM_SYMVER`` and ``HAVE_ATTR_SYMVER`` are forced off on FreeBSD to
  prevent empty-version ``sym@@`` symbols that lld rejects.

The build is driven by ``do_freebsd.sh``, which sets FreeBSD-specific cmake
options and calls ninja. Key options:

* ``WITH_LIBURING=OFF`` — io_uring is Linux-only; POSIX AIO is used instead.
* ``WITH_LIBCEPHFS_PROXY=OFF`` — Linux fscrypt dependency, disabled on FreeBSD.
* ``WITH_BLUESTORE=ON``, ``WITH_RBD=ON`` — fully supported.
* ``WITH_CEPHFS=ON`` — CephFS client (FUSE-based only; no kernel module).


MON creation
------------

Monitors are created by following the manual creation steps at::

    https://docs.ceph.com/en/latest/install/manual-freebsd-deployment/


OSD creation
------------

OSDs are created with :ref:`ceph-volume zfs <ceph-volume-zfs>`. For every disk
it creates a zpool ``ceph-osd-<id>`` that holds a zvol ``osd-block-<osd fsid>``,
which serves as the raw block device for BlueStore. Both are tagged with
``ceph:*`` ZFS properties, then the OSD is registered with the cluster and
``ceph-osd --mkfs`` is run::

  ceph-volume zfs inventory
  ceph-volume zfs prepare --data /dev/ada1

The OSD directory ``/var/lib/ceph/osd/ceph-<id>`` lives on tmpfs. After a reboot
``ceph-volume zfs activate --all`` rebuilds it from the ZFS properties, without
contacting a monitor. The ``ceph_osd`` rc.d script does this when the OSDs are
started.

The complete procedure is described in :ref:`freebsd_adding_osds`.