CARP works by allowing a group of hosts on the same network segment to share an IP address. This group of hosts is referred to as a "redundancy group." The redundancy group is assigned an IP address that is shared amongst the group members. Within the group, one host is designated the "master" and the rest as "backups." The master host is the one that currently "holds" the shared IP; it responds to any traffic or ARP requests directed towards that address. Each host may belong to more than one redundancy group at a time.
One common use for CARP is to create a group of redundant firewalls. The virtual IP that is assigned to the redundancy group is configured on client machines as the default gateway. In the event that the master firewall suffers a failure or is taken offline, the IP will move to one of the backup firewalls and service will continue unaffected.
CARP supports IPv4 and IPv6.
advbase and
advskew values).
It's possible for multiple CARP groups to exist on the same network segment. CARP advertisements contain the Virtual Host ID which allows group members to identify which redundancy group the advertisement belongs to.
In order to prevent a malicious user on the network segment from spoofing CARP advertisements, each group can be configured with a password. Each CARP packet sent to the group is then protected by an SHA1 HMAC.
Since CARP is its own protocol, it should have an explicit pass rule in filter rulesets:
pass out on $carp_dev proto carp
$carp_dev should be the physical interface that CARP is
communicating over.
# ifconfig carpN create # ifconfig carpN vhid vhid [pass password] [carpdev carpdev] \ [advbase advbase] [advskew advskew] [state state] [group|-group group] \ ipaddress netmask mask
carpN
vhid
password
carpdev
advbase
advskew
advbase when sending CARP advertisements.
By manipulating advskew, the master CARP host can be
chosen.
The higher the number, the less preferred the host will be when
choosing a master.
The default is 0.
Acceptable values are from 0 to 254.
state
init, backup,
and master.
group, -group
carp group.
Each group has a carpdemote counter affecting all carp
interfaces belonging to that group.
As described below, it can be useful to group certain interfaces together
for failover purposes.
ipaddress
mask
Further CARP behavior can be controlled via sysctl(8).
net.inet.carp.allow
net.inet.carp.preempt
advbase and advskew to preempt the master.
In addition, this option also enables failing over a group of interfaces
together in the event that one interface goes down.
If one physical CARP-enabled interface goes down, CARP will increase
the demotion counter, carpdemote, by 1 on interface groups
that the carp(4) interface is a member of, in effect causing all group
members to fail-over together.
net.inet.carp.preempt is 0 (disabled) by default.
net.inet.carp.log
# sysctl net.inet.carp.allow=1 # echo 'net.inet.carp.allow=1' >> /etc/sysctl.conf # ifconfig carp1 create # ifconfig carp1 vhid 1 pass mekmitasdigoat carpdev em0 advskew 100 10.0.0.1 netmask 255.255.255.0This sets up the following:
carp1
carp1 for virtual host #1, enables a password,
sets em0 as the interface belonging to the group, and makes
this host a backup due to the advskew of 100
(assuming that the master is set up with an advskew less
than 100)
Running ifconfig on carp1 shows the status of the
interface.
# ifconfig carp1
carp1: flags=8802<UP,BROADCAST,SIMPLEX,MULTICAST> mtu 1500
carp: BACKUP carpdev em0 vhid 1 advbase 1 advskew 100
groups: carp
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
When pfsync is set up to send and receive updates on the network, the default behavior is to multicast updates out on the local network. All updates are sent without authentication. Best common practice is either:
syncdev (see below)
syncpeer option (see
below) so that updates are unicast directly
to the peer, then configure
ipsec(4) between the hosts
to secure the pfsync traffic
pass on $sync_if proto pfsync
$sync_if should be the physical interface that pfsync is
communicating over.
# ifconfig pfsyncN syncdev syncdev [syncpeer syncpeer] [defer|-defer]
pfsyncN
syncdev
syncpeer
syncpeer.
defer
defer flag is used, the initial packet of a new
connection passing through the firewall will not be transmitted until
either another pfsync system has acknowledged the state table addition,
or a timeout has expired.
This adds small delays but allows traffic to flow when more than one
firewall might actively handle packets ("active/active"), e.g. with
certain ospfd(8),
bgpd(8), or
carp(4) configurations.
# ifconfig pfsync0 syncdev em1 upThis enables pfsync on the
em1 interface.
Outgoing updates will be multicast on the network, allowing any other
host running pfsync to receive them.
An example scenario: two firewalls, fw1 and fw2.
+----| WAN/internet |----+
| |
em2| |em2
+-----+ +-----+
| fw1 |-em1----------em1-| fw2 |
+-----+ +-----+
em0| |em0
| |
---+-------Shared LAN-------+---
em1.
Both are connected to the LAN on em0 and to a WAN/internet
connection on em2.
IP addresses are as follows:
fw1 will be the preferred master.
To configure fw1, begin by enabling preemption and group interface failover:
# sysctl net.inet.carp.preempt=1 # echo 'net.inet.carp.preempt=1' >> /etc/sysctl.confConfigure pfsync:
# ifconfig em1 10.10.10.1 netmask 255.255.255.0 # ifconfig pfsync0 syncdev em1 # ifconfig pfsync0 upConfigure CARP on the LAN side:
# ifconfig carp1 create
# ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \
172.16.0.100 netmask 255.255.255.0
Configure CARP on the WAN/internet side:
# ifconfig carp2 create
# ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \
192.0.2.100 netmask 255.255.255.0
Then configure fw2 accordingly:
# sysctl net.inet.carp.preempt=1
# echo 'net.inet.carp.preempt=1' >> /etc/sysctl.conf
# ifconfig em1 10.10.10.2 netmask 255.255.255.0
# ifconfig pfsync0 syncdev em1
# ifconfig pfsync0 up
# ifconfig carp1 create
# ifconfig carp1 vhid 1 carpdev em0 pass lanpasswd \
advskew 128 172.16.0.100 netmask 255.255.255.0
# ifconfig carp2 create
# ifconfig carp2 vhid 2 carpdev em2 pass netpasswd \
advskew 128 192.0.2.100 netmask 255.255.255.0
Examples:
/etc/hostname.carp1/etc/hostname.pfsync0
To failover a particular CARP group, shut down the carp interface on the
master node.
This will cause the master to advertise itself with an "infinite"
advbase and advskew.
The backup host(s) will see this and immediately take over the role of master.
# ifconfig carp1 downAn alternative is to increase the
advskew to a value that's
higher than the advskew on the backup host(s).
This will cause a failover but still allow the master to participate in
the CARP group.
Another method of failover is to tweak the CARP demotion counter. The demotion counter is a measure of how "ready" a host is to become master of a CARP group. For example, while a host is in the middle of booting up it's a bad idea for it to become the CARP master until all interfaces have been configured, all network daemons have been started, etc. Hosts advertising a high demotion value will be less preferred as the master.
A demotion counter is stored in each interface group that the CARP interface
belongs to.
By default, all CARP interfaces are members of the "carp" interface group.
The current value of a demotion counter can be viewed using
ifconfig:
# ifconfig -g carp carp: carp demote count 0In this example, the counter associated with the "carp" interface group is shown. When a CARP host advertises itself on the network, it takes the sum of the demotion counters for each interface group the carp interface belongs to and advertises that value as its demotion value.
Now assume the following example. Two firewalls running CARP with the following CARP interfaces:
First, assign each to a new interface group, in this case named "internal":
# ifconfig carp1 group internal
# ifconfig carp2 group internal
# ifconfig internal
carp1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
carp: MASTER carpdev em0 vhid 1 advbase 1 advskew 100
groups: carp internal
inet 10.0.0.1 netmask 0xffffff00 broadcast 10.0.0.255
carp2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
carp: MASTER carpdev em1 vhid 2 advbase 1 advskew 100
groups: carp internal
inet 10.0.1.1 netmask 0xffffff00 broadcast 10.0.1.255
Now increase the demotion counter for the "internal" group:
# ifconfig -g internal internal: carp demote count 0 # ifconfig -g internal carpdemote 50 # ifconfig -g internal internal: carp demote count 50The firewall will now gracefully failover on the carp1 and carp2 groups to the other firewall in the cluster while still remaining the master on carp3 and carp4. If the other firewall started advertising itself with a demotion value higher than 50, or if the other firewall stopped advertising altogether, then this firewall would again take over mastership on carp1 and carp2.
To fail back to the primary firewall, reverse the changes:
# ifconfig -g internal -carpdemote 50 # ifconfig -g internal internal: carp demote count 0Network daemons such as bgpd(8) and sasyncd(8) make use of the demotion counter to ensure that the firewall does not become master until BGP sessions become established and IPsec SAs are synchronized.
carp0).
So, write the rulesets accordingly.
Don't forget that an interface name in a PF rule can be either the name of
a physical interface or an address associated with that interface.
For example, this rule could be correct:
pass in on fxp0 inet proto tcp from any to carp0 port 22But replacing the
fxp0 with carp0 would not work as
desired.
Don't forget to pass proto carp
and proto pfsync!