#- # Copyright (c) 2015-2016 Dridi Boukelmoune # Copyright 2017 UPLEX - Nils Goroll Systemoptimierung # # Authors: Dridi Boukelmoune # Nils Goroll # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. $Module dynamic 3 Varnish dynamic backends module .. role:: ref(emphasis) DESCRIPTION =========== This module provides a varnish director for dynamic creation of backends based on calls to the system's network address resolution services, which, in turn, typically use information from the ``/etc/hosts`` file and the Domain Name Service (DNS), but can be configured to use other sources like LDAP (see :ref:`nsswitch.conf(5)`). While standard varnish backends defined in VCL may also be defined in terms of host names, changes of the name service information will only be picked up with a VCL reload. In contrast, for dynamic backends provided by this module, * name resolution information will be refreshed by background threads after a configurable time to live (ttl) * resolution to multiple network addresses is supported .. _ref_vmod_dynamic_share: BACKEND SHARING =============== By default (``share = "DIRECTOR"`` parameter), backends are shared per director: Only one backend per director instance and ip address is created. The ``share = "HOST"`` parameter allows to limit sharing to backends created for the same hostname (``host`` argument to the ``.backend()`` method). .. _ref_vmod_dynamic_probe: PROBING ======= A probe to be used with dynamically created backends can be specified. Note that with ``share = "DIRECTOR"``, a ``Host:`` header is only sent with probes if the ``host_header`` argument is provided during director initialization (or because a custom probe ``request`` contains one). With ``share = "HOST"``, the ``Host:`` header for probes defaults to the backend's hostname (``host`` argument to the ``.backend()`` method). Consider setting the ``initial`` attribute of probes at least as high as the ``threshold`` attribute. Otherwise transactions that trigger the first lookup of a domain will see a sick backend and fail. Irrespective of the ``initial`` attribute, transactions may still fail for backends which are actually sick. This can be mitigated using the ``retry`` transition in VCL. TTLs from DNS ============= A frequent use case of the dynamic director is to use the DNS TTL as the TTL of dynamic backends. Because there is no standard library function returning this information, the dynamic director currently supports this setup in coorperation other services supporting caching of dynamic DNS TTLs: * install and configure a system name service caching service like :ref:`nscd(8)` * use the *ttl* director initialization parameter as a minimum ttl. STATISTICS ========== Dynamic backends are created and deleted on demand and can be monitored just like VCL-defined backends. Their statistics will appear in VSM-tools like ``varnishstat`` as: * for ``share = "DIRECTOR"``:: VBE..().* * for ``share = "HOST"``:: VBE...().* LOGGING ======= This module may log ``VCL_Log``, ``Error``, and ``Debug`` records following a common pattern:: vmod-dynamic: %s %s %s %s [ %s ] | | | | | | | | | +- Additional information | | | +------ Event | | +--------- Host name | +------------ Director name +--------------- VCL name Lookup timestamps are also logged to help troubleshooting, using regular ``Timestamp`` records with the following pattern for event labels:: vmod-dynamic .() When a lookup thread is terminated, either because the VCL is cooling down or the ``domain_usage_timeout`` triggered, ``Timestamp`` records are logged with the event:: vmod-dynamic .() Done Not all logs belong to HTTP transactions, especially since DNS lookups happen in the background. In order to capture all logs from this module the simplest way with varnishlog is the following:: varnishlog -g raw -q '* ~ vmod-dynamic' It displays any individual record that contains the string ``vmod-dynamic`` whether it belongs to a transaction or not. .. raw:: pdf PageBreak When a lookup fails, the backends are left untouched and the error will be logged with the following event:: getaddrinfo () $Event vmod_event $Object director( STRING port = "http", STRING host_header = 0, ENUM { DIRECTOR, HOST } share = "DIRECTOR", PROBE probe = 0, ACL whitelist = 0, DURATION ttl = 3600, DURATION connect_timeout = 0, DURATION first_byte_timeout = 0, DURATION between_bytes_timeout = 0, DURATION domain_usage_timeout = 7200, DURATION first_lookup_timeout = 10, INT max_connections = 0, INT proxy_header = 0) Description Create a DNS director. The director creates backends with DNS lookups and chooses them in a round robin fashion. It accepts the following Parameters: - *port* (defaults to ``"http"``) The port to conncet to - *share* (defaults to ``"DIRECTOR"``) Backend sharing scope, see :ref:`ref_vmod_dynamic_share` - *host_header* (defaults to none) `host_header` attribute for dynamically created backends. See also :ref:`ref_vmod_dynamic_probe` - *probe* (defaults to none) Probe to use. See also :ref:`ref_vmod_dynamic_probe` - *whitelist* - an acl (defaults to none) Only name resolution results matching the acl will be used. - *ttl* - delay between lookups (defaults to one hour) Minimum backend lifetime before address resultion is re-checked - *domain_usage_timeout* Delay until an unused domain and its backends are removed (defaults to two hours) - *first_lookup_timeout* Delay until the director fails lookup when a domain is requested for the first time and there is no response from the name service (defaults to ten seconds) Parameters to set attributes of backends See varnish documentation for details - *connect_timeout* (defaults to global *connect_timeout*) - *first_byte_timeout* (defaults to global *first_byte_timeout*) - *between_bytes_timeout* (defaults to global *between_bytes_timeout*) - *max_connections* (defaults to zero, unlimited) - *proxy_header* - version of the PROXY protocol to use, or zero to disable it (defaults to zero, valid versions are one or two) .. raw:: pdf PageBreak Example :: probe www_probe { .window = 8; .initial = 7; .threshold = 6; .interval = 5s; } acl www_acl { "192.168"/24; } sub vcl_init { new www_dir = dynamic.director( port = "80", probe = www_probe, whitelist = www_acl, ttl = 5m); } sub vcl_recv { set req.backend_hint = www_dir.backend("production.acme.com"); } .. raw:: pdf PageBreak $Method BACKEND .backend(STRING host = "") Description Pick a backend from the director for a given host name. If the host is not specified, it is picked from either ``bereq`` or ``req``. $Method VOID .debug(BOOL) Description Enable or disable debugging for a dynamic director, logging background operations related to backends management. PITFALLS ======== There is no support for lookups limited to IPv4 or IPv6 only. However it can be achieved by the means of a white list:: acl ipv4_only { "0.0.0.0"/0; } acl ipv6_only { "::0"/0; } With that you can restrict backends to the desired IP network, and monitor error logs with the ``acl-mismatch`` event. Knowing which addresses were rejected, you can fix your domains registration (DNS records, hosts file etc). SEE ALSO ======== * :ref:`vcl(7)` * :ref:`vsl(7)` * :ref:`vsl-query(7)` * :ref:`varnish-cli(7)` * :ref:`varnish-counters(7)` * :ref:`varnishstat(1)` * :ref:`getaddrinfo(3)` * :ref:`nscd(8)` * :ref:`nsswitch.conf(5)` COPYRIGHT ========= This document is licensed under the same licence as vmod-dynamic itself. See LICENCE for details. * Copyright (c) 2015-2016 Dridi Boukelmoune * Copyright 2017 UPLEX - Nils Goroll Systemoptimierung