{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Decentralized server orchestration\n",
"---\n",
"_Rick van de Loo_
\n",
"https://github.com/vdloo\n",
"\n",
"DevOps @ Byte"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# High available websites on mobile phones\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Moving devices are not that different from flaky containers/VMs\n",
"\n",
"A phone losing wifi is conceptually the same as a freezing VM"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# In the cloud everything breaks and great software exists to deal with that\n",
"\n",
"- service discovery\n",
"- load balancing\n",
"- automated failovers"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# We can't just use that cloud stuff to cluster mobile phones\n",
"\n",
".. or can we?"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# What are the problems?\n",
"\n",
"- mobile phones move fast (WIFI / 3G / offline)\n",
"- not in one zone / routes change\n",
"- often behind NAT (no inbound traffic)\n",
"- Android is not GNU/Linux"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## I need a decentralized virtual network\n",
"\n",
"- Can't do VPN -> single point of failure\n",
"- Can't do SSH jumphosts -> paths to nodes change\n",
"- Solution: overlay mesh networking"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## CJDNS\n",
"\n",
"- DHT mesh routing\n",
"- End-to-end encryption\n",
"- IPv6 address derived from public key\n",
"- TUN virtual network interface"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Finds a path transparently\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## The TUN network is flat, hops are abstracted out\n",
"\n",
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Connect directly to any peer\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Running GNU/Linux on Android\n",
"\n",
"- Root the phone\n",
"- Install the LinuxDeploy Android (APK on github)\n",
"- Create a chroot (Archlinux ARM)\n",
"- Now you've got a shell with a full Linux userland!\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Now we can just treat it like any server\n",
"\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Only have one real phone though\n",
"\n",
"Let's find a way to virtualize more\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Android-x86\n",
"\n",
"\n",
"\n",
"\n",
"AOSP port to x86\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## vdloo/android-x86-64-vagrant\n",
"\n",
"Packer script to bootstrap an Archlinux chroot in Android 6.0 Marshmallow and package a Vagrant box.\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"### Dealing with outages\n",
"\n",
"- Consul by Hashicorp\n",
"- Distributed DNS interface\n",
"- Distributed service discovery\n",
"- Distributed failure detection\n",
"- Distributed key value store"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Raft consensus algorithm\n",
"\n",
"\n",
"\n",
"\n",
"Also notice my perfect gif cropping skills"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Leader election\n",
"\n",
"\n",
"\n",
"Source: https://raft.github.io/"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Quorum requires (n/2)+1 nodes to be online\n",
"\n",
"I'm misusing it by making all nodes 'server' nodes"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Taping it all together with Python\n",
"\n",
"\n",
"\n",
"https://github.com/vdloo/raptiformica\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Framework for decentralized server provisioning\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Uses Consul KV to store data about peers\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## vdloo/consul-kv\n",
"\n",
"Python 3 client for the consul key/value store."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [],
"source": [
"# from consul_kv import Connection\n",
"# conn = Connection(endpoint='http://localhost:8500/v1/kv')\n",
"# conn.put_dict({'a_dict_key': 'some_value'})"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Now we have all we need to host distributed applications on mobile phones"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Distributed LEMP stack on Android\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Runs PHP just fine\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Webshop in your pocket, no problem\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Building a webapp specifically for this platform\n",
"\n",
"- Completely decentralized\n",
"- Reverse proxy through meshnet\n",
"- Round robin DNS with consul\n",
"- Use distributed KV as a database\n",
"- Autoscaling -> more nodes online == more workers\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Forked fc00.org -> public CJDNS network map (Flask)\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Modified to map my private network\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Rewrote backend to use consul-kv\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Bind a gunicorn on the IPv6 interface\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## All nodes use the Consul DNS server\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Register the service so it is loadbalanced\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Picking a reverse proxy\n",
"\n",
"- Using Consul with DNSmasq to loadbalance\n",
"- Nginx can't resolve DNS per request\n",
"- HAProxy can't resolve DNS per request\n",
"- H2O -> super lightweight and CAN resolve DNS per request!\n",
"\n",
"See: https://github.com/h2o/h2o\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Reverse proxy to round robin load balanced consul service\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Path of a request\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Result\n",
"\n",
"- Self registering and redundant\n",
"- Distributed and decentralized\n",
"- As long as there is a path any node can join\n",
"- Goes through firewalls / cross zones\n",
"- Nodes can switch from WIFI to mobile data transparently"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## 25 node network\n",
"\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"collapsed": true,
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Future\n",
"\n",
"- Move away from Consul to something with commutative properties\n",
"- Cluster more unconventional hardware\n",
"- Define triggers for cluster changes (less than x nodes -> do something)\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# That's it. Questions?\n",
"---\n",
"## You can find these slides and the Jupyter notebook here: https://github.com/vdloo/slides\n"
]
}
],
"metadata": {
"celltoolbar": "Slideshow",
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 1
}