and var app = require('express')()\nvar http = require('http')\nvar server = http.createServer(app)
app.get('/', function (req, res) {\n res.send('hello world')\n})
server.listen(3000)\ntextCopy to clipboard\n\nIn order to measure this, we will use autocannon, an HTTP/1.1 benchmarking tool inspired by wrk and built in Node.js:\n\n$ autocannon -c 100 -d 5 https://localhost:3000\nRunning 5s test @ https://localhost:3000\n100 connections with 1 pipelining factor
Stat Avg Stdev Max\nLatency (ms) 13.57 4.65 117\nReq/Sec 7087.34 834.74 8039\nBytes/Sec 1.49 MB 175.1 kB 1.7 MB
430k requests in 5s, 8.93 MB read\ntextCopy to clipboard\n\nLet’s see what happens when we add logging (in all the following example, we will be redirecting the output from stdout to /dev/null):\n\n'use strict'
var app = require('express')()\nvar http = require('http')\nvar server = http.createServer(app)
app.use(require('express-bunyan-logger')())
app.get('/', function (req, res) {\n res.send('hello world')\n})
server.listen(3000)\ntextCopy to clipboard\n\nWe would expect this to have similar throughput to the first one. However we got some surprising results:\n\n$ autocannon -c 100 -d 5 https://localhost:3000\nRunning 5s test @ https://localhost:3000\n100 connections with 1 pipelining factor
Stat Avg Stdev Max\nLatency (ms) 58.27 16.02 243\nReq/Sec 1682.17 203.94 1878\nBytes/Sec 354.99 kB 42.74 kB 409.6 kB
10k requests in 5s, 2.12 MB read\ntextCopy to clipboard\n\nRunning a server using bunyan for logging will reduce throughput by almost 80%. Winston yields slightly better results:\n\n'use strict'
var app = require('express')()\nvar http = require('http')\nvar winston = require('winston')\nvar winstonExpress = require('express-winston')\nvar server = http.createServer(app)
app.use(winstonExpress.logger({\n transports: [\n new winston.transports.Console({\n json: true\n })\n ]\n}))
app.get('/', function (req, res) {\n res.send('hello world')\n})
server.listen(3000)
$ autocannon -c 100 -d 5 https://localhost:3000\nRunning 5s test @ https://localhost:3000\n100 connections with 1 pipelining factor
Stat Avg Stdev Max\nLatency (ms) 28.81 8.5 149\nReq/Sec 3400 425.02 3703\nBytes/Sec 711.34 kB 88.54 kB 786.43 kB
20k requests in 5s, 4.28 MB read\ntextCopy to clipboard\n\nRunning a server using Winston (version 2) for logging will slow you down by at least 50%.\n\nAs part of our consultancy, Dave and myself were asked for a fast alternative, and there wasn’t one. Therefore we set ourselves to the task of writing the fastest possible Node.js logger we could manage. We wanted to build something that could easily replace Winston, Bunyan (or even Bole).",
"Introducing Pino\n\nOur logger is Pino, named after pine in Italian, because there is a pine in front of my house. Bunyan was a giant lumberjack, and Pino is (99%) API-compliant with Bunyan and Bole.\n\nPino is the fastest logger for Node.js (full benchmarks found here and here). While you can use Pino directly, you will likely use it with a Web framework, such as express or hapi.\n\nLet’s consider the following example using express-pino-logger:\n\n'use strict'
var app = require('express')()\nvar http = require('http')\nvar server = http.createServer(app)
app.use(require('express-pino-logger')({\n extreme: true\n}))
app.get('/', function (req, res) {\n res.send('hello world')\n})
server.listen(3000)\ntextCopy to clipboard\n\nWe can run the same benchmark as before:\n\n$ autocannon -c 100 -d 5 https://localhost:3000\nRunning 5s test @ https://localhost:3000\n100 connections with 1 pipelining factor
Stat Avg Stdev Max\nLatency (ms) 20.16 2.73 91\nReq/Sec 4819.34 89.14 4919\nBytes/Sec 1.02 MB 25.03 kB 1.05 MB
290k requests in 5s, 6.07 MB read\ntextCopy to clipboard\n\nUsing express-pino-logger increase the throughput of express-winston by almost 40%.",
"Anatomy of Pino\n\nPino is a fast logger built on several principles:\n\nIt has the fewest possible features; there are no transports, it just logs to a binary stream (a file or stdout), and no file rotation.\n\nIt only logs in JSON.\n\nIt needs to be as fast as possible.\n\nIt should support child loggers, to allow transactions to have their own metadata.\n\nDrop-in compatible API with bunyan, bole, winston wherever possible.\n\nInstantiating Pino is easy:\n\n'use strict'
var pino = require('pino')()
pino.info('hello world')\npino.error('this is at error level')\npino.info('the answer is %d', 42)\npino.info({ obj: 42 }, 'hello world')\npino.info({ obj: 42, b: 2 }, 'hello world')\npino.info({ obj: { aa: 'bbb' } }, 'another')\nsetImmediate(function () {\n pino.info('after setImmediate')\n})\npino.error(new Error('an error'))
var child = pino.child({ a: 'property' })\nchild.info('hello child!')
var childsChild = child.child({ another: 'property' })\nchildsChild.info('hello baby..')\ntextCopy to clipboard",
"The Pino family\n\nPino has a set of companion tools, each of which have been meticulously implemented to have a minimal impact on performance:\n\nexpress-pino-logger: express integration\n\nkoa-pino-logger: koa integration\n\nhapi-pino: Hapi integration\n\nrestify-pino-logger: restify integration\n\npino-http: barebone logger for node http module, basis of express, koa and restify integrations\n\npino-socket: forwards pino logs over a TCP or UDP socket",
"Conclusion\n\nPino is the fastest logger in town (see the benchmarks), and it uses a full set of performance optimizations to achieve this goal, from avoiding JSON.stringify, to rolling our own printf style formatting module and flattening strings. These techniques reduce the overhead of logging by 40%. In extreme mode, we also delay flushing to disk, to reduce it even more, by 60%. We have adopted Pino in some projects, and it has increased the throughput (req/s) of those applications by a factor of 20-100%. Stay tuned for more blog posts on the techniques that Pino uses to be so fast, as each of those has its own story."
],
"categories": {
"primary": "perf",
"others": [
"backend",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-08-05-announcing-victory-0-10-2-and-victorynative": {
"href": "https://nearform.com/digital-community/08/05/announcing-victory-0-10-2-and-victorynative",
"postType": "blog",
"slug": "digital-community-08-05-announcing-victory-0-10-2-and-victorynative",
"date": "2016-08-05",
"title": "Announcing Victory 0.10.2 and VictoryNative",
"authors": [
"EMMA BRILLHART"
],
"content": [
"Check out the Victory docs here.\n\nIt’s been a while since the release of Victory 0.9.0 back in June, so we’re excited to add several new features and bug fixes in order to continue making the creation of D3 charts in React as painless as possible. This post will explore some of the highlights of the new release, including cool new features like VictoryCandlestick, VictoryErrorBar, and the highly anticipated VictoryNative! Note that this version of Victory has breaking changes due to changes in default styles and some default props.\n\nFeatures\n\nWe’ve added two new components to VictoryChart: VictoryErrorBar and VictoryCandlestick. They’re both as composable and flexible as the components that came before them, giving you a wider range of chart types that you can put together using Victory.\n\nVictoryCandlestick\n\nVictoryCandlestick takes data in the format [{x: value, open: value, high: value, low: value}]. Data accessor props for all values are included. Since candlestick charts are most commonly used for commodity and stock price tracking, open and close are the opening and closing price for a given period of time, while high and low are the absolute high point and low point that the cost reached over that same period. Check out this example to see how you can use Candlestick in conjunction with other Victory components!\n\nVictoryErrorBar\n\nVictoryErrorBar expects data in the form [{x: value, y: value, errorX: err, errorY: err}], where err is a number or two-value array for asymmetric errors. The component includes data accessor props for x, y, errorX, and errorY. VictoryErrorBar can be used on its own or with VictoryChart, and looks great combined with other chart types like VictoryScatter.\n\nVictoryTheme\n\nAnother exciting thing that we’ve added is theme support for Victory components! All components can now take a theme prop, which is an object that should be structured approximately like so:\n\n```js export default { area: { data: {}, labels: {}, parent: {} }, axis: { axis: {}, axisLabel: {}, grid: {}, ticks: {}, tickLabels: {} }, bar: { data: {}, labels: {}, parent: {} }, candlestick: { data: {}, labels: {}, parent: {}, props: {} }, errorbar: { data: {}, labels: {}, parent: {} }, line: { data: {}, labels: {}, parent: {} }, pie: { props: {}, style: { data: {}, labels: {}, parent: {} } }, scatter: { data: {}, labels: {} parent: {} }, props: {} }; ```\n\nTo see more specifics, check out Material Theme, which is included in VictoryCore. Custom themes are supported as well, as long as they have the same general structure!\n\ndomainPadding\n\nFinally, the domainPadding prop has also gotten some upgrades – it’s now available on all components that are compatible with VictoryChart, it supports negative and asymmetric values, and it’s applied automatically to grouped bar charts to prevent charts from overflowing the axis.\n\nFor a complete overview of what’s new, check out the Victory Changelog.\n\nPerformance Improvements\n\nWe made it easier to use Victory with large datasets by making some low-hanging performance improvements, including replacing Object.assign with the lodash assign method, and replacing map and reduce methods with length-cached for loops in methods that are responsible for rendering elements.\n\nBug Fixes\n\nKey bug fixes include fixing various issues that arose when using log scales, better handling of cross axis positioning when using dates as axis values, and improving how chart components handle being passed empty data arrays or single data points.\n\nAnd... Announcing VictoryNative!\n\nCreate the same Victory SVG charts that you love in React Native. See the repo here.\n\nThe API for VictoryNative is identical to Victory, so you can do all the same stuff that you love in the browser on iOS or Android. We were excited to see how little code we needed to write in order to adapt the components to be usable in React Native.\n\nIf you have any questions or problems you’d like us to address, create an issue in the VictoryNative repo or the Victory repo, or contact us on our Gitter channel.\n\nThanks for using Victory!\n\nCheck out the Victory docs here."
],
"categories": {
"primary": "frontend",
"others": [
"mobile"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-pre-compiling-node-js-modules": {
"href": "https://nearform.com/insights/pre-compiling-node-js-modules",
"postType": "blog",
"slug": "insights-pre-compiling-node-js-modules",
"date": "2016-08-09",
"title": "Pre-compiling Node.js Modules",
"authors": [],
"content": [
"Automated Publishing Toolchains for Node.js Modules\n\nI recently worked on a pretty cool feature for autocannon that makes it easier for users to download and use it. The feature is the pre-compilation of the native Node.js modules that autocannon depends on. This means that autocannon users don’t have to compile dependencies themselves. Therefore, they don’t need to have build tools installed, and this lowers the entry barrier for using Autocannon.\n\nAutocannon itself only has a single native module that needs to be compiled, and that is native-hdr-histogram . To do this, I needed to switch out node-gyp for node-pre-gyp , and “publish” compiled versions of the Node.js modules on AWS S3.\n\nOur goal was to give users running node 4, 5 & 6 the ability to download these native Node.js modules, without the need to compile them. To do this, we needed to compile the binaries for each Node.js version for Windows 64 and 32 bit, Linux 64 and 32 bit and OSx 64 bit only. That means on a build and publish, we needed to compile 15 binaries and publish them to some form of publicly available cloud storage.\n\nCI automation to the rescue.\n\nPrerequisites\n\nTo move forward with this work, you will need to create an Amazon S3 bucket, and have an IAM user policy with a user configured with the ability to publish to that bucket. You will also need this user’s access key and secret access key. Read the Readme of node-pre-gyp over here and figure it out. For this, we created a bucket called autocannon-assets, in eu-west-1. These minor details are important to note. We use the url for this bucket ( https://autocannon-assets.s3-eu-west-1.amazonaws.com ) to build sane urls to download the binaries.\n\nThe Setup\n\nnode-gyp is a build tool which is used extensively in Node.js projects. node-pre-gyp is a wrapper around this. It adds functionality to build and publish Node.js modules to AWS S3 buckets and download those published Node.js modules when running npm install. It can fall back to compiling them the way node-gyp does. However, this is only when there is no published version available on S3 which is compatible with the user’s installed node version. In this case, users will need compatible build tools to build it.\n\nWith native-hdr-histogram, we aimed to automate as much of the publishing and building process as possible. To do this, we made heavy usage of CI tooling. For Linux and OSX support, we used Travis, and to support Windows we needed to add Appveyor to the mix. To support these CI tools, we needed to have scripts created for the auto publishing of binaries. Figuring out all the bits and bobs that the scripts needed took a while, and I’m pretty sure I killed Travis’s osx build infrastructure while working on this! To run CI tools with the ability to publish to S3 buckets, we needed to embed our keys within the repository. This is a pretty big no no if you don’t want people getting access to your infrastructure. To safely embed the keys in the corresponding .travis.yml and appveyor.yml , you need to first encrypt them. There is a different path for encrypting these keys within each platform. To encrypt these keys on Travis you must use a Ruby gem. To install this gem, simply do the following in the command line: gem install travis After it is installed, you then need to encrypt the keys with the following commands. (This is best done in the directory of the project; otherwise you must add -r owner/project to the command)\n\ntravis encrypt node_pre_gyp_accessKeyId=${node_pre_gyp_accessKeyId} travis encrypt node_pre_gyp_secretAccessKey=${node_pre_gyp_secretAccessKey}\ntextCopy to clipboard\n\nThis will then output something like secure: “........ encrypted .........” which can be put into your .travis.yml like so:\n\nenv:\n global:\n - secure: xb95WkJu5sab...............\n - secure: wugo4xhy7hO5SErSxy...............\ntextCopy to clipboard\n\nOn Appveyor, the project owner needs to be the one to encrypt the keys. To do this, you must be signed in as the project owner, and navigate to the encrypt data page . On this page, you simple copy and paste your access key and secret access key into the box and get the results. To then use these keys, embed them into your appveyor.yml like so:\n\nenvironment:\n node_pre_gyp_accessKeyId:\n secure: BB4XzOLXvG+XMmXyCVwqaFF/hKJnYvjIACQMD423QvY=\n node_pre_gyp_secretAccessKey:\n secure: BFW25HQ0Jdq8IpysC59Y2mh1J9IsXwWB+FTElHVSYVl5xDwwZPHrXwctZQLzKrw9\ntextCopy to clipboard\n\nYou can read the readme of node-pre-gyp for travis and appveyor if you want to know more.\n\nWe didn’t always want to publish binaries when they were run through CI though, so we needed to put a safeguard in place. That was to only publish module binaries when the the commit message contained [publish binary]. We prepend version bumps with this, to trigger building and publishing in the CI server. eg, [publish binary] 0.3.0 would cause a publish. There is already an existing module which essentially has scripts which does everything that we needed, which is the node-sqlite3 module by mapbox . This module had many scripts which I was able to repurpose for our uses under the BSD license.\n\nSuccessful build and publish in Travis and Appveyor with a commit tagged with “[publish binary]“.\n\nBasically, if you want do something similar, take the /scripts/ folder from native-hdr-histogram , along with the Makefile, and copy and modify the appveyor.yml and .travis.yml files as you need. It looks scarier than it is, they’re already configured to handle node 4-6 for you. You might need to change the test target in the Makefile depending on your setup.\n\nYou also need to modify your package.json, and add a binary attribute . The module_path, remote_path, and package_name can be copied as is, but the host must be changed to match your S3 bucket URL. The module_name is the name of the compiled binary, I named it “histogram” for simplicity.\n\nAdditionally, using node-gyp requires you to create a binding.gyp file. When using node-pre-gyp, you must modify this file to include a “target” with the exact same content as this one , and you should modify your normal build target to use the module_name value used in your package.json, by setting its “target_name” to “<(module_name)”.\n\nYour index.js, or rather, the main file that you export from your package.json should require the native file for exporting, using node-pre-gyp. It should end up similar to the one from here , assuming your binary section was similar to mine.\n\nNow, when you want to publish a binary, all you need to do is add [publish binary] to your commit and push it to Github. It will take a while, because appveyor is kinda ever so slightly slow, but in the meantime if a user does download your native module without the published version available, they will simply build it themselves locally.\n\nConclusion\n\nSo, I’m sorry this was super long, technical, and didn’t have many examples, but it is one of the cooler technical things I have worked on recently that I wanted to share with people. Using this, we can create really nice automated publishing toolchains for any native Node.js modules and isn’t that what everyone wants?\n\nHow to keep developers happy; passing builds in their CI.\n\nYou may find these articles about Node.js modules helpful as well.\n\nNode.js Modules - NearForm bootcamp series\nThe future of Native Modules in Node.js"
],
"categories": {
"primary": "backend",
"others": [
"cloud",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-08-15-introducing-webpack-dashboard": {
"href": "https://nearform.com/digital-community/08/15/introducing-webpack-dashboard",
"postType": "blog",
"slug": "digital-community-08-15-introducing-webpack-dashboard",
"date": "2016-08-15",
"title": "Introducing Webpack Dashboard",
"authors": [
"KEN WHEELER"
],
"content": [
"> How about a nice game of chess?\n\nAs a child, I always wanted to be a hacker. I remember the first time my father showed me how to dir /p. I thought I was hacking the planet. Then I grew up, and I realized the reality of it was a bit different. I still use the CLI all the time, but it’s far less glamorous in daily use than it had been portrayed in film & media during my childhood.\n\nBut that doesn’t mean we still can’t have fun. I needed some command line nostalgia in my life, so I made it happen and I’m here to introduce Webpack Dashboard.\n\nIf you use webpack with regularity, specifically the webpack-dev-server or webpack-dev-middleware, you are probably used to seeing something like this:\n\nNow don’t get me wrong, its a great collection of information, but I’ve always found it to be hard to derive any useful information from it without waiting for it to finish and then scrolling upwards until I could find the information I wanted.\n\nA while back, I came across a library called blessed, that lets you easily create command line interfaces for things like dashboards. I had tinkered with it, but didn’t really have a great dashboard idea. Until now. I realized this sort of thing would actually really help organize and intuitively display dev server output, so I created webpack-dashboard. Here is what your dev server output can look like now:\n\nCheck it out in action:\n\nFor the initial release, we have the following features:\n\nLog: Logs errors from the build process\nStatus: The status of the build process\nOperation: The specific operation taking place in the build process\nProgress: Live build progress\nModules: A list of your modules, their sizes and percentage of bundle size\nAssets: A list of assets output from the build\n\nSo if you want to give it a spin, head on over to:\n\nhttps://github.com/FormidableLabs/webpack-dashboard"
],
"categories": {
"primary": "frontend",
"others": [
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-08-22-make-dope-beats-with-reactjs": {
"href": "https://nearform.com/digital-community/08/22/make-dope-beats-with-reactjs",
"postType": "blog",
"slug": "digital-community-08-22-make-dope-beats-with-reactjs",
"date": "2016-08-22",
"title": "Make Dope Beats with ReactJS",
"authors": [
"KEN WHEELER"
],
"content": [
"“With a kick, snare, kicks and high hat, skilled in the trade of that old boom bap”\n\nBefore I was a software engineer, I was a music producer. Specifically, I made beats for rap songs. Then file sharing put a dent in the music industry, and it became much harder to do as a career. But hey, I had programming to fall back on!\n\nI still made music as a hobby over the years, but less and less seriously as time went on. Very recently, I got news that Akai was discontinuing its iconic offering of standalone hardware MPC drum machines. This made me deeply sad. I mowed untold numbers of lawns for my first MPC and it felt to me like the death of an era.\n\nSo I did what any geek that writes React code all day long would do. I wrote a library so that I can make beats with ReactJS. I call it react-music.\n\nreact-music let you make beats using JSX code and hooks into the Web Audio API to turn your code into beautiful music. There is no UI, with the exception of a play/stop button and a spectrum analyzer that renders on your page:\n\nThe code is regular React components that implements the sort of features you might find in a sequencer/sampler/DAW via the props API:\n\nOpen up your running app in a browser, and now you have super hot beats:\n\nThe library is still very young at this point, and has a lot of work on the roadmap, but if you’d like to make some hot fire with the power of React, check it out here:\n\nhttps://github.com/FormidableLabs/react-music"
],
"categories": {
"primary": "frontend",
"others": [
"ai"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-08-29-terminal-charts-with-victory-cli": {
"href": "https://nearform.com/digital-community/08/29/terminal-charts-with-victory-cli",
"postType": "blog",
"slug": "digital-community-08-29-terminal-charts-with-victory-cli",
"date": "2016-08-29",
"title": "Terminal Charts with Victory CLI",
"authors": [
"KEN WHEELER"
],
"content": [
"The ability to collect, analyze, triangulate and visualize vast amounts of data in real time is something the human race has never had before.\n\nRick Smolan\n\nAfter pushing out webpack-dashboard and seeing how much people enjoy terminal-based tooling, I thought perhaps I should see what else I can explore in this space.\n\nFormidable’s ReactJS-based data visualization library Victory seemed like an awesome candidate for a command-line render target. I saw a project, hyperchart, by Matthew Conlen, that allowed users to generate interactive Victory charts into hyperterm-window.\n\nThis looked awesome, but I was looking for a way to expand the feature set and operate in a broader range of TERMs. I had also seen some ASCII/ANSI style visualizations in blessed-contrib while building webpack-dashboard, and they seemed super cool. However, porting would be too much of a challenge given the differences in APIs. Of course, ASCII/ANSI style charts didn’t seem practical for the mainstream use case.\n\nThen I realized something: Victory outputs valid SVG markup. This, coupled with ReactDOM.renderToString, actually makes command-line chart generation really easy. I got to work on it, and today I’m happy to announce the initial release of Victory CLI!\n\nVictory CLI is an easy-to-use command-line interface for generating and displaying Victory-based data visualizations. It can:\n\nRender a variety of chart types\nUse built-in themes\nRender to PNG\nRender to SVG\nDisplay PNG renders in iTerm3\nTake a custom JSON data input\nTake a custom component script input to render from\n\nCheck it out in action:\n\n\nIf you think this is cool and want to give it a try, just npm install -g victory-cli and head over to https://github.com/FormidableLabs/victory-cli to get started!"
],
"categories": {
"primary": "data",
"others": [
"frontend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-09-08-builder-3-0-battle-tested-multiple-project-management": {
"href": "https://nearform.com/digital-community/09/08/builder-3-0-battle-tested-multiple-project-management",
"postType": "blog",
"slug": "digital-community-09-08-builder-3-0-battle-tested-multiple-project-management",
"date": "2016-09-08",
"title": "Builder 3.0: Battle Tested Multiple Project Management",
"authors": [
"BRIAN BECK"
],
"content": [
"Today we’re excited to announced the release of Builder 3.0, which contains improvements based on our experience using Builder in more than 100 projects over the past 10 months.\n\nBuilder solves the problem of multiple projects needing the same dependencies, configuration, and npm tasks. For example, every time we create a new repository for a React component, we can use a Builder “archetype” that automatically provides our standard Babel, eslint, and webpack settings. We love using npm scripts for our tasks, so the archetype also provides a base set of tasks for things like building, testing, running a development server, etc.\n\nIt does more than that, but you should check out the Builder homepage to read more. Let’s talk about this release!\n\nWhat’s New\n\nThe focus of version 3.0 was making Builder more production-ready. Here are a few changes we made along those lines.\n\nspawn vs. exec\n\nNode’s child_process module offers two ways to launch a child process: spawn and exec. spawn accepts an executable to run and an array of arguments to pass it, while exec takes a single command string, exactly as you’d type it on the command line.\n\nIf you think about what tasks look like in an npm config, they’re all complete command strings:\n\n```json { \"scripts\": { \"build\": \"babel --out-dir lib src\", \"clean\": \"rimraf lib\", \"prepublish\": \"npm run clean && npm run build\", \"test\": \"karma start\" } } ```\n\nAt first glance, these strings can’t be passed as commands to spawn without parsing them, because spawn needs the arguments separated into an array. We didn’t want to get into the business of parsing shell commands, so we reached for exec.\n\nWe got by with this initial choice for a while, but eventually noticed issues. exec buffers output, meaning it captures all of the command’s output into a string. This means you need to worry about buffer size and tell exec what you want the maximum to be. To make this capturing work, it also means that your commands are writing their output to a pipe rather than the true stdout and stderr file descriptors. This shouldn’t really make a difference, but as we’ll see later, it does.\n\nTo get around the output buffering issue while still avoiding any command parsing, we copied the way npm (and exec itself) runs commands:\n\n```javascript child_process.spawn([\"sh\", [\"-c\", \"command string here\"]]) // on Windows: child_process.spawn([\"cmd\", [\"/d /s /c\", \"command string here\"]]) ```\n\nThis is exactly what exec does: it just passes the full command string along to the shell. But by taking matters into our own hands and using spawn, we also get better control over output buffering. Specifically, we can use the stdio: \"inherit\" option to allow child processes to write directly to stdout and stderr, without us doing any buffering in between. Much cleaner!\n\nPipes and process.exit\n\nWhen we were still using exec to launch processes, we noticed that some tasks would consistently have their output cut off. (This especially stood out for certain programs that used ANSI formatting codes, where getting cut off meant that the formatting was never reset. The remaining output from completely unrelated tasks was getting colored and underlined!) Was it due to the aforementioned maximum buffer size? This particular command wasn’t writing anywhere near that limit, so something else must be going on...\n\nWe found that some Node programs use the following pattern (simplified here):\n\n```javascript console.log(\"Lint error: missing semicolon\"); ... console.log(\"Lint failed!\"); process.exit(1); ```\n\nThis looks fine, and will appear to work under most circumstances. But it turns out there’s a known gotcha here. If you log a bunch of data, then call process.exit, and your process is piping its output to a parent Node process instead of writing to the standard output stream, then the output will be cut off! Anything after the first 8192 bytes (in our testing) won’t make it to the parent’s end of the pipe in time.\n\nMany programs never consider how they were launched or where their output is actually going, so this bug is easy to miss when you’re writing a Node CLI program. If you’re writing such a script and use process.exit, you can fix this issue like so:\n\n```javascript function safeExit(code) { process.on(\"exit\", function() { process.exit(code); }); } console.log(\"Lint failed!\"); safeExit(1); ```\n\nThis pattern will let the event loop and Node process end naturally, giving it time to flush its output instead of terminating immediately. Then when it’s ready to exit, it will still use the exit code you provided. (This was an issue with some programs that Builder was running, but we also updated Builder itself to use this pattern, in case another process wants to spawn Builder.)\n\nSwitching to spawn with stdio: \"inherit\" also fixed the programs we were running that had this issue, since they were no longer writing to a pipe.\n\nBetter Windows support\n\nFor 3.0, we added AppVeyor to our continuous integration (CI) setup so that our test suite runs on Windows. This was tricky; since Builder is a task runner, our tests need to set environment variables, launch real processes, and capture output – all things that can (and do) differ across different platforms.\n\nRunning on Windows revealed a few bugs that were previously undiscovered. For example, Windows has case-insensitive environment variables. If you set the PATH environment variable in a Node process, it is internally transformed to set Path on Windows. Builder now accounts for this behavior.\n\nSome small niceties\n\nWe’ve made the help and version commands nicer to use by changing their default log level to be less noisy. Now you’ll see only the usage info, task list, and version number by default – unadultered by the internal task administrivia that we used to show.\n\nTry it out\n\nCheck out the latest version of Builder like so:\n\n```bash npm install builder ```\n\nAnd follow along or contribute here: github.com/FormidableLabs/builder\n\nThanks for reading!"
],
"categories": {
"primary": "frontend",
"others": [
"backend",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-09-13-introducing-nested-routing-in-redux-little-router": {
"href": "https://nearform.com/digital-community/09/13/introducing-nested-routing-in-redux-little-router",
"postType": "blog",
"slug": "digital-community-09-13-introducing-nested-routing-in-redux-little-router",
"date": "2016-09-13",
"title": "Introducing Nested Routing in Redux Little Router",
"authors": [
"TYLER THOMPSON"
],
"content": [
"Ever since we announced Redux Little Router, we’ve worked hard to simplify the API, reduce boilerplate, and ease adoption for beginners. To advance these goals, we’ve introduced two new features: nested routes and nested fragments.\n\nCheck out Redux Little Router on GitHub!\n\nNested Routes\n\nThe first step in installing Redux Little Router into your project is to define a “route configuration.” This associates arbitrary metadata with your routes, allowing you to retrieve useful route-specific values when the current URL matches a route (think page titles, data requirements, or conditional visibility of child components).\n\nPrior to v9.0.0, the route configuration looked like this:\n\n```js const routes = { '/': { title: 'Home' }, '/messages': { title: 'Message' }, '/messages/:user': { title: 'Message History' } }; ```\n\nThere’s problems with this approach. There’s significant repetition of child routes, which irritates more as your routes grow. Worse, flat routes discard all knowledge of hierarchy. Knowing details about parent routes is the only way to enable features that depend on their position in the route hierarchy (consider an “up” button that allows users to travel up the route tree).\n\nNested routes solve both problems in a backwards-compatible way. Route definitions now look like this:\n\n```js const routes = { '/': { title: 'Home' '/messages': { title: 'Message', '/:user': { title: 'Message History' } } }, '/flat-routes-still-work': { title: 'Flat routes still work!' } }; ```\n\nThe result key of your router reducer now recursively attaches parent route results!\n\n```js // For a URL matching /messages/:user { pathname: '/messages/a-user-has-no-name', params: { user: 'a-user-has-no-name' }, query: { some: 'thing' }, result: { title: 'Message History' parent: { title: 'Message' parent: { title: 'Home' } } } } ```\n\nNotice that nested and flat routes can intermingle, with the caveat that flat routes won’t receive any parent route results.route\n\nNested Fragments\n\nTreating routes as flat led to even worse problems for s. Consider the following:\n\n```js\n\nHome\nMessages\nMessage History\n\n```\n\nThat’s a lot of work and repetition for a simple nested UI!\n\nv9.0.0 introduces , backporting the old behavior as . If you’re importing Fragment in your current project, your behavior won’t change! We’ll make the default in the next major version.\n\n lets you nest fragments to match your UI hierarchy to your route hierarchy, much like the component does in react-router. Child fragments derive their parent route from their parent fragment. Given a URL of /home/bio/dat-boi, and the following elements:\n\n```js\n\nHome\nBios\nDat Boi\n\nSomething something whaddup\n\n```\n\n...React will render:\n\n```js\n\nHome\nBios\nDat Boi\n\nSomething something whaddup\n\n```\n\nIf you’re using one Fragment type exclusively, you can take advantage of ES6 module syntax to reduce verbosity:\n\n```js import { RelativeFragment as Fragment } from 'redux-little-router'; // or import { AbsoluteFragment as Fragment } from 'redux-little-router'; ```\n\nWe believe that embracing the hierarchy of routes empowers the URL to make smarter decisions about application state. Try out the new version of Redux Little Router today! We’d love feedback on the new APIs.\n\nThere’s more to come! We plan on making the store enhancer initialization less ambiguous between client and server renders by integrating into the node server ecosystem. Stay tuned!"
],
"categories": {
"primary": "frontend",
"others": [
"product"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-09-15-introducing-react-game-kit": {
"href": "https://nearform.com/digital-community/09/15/introducing-react-game-kit",
"postType": "blog",
"slug": "digital-community-09-15-introducing-react-game-kit",
"date": "2016-09-15",
"title": "Introducing React Game Kit",
"authors": [
"EMMA BRILLHART"
],
"content": [
"See Ken’s slides from his React Next talk.\n\nIntroducing React Game Kit\n\nReact Game Kit is Formidable’s newest release, written by the one and only Ken Wheeler. Since Ken is busy killing it in Israel at ReactNext, I’m giving a rundown of what React Game Kit is and why you should use it. Ken’s slides from his React Next talk can be found here, and I highly recommend you take a look at them.\n\nGame Dev Basics\n\nShould Can you build a game with React?\n\nYou sure can!\n\nWhy would you build a game with React?\n\nThe same game code can work on the web, iOS & Android\nYou primarily write React code\nYou dont feel like learning Unity\nYou can hot reload game logic\n\nWhat is a game?\n\n“A form of play or sport, especially a competitive one played according to rules and decided by skill, strength, or luck.”\n\nToday we are going to learn how to make a 2d platformer game with ReactJS.\n\nBasic Concepts\nGame Loop\n\nA programmatic loop that gets input, updates game state and draws the game.\n\nTick\n\nEach step of the loop.\n\nUpdate Function\n\nA function called on each tick where game logic is checked.\n\nStage\n\nThe main game container to which game entities are added.\n\nSprite\n\nAn often animated bitmap graphic derived from a larger tiled image of states and steps.\n\nTileMap\n\nA large graphic created by rendering a matrix of position indexes derived from a smaller set of common tiles.\n\nPhysics Engine\n\nA class that simulates physical systems.\n\nRigid Body Physics Engine\n\nA physics engine that assumes that physical bodies are not elastic or fluid.\n\nPhysics World\n\nA class that provides a set of conditions that the simulation abides by.\n\nPhysics Body\n\nA class that acts as an entity inside the physics world.\n\nThis sounds hard.\nBut it doesn’t have to be!\n\nIntroducing: react-game-kit. A collection of ReactJS components and utilities that help you make awesome games. It’s pretty fun, all of the slides here are built with it. Oh, and it works on React Native too!\n\nThe Loop\nHow does the loop work?\nrequestAnimationFrame\nlet animationFrame; \nconst loop = () => { // Update logic\n animationFrame = requestAnimationFrame(loop); \n}\nanimationFrame = requestAnimationFrame(loop);\njsCopy to clipboard\nHow can I implement this in React with react-game-kit?\nclass Game extends Component { \n render() { \n return ( \n // Child components get this.context.loop \n ) \n } \n}\njsCopy to clipboard\nWait, how does context work?\nclass Example extends Component { \n static contextTypes = { loop: PropTypes.object, };\n loop = () => { \n //Do stuff here \n }; \n componentDidMount() { \n this.loopID = this.context.loop.subscribe(this.loop); \n }\n componentWillUnmount() { \n this.context.loop.unsubscribe(this.loopID);\n }\n}\njsCopy to clipboard\nScaling\nHow can we size and scale our game?\n\ntransform: scale()\nRounding errors on subpixel floats mean we have to manually round & scale. react-game-kit provides a Stage component to help with this.\n\nclass Game extends Component { \n render() { \n return ( \n // Child components get this.context.scale \n ) \n }\n}\njsCopy to clipboard\n\nMost screens you are targeting will have a 16:9 aspect ratio.\n\ngetWrapperStyles() { \n const x = Math.round(this.state.x * this.context.scale); \n return { \n position: 'absolute',\n transform: `translate(${x}px, 0px) translateZ(0)`,\n transformOrigin: 'top left', \n }; \n} \njsCopy to clipboard\nThat’s cool, but won’t my images be blurry?\n\nNot if you pixelate them!\n\ngetImageStyles() { \n const scaledWidth = Math.round(this.props.width * this.context.scale); \n return { width: scaledWidth, imageRendering: 'pixelated' }; \n} \njsCopy to clipboard\nSprites\nSo, how do sprites work?\n\nCreate a sprite map.\n\nclass Sprite extends Component { \n render() { \n return () \n } \n} \njsCopy to clipboard\ngetImageStyles() { \n const left = this.state.step * tileWidth; \n const top = this.state.state * tileHeight;\n return { \n position: 'absolute', \n transform: `translate(-${left}px, -${top}px)`, \n }; \n} \njsCopy to clipboard\nreact-game-kit provides a Sprite component to simplify this process.\nreturn ( );\njsCopy to clipboard\nTilemaps\nWhat in the world is a tilemap?\n\nTile maps use a tile atlas to use a few graphics to create a “level”.\n\nTile maps have tiles and layers.\n\nOk, so what does the map look like?\nconst tileMap = { \n rows: 4, \n columns: 8, \n layers: [ \n [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, ], \n ], \n};\njsCopy to clipboard\nParsing a tile map\nlayers.forEach((l, index) => { \n const layer = [];\n for (let r = 0; r < rows; r++) { \n // Loop over rows \n for (let c = 0; c < columns; c++) { \n // Loop over columns \n const gridIndex = (r * columns) + c; \n // Get index in grid \n if (layer[gridIndex] !== 0) { \n // If it isn't 0 \n layer.push({ row: r, column: c, tileIndex: layer[gridIndex] }) \n } \n } \n } \n}\njsCopy to clipboard\nreact-game-kit provides a TileMap component to simplify this process.\ngetTileStyles(column, row, size) { \n const left = column * size; \n const top = row * size; \n return { \n height: size, \n width: size, \n overflow: 'hidden', \n position: 'absolute', \n transform: `translate(${left}px, ${top}px)`, \n }; \n}\njsCopy to clipboard\nWhy not just make it one big image?\nif (tile.index === 2) { \n return ; } return ; )} layers={[[ 0, 0, 2, 0, 2, 0, 1, 1,]]\n}\njsCopy to clipboard\nPhysics Engine\n\nYou probably don’t need a physics engine.\n\n// Update loop \nconst update = () => { \n if (rightKeyPressed) { character.x += 1; } \n if (leftKeyPressed) { character.x -= 1; } \n};\njsCopy to clipboard\n\nBut let’s say you do want physics. react-game-kit provides physics helpers provided by matter-js.\n\nclass Game extends Component { \n render() { \n return ( \n // Children get this.context.engine \n ) \n }\n}\njsCopy to clipboard\nclass WorldExample extends Component { \n physicsInit = (engine) => { \n const ground = \n Matter.Bodies.rectangle( 512, 448, 1024, 64, { isStatic: true });\n Matter.World.addBody(engine.world, ground); \n } \n render() { return ; }\n} \njsCopy to clipboard\n\nWhen using matter-js physics, it’s important to do physics updates after the world has updated.\n\nclass WorldChild extends Component { \n update = () => { \n //Logic goes here \n } \n componentDidMount() { \n Matter.Events.on(this.context.engine, 'afterUpdate', this.update); \n } \n componentWillUnmount() { \n Matter.Events.off(this.context.engine, 'afterUpdate', this.update);\n }\n} \njsCopy to clipboard\nUsing physics bodies\nclass WorldChild extends Component { \n //... \n render() { \n return ( { this.body = b; }} > ); } } \njsCopy to clipboard\nclass WorldChild extends Component { \n move = (body, x) => { \n Matter.Body.setVelocity(body, { x, y: 0 }); \n }; \n update = () => { \n const { body } = this.body; \n if (keys.isDown(keys.LEFT)) { \n this.move(body, -5); \n } else if (keys.isDown(keys.RIGHT)) { \n this.move(body, 5); \n } \n };\n //...\n}\njsCopy to clipboard\nPerformant use of physics data for positioning\nmobx\nimport { observable } from 'mobx';\nclass GameStore { \n @observable characterPosition = { x: 0, y: 0 }; \n} \n\nexport default new GameStore(); \njsCopy to clipboard\nimport { observer } from 'mobx-react'; \n\n@observer class WorldChild extends Component { \n move = (body, x) => { \n Matter.Body.setVelocity(body, { x, y: 0 }); \n }; \n update = () => { \n const { body } = this.body; \n if (keys.isDown(keys.LEFT)) { \n this.move(body, -5); \n } else if (keys.isDown(keys.RIGHT)) { \n this.move(body, 5); \n }\n store.characterPosition = body.position; \n }; \n //...\n } \njsCopy to clipboard\nimport { observer } from 'mobx-react'; \n \n@observer class Enemy extends Component { \n getStyle() { \n const {x, y} = store.characterPosition; \n return { position: 'absolute', transform: `translate(-${x}px, -${y}px)` }\n } \n\n render() { return } \n} \njsCopy to clipboard\n\nNow go forth and build some dope games in React, and check out the source code for react-game-kit."
],
"categories": {
"primary": "frontend",
"others": [
"product"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-09-19-victory-0-12-0-the-one-true-tooltip": {
"href": "https://nearform.com/digital-community/09/19/victory-0-12-0-the-one-true-tooltip",
"postType": "blog",
"slug": "digital-community-09-19-victory-0-12-0-the-one-true-tooltip",
"date": "2016-09-19",
"title": "Victory 0.12.0: The One True Tooltip",
"authors": [
"LAUREN EASTRIDGE"
],
"content": [
"Victory Update: Tooltips Are Here!\n\nWe are pleased to announce that the latest release of Victory finally includes tooltips! This feature has been in demand for a while, but nailing down an implementation strategy proved challenging. No other chart element exhibits the range of behaviors or visual possibilities as the simple tooltip.\n\nThe One True Tooltip was impossible to build because it was impossible to define. Instead, we built a set of components that could be configured and combined to create The One True Tooltip however it might look.\n\nTooltip Demonstration\n\n\n {pointer}\n \n );\n }\n}\njsCopy to clipboard\n\nOnce written, the custom component is supplied to VictoryTooltip like `x: ${datum.x} \\n y: ${datum.y}`}\n data={[\n { x: 1, y: -5 },\n { x: 2, y: 4 },\n { x: 3, y: 2 },\n { x: 4, y: 3 },\n { x: 5, y: 1 },\n { x: 6, y: -3 },\n { x: 7, y: 3 },\n ]}\n/>\njsCopy to clipboard\n\nWe decided it was a little tedious to supply the same data to each component, so we made an enhancement to the VictoryGroup wrapper component so that it can supply data to all its children. The equivalent example using VictoryGroup looks like this:\n\n\n `x: ${datum.x} \\n y: ${datum.y}`}\n />\n \njsCopy to clipboard\n\nMuch cleaner!\n\nCustomizing VictoryVoronoiTooltip\n\nVictoryVoronoiTooltip is an instance of the VictoryVoronoi component that uses VictoryTooltip as a default labelComponent. Because of this modular assembly, any customizations that are possible for VictoryTooltip are easily applied to VictoryVoronoiTooltip by simply replacing components.\n\nTo specify props such as cornerRadius and pointerLength from VictoryVoronoiTooltip, add them directly to the label component labelComponent like so:\n\n\n {(image) => \njsCopy to clipboard\n\nRequiring a function that takes the active image and returns the active children makes Old package to diff (default `@latest`)\n -n, --new New package to diff (default `process.cwd()`)\n --no-colors Disable colors in the outputted diff\n\nExamples:\n\n # Compare (old) npm registry `latest` version vs. (new) local `process.cwd()`\n $ publish-diff\n\n # Compare (old) npm version vs. (new) latest npm version\n $ publish-diff -o rowdy@0.4.0 -n rowdy@latest\n\n # Compare (old) git tag/hash vs. (new) git tag/hash\n $ publish-diff -o FormidableLabs/rowdy#v0.4.0 -n FormidableLabs/rowdy#v0.5.0\nbashCopy to clipboard\nLet’s diff some releases\n\nThe most common expected use case is a local source repository checkout that is ready for publication to the npm registry. Running\n\n$ publish-diff\nbashCopy to clipboard\n\nchecks the local project against the latest npm registry package to produce a diff analogous to what will happen when a developer finally types npm publish.\n\nUnder the hood, publish-diff relies on the quite flexible npm pack command, which opens up a lot of possibilities for diffing versions of local and/or remote packages via:\n\nlocal file (/PATH/TO/my-project)\nnpm registry tag (my-project@beta) / version (my-project@1.2.3)\ngit tag (GitHubOrg/my-project#v1.2.3) / hash (GitHubOrg/my-project#fe25a22)\n\nIn this manner, we can easily audit previous publications to the npm registry or git tagged source. For example, let’s see the diff for the v0.4.0 vs. v0.5.0 release of rowdy (a functional test wrapper):\n\n$ publish-diff -o rowdy@0.4.0 -n rowdy@0.5.0\nbashCopy to clipboard\n\nwhich gives us:\n\nFrom here, there are lots of other comparison options:\n\n # Diff git tag/hash vs. latest on npm registry\n $ publish-diff -o FormidableLabs/rowdy#v0.4.0 -n rowdy\n $ publish-diff -o FormidableLabs/rowdy#504735c -n rowdy@latest\n $ publish-diff -o FormidableLabs/rowdy#v0.4.0 -n rowdy@latest\n\n # Diff two old versions from git tag/hash\n $ publish-diff -o FormidableLabs/rowdy#v0.4.0 -n FormidableLabs/rowdy#v0.5.0\n $ publish-diff -o FormidableLabs/rowdy#504735c -n FormidableLabs/rowdy#fe25a22\nbashCopy to clipboard\n\nThe ability to diff past published versions is not only useful for package maintainers, but for consuming projects as well. For example, when deciding whether or not to upgrade a given library dependency to a specific version, a developer can now get a very accurate picture of how the code in that dependency will change upon upgrade.\n\nPublish with confidence\n\nWith this brief tour under our belt, publish-diff offers a simple means to diff your packages right before publication to ensure everything is correct. The project is available at https://github.com/FormidableLabs/publish-diff and we welcome any comments, issues, and feature requests.\n\nHappy diffing!"
],
"categories": {
"primary": "backend",
"others": [
"oss",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-10-25-on-releasing-my-first-oss-project-at-thirty-five": {
"href": "https://nearform.com/digital-community/10/25/on-releasing-my-first-oss-project-at-thirty-five",
"postType": "blog",
"slug": "digital-community-10-25-on-releasing-my-first-oss-project-at-thirty-five",
"date": "2016-10-25",
"title": "On releasing my first OSS project at thirty-five",
"authors": [
"JASON WILSON"
],
"content": [
"A couple weeks ago I released my first open source project at the ripe old age of thirty-five. A dinosaur by tech industry standards or whatever. Computers and programming were something that always came naturally to me in my youth— my first languages were C and x86 assembly. Growing up I developed ANSI viewers for the local art scene and eventually joined ACiD at the age of fifteen. At the time this was equivalent of a lifetime achievement award in my young mind.\n\nThe ANSI art scene was generally closed source and everyone held their latest tricks for smooth text mode scrolling (mostly written in hand rolled assembly) and rendering techniques close to their chest. This is how groups differentiated themselves by having the best art, best viewer, app generators, etc.\n\nI distinctly remember the first time I installed linux (Redhat 5.2) sometime in 1999 on my parents Intel Pentium 200mhz computer. I’ve been a user of open source ever since. Dabbling in the optimal kernel configuration for whatever hardware I was using at the time. In school we used Linux, GCC, and everything in-between. In my first job out of school at Boeing we were fortunate enough to adopt open source and incorporate boost, log4j, and other libraries into our software.\n\nDo you see a trend here? Always consuming open-source but never giving back. Fast forward to present day where collaborative tools like GitHub have combined social networking and coding, forever changing the landscape. I’m fortunate enough to work at Formidable, which embraces and encourages open-source contributions. This is table stakes for the modern developer; even companies like Microsoft have changed their stance on OSS.\n\nIn the beginning of October I got an itch to build something and finally give back. Inspired by Ken Wheeler’s webpack-dashboard I had the idea to build something similar for Node.js. I’ve been building systems in Node for the last four years and felt the constant pain of monitoring a Node process from the console. Why not slap a terminal based front-end on it?\n\nAnd so I did. Hunkering down evenings and weekends to create and build for myself and the greater community. As someone who has traditionally worked on framework and infrastructure teams it was exciting to navigate blessed-contrib’s endless configurability to create nodejs-dashboard. I iterated on various layouts trying to get the graphs looking just right while not taking up too much screen real estate. Here I was again developing console based apps, not too dissimilar from my days slinging ANSI viewers in the early 1990s. After two weeks I was happy enough with the progress to call it GEFN (good enough for now) and hit publish.\n\nThoughts of excitement and dread raced through my mind. Would people like it? How many stars would I get? What if I don’t get any? Does it matter? The morning of the release it bounced back and forth from first to second page on Hacker News. Then much to my surprise it hit escape velocity and remained on front page for the remainder of the day. Nerd stardom achievement unlocked. Github stars slowly rising into the hundreds, then thousands. I had achieved “success” (validation?) on my first OSS project at thirty-five years old. It felt great to give back to a community I’d taken from for so long.\n\nThe tech industry has come under heavy fire lately for cases of age discrimination. I don’t doubt that this is happening in our industry and others. But the reality is we’re not miners on the frontier and compilers don’t discriminate against the developer (unless you write broken code). So I want to leave you with this: whether you’re an industry veteran or just getting around to “Hello World”, just go for it. Build something you want to use and release it open-source. It will change you."
],
"categories": {
"primary": "oss",
"others": []
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-12-01-introducing-whackage": {
"href": "https://nearform.com/digital-community/12/01/introducing-whackage",
"postType": "blog",
"slug": "digital-community-12-01-introducing-whackage",
"date": "2016-11-01",
"title": "Introducing Whackage",
"authors": [
"JANI EVÄKALLIO"
],
"content": [
"A multi-repo developer tool for React Native app and library authors\n\nWe love React Native. Sometimes, though, it doesn’t feel like it reciprocates your love. One such time is when working with codebases that span across multiple repositories. When you want to make changes to a package your app depends on, achieving that snappy hot-reloading workflow you’ve grown accustomed to suddenly becomes a lot more difficult.\n\nPackager woes\n\nIf you’ve developed libraries for React Native, you’ve probably noticed that “Packager”, the development server that bundles your app’s JavaScript sources, doesn’t play nicely with npm link. You might have experienced the dreaded Haste @providesModule name collision error, or had the packager crash with a mysteriously missing Babel preset.\n\nThe React Native core team is working on improving the Packager, and better multi-repo support is high on the wish list. In the meantime, while developing Victory Native, our charting library for React Native, we needed a solution that works today.\n\nEnter whackage\n\nWhackage is a hot-reloading-friendly npm link replacement that works with React Native. It synchronizes changes in your local workspace to your project’s node_modules without using symlinks, and automatically generates a packager blacklist for linked modules to avoid Haste naming collisions.\n\nYou can install Whackage with:\n\n$ npm install -g whackage\nbashCopy to clipboard\n\nYou now have a command line tool called whack. To get started, initialize a new whackage.json in your project directory with whack init, and link a local copy of your package with whack link :\n\n$ whack init\n$ whack link ../../victory-core-native\nbashCopy to clipboard\n\nYour project now has a whackage.json file that looks something like this:\n\n{\n \"include\": \"/**/*.js\",\n \"exclude\": [\"/node_modules/*\", \".babelrc\", \".git\"],\n \"dependencies\": {\n \"victory-core-native\": \"../../victory-core-native\"\n }\n}\njsonCopy to clipboard\n\nAll that is left to do is to start the Whackage server with whack run . In a typical React Native project the npm task to start the Packager is start, in which case the corresponding whackage command is:\n\n$ whack run start\nbashCopy to clipboard\n\nAnd that’s it! You can now edit code in the linked package, and see the changes reflected in your app immediately, as they should be.\n\nGive it a whack\n\nWhackage is, suffice to say, a blunt instrument. As its name suggests, it’s a bit wacky and a bit hacky, but it gets the job done. We wrote it to scratch our own itch, and we hope it can be helpful to you, too.\n\nThe source code is available at https://github.com/FormidableLabs/whackage. For more information, read the docs, or turn to whack --help for assistance."
],
"categories": {
"primary": "mobile",
"others": [
"frontend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-pino-the-fastest-node-js-logger-for-production": {
"href": "https://nearform.com/insights/pino-the-fastest-node-js-logger-for-production",
"postType": "blog",
"slug": "insights-pino-the-fastest-node-js-logger-for-production",
"date": "2016-11-08",
"title": "Sematext Guest Post: PINO – The Fastest Node.js Logger for Production",
"authors": [],
"content": [
"Increasing the Performance of Node.js Applications\n\nGuest post by Stefan Thies from Sematext. Thanks Stefan.\n\nDuring the Node Interactive event in Amsterdam we had the pleasure of speaking with Matteo Collina from NearForm about the blazingly fast logger “Pino”.\n\nIn this post we’ll share our insights into Pino and the additional steps required to have a reliable and fast log management solution for Node.js applications.\n\nLet’s first summarize Matteo’s talk: The tool named “ Pino ” was developed by NearForm to increase the performance of Node.js applications by making logging as fast as possible .\n\nThe idea is simple: the faster the logger the less CPU it uses, the more CPU time the app will have to serve e.g., HTTP requests. NearForm tuned Pino to be the fastest logger for Node.js, thus increasing throughput and reducing the response latency of their customers’ apps, which of course saves money for cloud computing resources!\n\nPerformance of Node.js logging frameworks. Source: The cost of Logging\n\nWhy is Pino so fast?\nOptimized JS code, e.g. for serialisation and string operations\nPino streams logs in line-delimited JSON format to any given output stream (default: standard output stream)\nMinimalistic functionality in the Pino core\nPino has no functionality for log rotation or compression\nPino itself does not include transport layers or format templates. Transports are separate modules, running in a separate process. For example:pino-elasticsearch reads from standard input and inserts Pino-formatted logs into Elasticsearch, e.g.:node app.js | pino-elasticsearch -i log_index -H elasticsearch-server -p 9200\nWhat makes Pino so easy to use?\nThe Pino API and output format are compatible with Bunyan (popular logging framework). With very little code modification Bunyan users can switch to Pino! Code Example: [[BASE64:shell]]dmFyIHBpbm8gPSByZXF1aXJlKCdwaW5vJykoKQpwaW5vLmluZm8oJ2hlbGxvIHdvcmxkJykKcGluby5lcnJvcigndGhpcyBpcyBhdCBlcnJvciBsZXZlbCcpCnBpbm8uaW5mbygndGhlIGFuc3dlciBpcyAlZCcsIDQyKQ==\nSeparate tools for transports connected via Unix pipes:\npino-elasticsearch\npino-socket\nPlugins for the most popular web frameworks are available, logging request details including response times:\npino-hapi\npino-express\npino-koa\npino-restify\nPino-http\nAwesome, speeding up logging for Node.js web frameworks! But what needs to be done to use this in production?\n\nThe minimalistic approach of Pino requires some additional steps to be done. Running additional Node.js tasks in separate processes helps utilize the CPU resources much better. For example, Pino log events could be streamed to Elasticsearch with pino-elasticsearch using 2 separate processes.\n\nIf you don’t want to deal with maintaining your own Elasticsearch you can use Logsene- Hosted Elasticsearch for Logs and simply point pino-elasticsearch to it; just replace the IndexName with the token for your Logsene app.\n\nnode my-api-service.js | pino-elasticsearch --index IndexName -H https://logsene-receiver.sematext.com -p 443 -b 1000\n\n\nThis means there is no slowdown in the Node.js web server app bound to one CPU core (assuming no worker processes used), while the elasticsearch transport might be processed on the second core.\n\nPino-elasticsearch is designed for simplicity and plays well withunix pipes. Our tests did show that pino-elasticsearch uses nearly constant memory (<100 MB) and low CPU <5% during our tests – a low overhead to stream logs in Elasticsearch!\n\nHowever, in production we might face more complex situations, e.g.:\n\nNetwork connections to Elasticsearch might become slow or they might break\nElasticsearch might reject requests when it gets overloaded\n\nTo avoid dropped logs in either situation it is safer to persist logs locally first, in case of shipping failure, and track the status of log forwarding for later transmission.\n\nWriting logs to standard output is “state-of-the-art” today because it fits the logging model used by Docker, Upstart and Systemd well.\n\nThat’s why the best practice is to combine apps using Pino with existing tools:\n\nUpstart and Systemd to collect standard output and error stream and write them to files\nLog rotate or journald to handle log file rotation\nLog shippers to forward logs securely and reliably to central log storage and indexing service where search and analysis takes place.\n\n[1]\n\nNote that if your application is deployed in a Docker container there are other ways of handling logs, for example by Docker logging drivers or agents like Sematext Docker Agent – probably the easiest way to deal with this issue! But as not everybody moved to Docker in production yet let’s have a look at typical Linux deployments.\n\nWe checked several logging setups of a restify server and compared logging with Bunyan and Pino on a server with 4 CPU cores and 8 GB RAM. Then we added a log shipper to forward logs to Elasticsearch/Logsene to see the impact on the performance of the restify server.\n\nThe chart below shows that using restify-pino-logger instead of restify-bunyan-logger increased the number of HTTP requests per second by more than 27%.\n\nAdding a reliable log shipper in parallel, which used the second CPU core, slowed down the restify server by only 3%. This means a performance gain of 24% for the restify server itself (vs. bunyan logger) and having a reliable log shipper setup in parallel.\n\n[2] Note: The move from Bunyan to Pino is really easy, as Pino API is compatible with Bunyan. We used the restify servers from https://github.com/pinojs/restify-pino-logger/tree/master/benchmarks for our tests.\n\nEXAMPLE: LINUX DEPLOYMENT\n\nLet’s walk through a complete Linux deployment of a simple Node.js service. One that uses the Pino Logger (API), upstart, logrotate and logagent to ship logs into Elasticsearch/Kibana. Upstart service file for the Node.js server application: We assume Node.js app logs with Pino to standard output/error stream, e.g. my-api-service/app.js:\n\nvar restify = require('restify')\nvar server = restify.createServer({name: 'app'})\nserver.use(require('restify-pino-logger')())\nserver.get('/', function (req, res) {\n res.send('hello world')\n})\njsCopy to clipboard\n\nUpstart collects logs in /var/log/upstart/serviceName.log – a typical upstart config file looks like this:\n\n#/etc/init/my-api-service.conf\nstart on filesystem and net-device-up IFACE!=lo\nstop on runlevel[016]\nrespawn\nrespawn limit 5 60\nkill timeout 30\nsetuid nobody\nsetgid nogroup\nlimit nofile 32768 32768\n# Token for SPM Node.js Monitoring - https://sematext.com/spm/integrations/nodejs-monitoring\nenv SPM_TOKEN=XXXX\nenv NODE_MONITORING=\"-r /usr/lib/node_modules/spm-agent-nodejs\"\nenv NODE_ENV=production\nenv WEB_CONCURRENCY=2\nenv PORT=80\nenv NODE_OPTIONS=\"--max-old-space-size=250\"\nenv APP_JS_FILE=/usr/lib/node_modules/my-api-service/app.js\nchdir /tmp\nexec /usr/bin/node $NODE_MONITORING $NODE_OPTIONS $APP_JS_FILE\n\n\nThe service needs to be enabled:\n\nsudo initctl reload-configuration\nsudo start my-api-service\n\nLogrotate\n\nLogrotate is a Linux service typically run once a day to rename, compress and delete old log files, while keeping the last N log files.\n\nBy default logrotate is configured to rotate files weekly:\n\n> cat /etc/logrotate.conf\n# see \"man logrotate\" for details\n# rotate log files weekly\nweekly\n# ...\n# packages drop log rotation information into this directory\ninclude /etc/logrotate.d\n\n\nIf you like to rotate logs daily (or by size) for the new service, then you have to create a new file in /etc/logrotate.d to define service specific rules:\n\n/var/log/upstart/my-api-service {\n daily\n missingok\n rotate 8\n compress\n delaycompress\n notifempty\n}\n\nLog shipper setup for Elasticsearch\n\nLog shippers can stream our /var/log/upstart/my-api-service.log log file to Elasticsearch. Most of the popular log shippers can deal with rotating log files and queue logs in local buffers when Elasticsearch is not available.\n\nLet’s use @sematext/logagent so we don’t need to install Ruby or Java on our Node.js server and because Logagent can deal with JSON logs in Pino/Bunyan format; also, because we want to make sure we don’t lose any logs if the destination is temporarily unreachable (Logagent will buffer logs locally and try to ship them later). Let’s also skip the Elasticsearch setup and just configure it to ship logs to Logsene, which is compatible with the Elasticsearch API:\n\nsudo npm i -g @sematext/logagent\n# setup with the index name / Logsene Token and a glob pattern to watch files\nsudo logagent-setup LOGSENE_TOKEN ‘/var/log/**/*.log’\n\n\nDone! Logagent runs as a Linux service, watches log files produced by Pino, and ships them reliably to Logsene, with local buffering and all.\n\nIf you prefer to use a different Elasticsearch server (e.g. your own) you must configure the Elasticsearch URL and index name in /etc/sematext/logagent.conf, e.g.:\n\noutput:\n elasticsearch:\n old_url: http://elasticsearch-server:9200\n index: my_api_service_logs\n\n\nThen restart the service with sudo service logagent restart . You might want to check logagents status messages in /var/log/upstart/logagent.log :) As soon as logs arrive in ELK, we can use Kibana (if you are using Logsene, you can use Kibana that is integrated in it) to analyze Express/Hapi/Restify logs created by Pino:\n\n[3]\n\nWrap up\n\nNode.js Apps can easily benefit from Pino. Switching to Pino for logging will save precious CPU cycles typically spent on inefficient logging. The proper production deployment needs to include additional tasks like log rotation, and log forwarding to a centralized Log Management System to analyze the logs. Using the right tools and settings, a fast & reliable setup can be achieved quickly and easily.\n\nMore information on Pino\nAnnouncing Pino V5.0.0\nCollect fast Pino logs in an Azure Event Hub"
],
"categories": {
"primary": "backend",
"others": [
"perf",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-11-09-flexible-charting-in-react-with-victory": {
"href": "https://nearform.com/digital-community/11/09/flexible-charting-in-react-with-victory",
"postType": "blog",
"slug": "digital-community-11-09-flexible-charting-in-react-with-victory",
"date": "2016-11-09",
"title": "Flexible Charting in React with Victory (and Introducing FormidableCharts)",
"authors": [
"LAUREN EASTRIDGE"
],
"content": [
"Victory: React charts tailored to your data\n\nCharting directly with d3 can be difficult, but other libraries are often too simplistic. Enter Victory: React charting that is easier than direct use of d3.js but with as much flexibility as possible. Victory allows fully customized charts ranging from basic to highly complex. To learn more about the complete Victory API, check out the documentation.\n\nFlexibility and Customizability\nModular Chart Components\n\nVictory is built around a set of modular chart components that can be combined to create custom charts. Victory favors composition over configuration, allowing users to build up complicated visualizations from smaller components.\n\n\n \n \n \njsCopy to clipboard\nFunctional Styles\n\nFunctional styles and props are supported whenever possible. Want to make data points below a certain threshold a different color than the rest of the data? It’s easy in Victory.\n\nstyle={{data: {fill: (datum) => datum > 0 ? \"red\" : \"blue\"}}}\njsCopy to clipboard\nCustom Components\n\nVictory components render default primitive components (like Area, Line, Bar, VictoryLabel etc.). These simple, stateless components are defined in defaultProps, and my be easily altered or replaced with completely custom components. Modify the primitives or write components that you need from scratch without having to abandon the rest of the Victory library.\n\n\n }\n labelComponent={}\n />\n \njsCopy to clipboard\nReact Native Compatible\n\nTo use the full (identical!) Victory API on iOS or Android, simply add Victory Native to your React Native project with npm install --save victory-native. Victory Native uses react-native-svg to build the same Victory charts that you love from the browser on mobile operating systems.\n\nFlexible Event System\n\nVictory has a flexible event system with no hard coded events. This allows users to attach event handlers to any specific rendered element or elements, and modify any other specific element or elements. Event handlers are called with the event and all of the props that define the specific element that the event handler was attached to, and modification functions are called with all of the props that define each targeted element.\n\nSome components, like VictoryTooltip, also have defaultEvents. Default events on components supplied via dataComponent, labelComponent, or similar, are picked up. This means that labelComponent={\n ${ render(
\n`;\nconst header = render(
. This cacheKey uniquely and globally identifies the expected content of the render - not just for the node you're labeling. The cacheKey should also incorporate whatever props or context might influence how a component is rendered.\n\nHere's what it might look like:\n\nconst Child = ({ val }) => (\n \n ComponentA\n
\n);\n\nconst Parent = ({ toVal }) => (\n \n {\n _.range(toVal).map(val => (\n
\n);\n\nPromise.resolve()\n // The first render will take the expected duration.\n .then(() => render(\n { `Our counter is at: ${state.counter}` }\n
\n));\n\nrender(\n { `Our counter is at: ${state.counter}` }\n
\n)));\n\nrender(\n This is the GrandChild.\n {state.fromParent}\n {state.fromGrandParent}\n
\n));\n\nconst Parent = provideState({\n initialState: () => ({ fromParent: \"ParentValue\" })\n})(() => (\n \n This is the Child.\n
\n));\n\nconst GrandParent = provideState({\n initialState: () => ({ fromGrandParent: \"GrandParentValue\" })\n})(() => (\n \n This is the Parent.\n
\n));\njsCopy to clipboard\nEffective Immediately\n\nFreactal combines side effects and state updates into a single abstraction, simply named \"effects.\" Effects resemble an action/reducer pair in Redux, but they embrace async as a first principle and allow for composable handling of intermediate state.\n\nprovideState accepts an effects key with a value of named effect definitions. An effect is just a function with the signature:\n\n(effects, ...args) => Promise newState>\njsCopy to clipboard\n\nThe effects argument contains all of the defined effects in this container, as well as the effects defined by its ancestors. ...args are any arguments you pass when calling the effect (e.g. the 10 in effects.fetchPosts(10)). That function returns a Promise that resolves to an updater function, state => newState, which resembles a Redux reducer.\n\nDon't memorize that, though! It's painless in practice, and partial application is the bedrock for the powers of effects.\n\nSimple state changes are concise, since freactal automatically Promise.resolves any synchronous updater functions provided:\n\n{\n addOne: () => state => Object.assign({}, state, { counter: state.counter + 1 })\n}\njsCopy to clipboard\n\nEven easier, you can use the softUpdate helper to replace the common Object.assign pattern:\n\n// { counter: 1 }\n{\n increment: softUpdate((state, increment) => ({\n counter: state.counter + increment\n }))\n}\n\neffects.increment(5)\n// { counter: 6 }\njsCopy to clipboard\n\nThis makes the bread and butter of state management a breeze.\n\nAsync effects don't require special treatment, since promises already provide the plumbing. If you define this effect:\n\nprovideState({\n initialState: () => ({ posts: [] }),\n effects: {\n fetchPosts: (effects, limit) =>\n fetch(`/api/posts?limit=${limit}`)\n .then(result => result.json())\n .then(posts => state => Object.assign({}, state, { posts }))\n }\n});\njsCopy to clipboard\n\n...you can call it like this in a child component:\n\n\nconst Child = injectState(({ state, effects }) => {\n const onClick = () => effects.fetchPosts(10);\n return (\n \n { state.posts.map(post =>
{post.body}
) }\n
Fetch some posts! \n
var server = http.createServer((req, res) => {\n for (var i=0; i<1000; i++) {\n server.on('request', function leakyfunc() {});\n }
res.end('Hello World\\n');\n}).listen(1337, '127.0.0.1');\nserver.setMaxListeners(0);
console.log('Server running at https://127.0.0.1:1337/. Process PID: ', process.pid);\ntextCopy to clipboard\n\nEvery request adds another 1,000 leaky listeners. We are going to simulate requests using autocannon – A HTTP benchmarking tool written in node. It can be installed with npm i -g autocannon . If we run autocannon -c 1 -d 60 https://localhost:1337 in one shell, and in another shell view our process: top -pid we will see very high and erratic memory usage for this node process. Our node process has gone wild. So how do you diagnose from here?\n\nMemory leak detection\n\nThe memwatch-next module is great for detecting memory leaks. Lets install into our app as follows: npm install --save memwatch-next . Then, in our code, add:\n\nconst memwatch = require('memwatch-next');\ntextCopy to clipboard\n\nand also a listener for the leak event:\n\nmemwatch.on('leak', (info) => {\n console.error('Memory leak detected:\\n', info);\n});\ntextCopy to clipboard\n\nNow when we run our test again, we see the following emitted:\n\nMemory leak detected:\n{\n growth: 14280616,\n reason: 'heap growth over 5 consecutive GCs (3s) - -2147483648 bytes/hr'\n}\ntextCopy to clipboard\n\nMemwatch has detected a memory leak! Memwatch defines a leak event as:\n\n“‘A leak event will be emitted when your heap usage has increased for five consecutive garbage collections.’”\n\nSee memwatch for more details.\n\nMemory leak analysis\n\nSo weve detected that we have a memory leak great! But what now? The next step is to find where your leak is. The example leak is pretty obvious, but the process for finding one is always the same:\n\nCreate heap dumps at different time intervals.\nCompare the dumps to see whats growing.\n\nLets see two ways of doing this.\n\nNode’s –inspect Flag \n\nNodes --inspect flag landed in node version 6. This feature lets you debug and inspect your node process from within Chromes DevTools . Simply start the application passing the --inspect flag:\n\n$ node --inspect index.js\nDebugger listening on port 9229.\nWarning: This is an experimental feature and could change at any time.\nTo start debugging, open the following URL in Chrome:\n chrome-devtools://devtools/bundled/inspector.html?experiments=true&v8only=true&ws=127.0.0.1:9229/9fe96182-d16e-4028-9e17-3c1d20d8398a\ntextCopy to clipboard\n\nFollow the URL to the inspector and navigate to the memory tab.\n\nDevTools Memory Tab Heres what we will do:\n\nHit the application with autocannon -c 1 -d 60 https://localhost:1337\nTake a heap snapshot after roughly 10 seconds, and again after 30 seconds.\n\nHeap Snapshot Comparison The comparison view shows us what has happened between snapshots. We can see a huge number of closure objects have been created. The delta columns also give us an idea of how much growth there has been. Lets take a deeper look:\n\nFound our Memory Leak Now it is clear that our leakyfunc() is the root of the problem. The Retainers view shows where the reference to our leakyfunc() object is held in memory. We can also find exactly where it was created in our code.\n\nMemory Leak Details For a great overview of memory profiling in DevTools, see Taming The Unicorn: Easing JavaScript Memory Profiling In Chrome DevTools .\n\nHeapdump \n\nnode-heapdump is a great module. If the --inspect option is not available, or you need to create snapshots programmatically, heapdump can be used to create a snapshot from within the application code. For more on heapdump, see the following StrongLoop blog post . Let’s use heapdump in our code now, so that every time a memory leak is detected, we write out a snapshot of the V8 stack to disk:\n\nmemwatch.on('leak', (info) => {\n console.error('Memory leak detected:\\n', info);\n heapdump.writeSnapshot((err, filename) => {\n if (err) console.error(err);\n else console.error('Wrote snapshot: ' + filename);\n})\ntextCopy to clipboard\n\nRun our test again, and you should see some ‘.heapsnapshot’ files created in our application folder. Now in Chrome, launch DevTools (alt-cmd-i on Mac), hit the Memory tab and hit Load to load in our snapshots. Now we can find the leak as we did in the previous section.\n\nConcluding observations\n\nThe above outlines one method of ‘self detecting’ memory leaks in Node and these are my concluding observations:\n\nheapdump has some caveats about snapshot size, etc. Read the fine print carefully. It's also CPU intensive to take a snapshot.\nThere are other ways of generating snapshots with heapdump. Pick whichever way suits your application best, e.g. sending sigusr2 to the process, etc. This is also possible with memwatch-sigusr2.\nConsider where and when you enable memwatch/heapdump. Only enable in testing and not in production. Also, consider how many heapdumps to write out before possibly bailing out of the process. Again, do whatever best suits your application.\nConsider the other ways of detecting memory growth from within your application, i.e. monitoring the size of process.memoryUsage().rss as an alternative to memwatch.\nWhen a memory leak is detected, you should also make sure it gets picked up by your monitoring and alerting tools so humans are told about it.\nBeware of false positives: short memory spikes can appear as leaks to memwatch. If your app suddenly gets busy and uses a big chunk of memory which it takes a while to process, this processing time can span multiple garbage collections and memwatch will report a leak. However, there may be nothing wrong here per se. When your app is done processing, the memory usage should settle down again. What you’re really looking for is persistent reporting of memory leaks. One-off ‘blips’ may actually be ok.\n\nThe gist for source code referred to in this article can be found here . Having trouble with detecting memory leaks in node.js or want to know more on self-detection of memory leaks? You can tweet me or connect with me on LinkedIn .\n\nIf you’d like to know more about how NearForm can help you, contact us here"
],
"categories": {
"primary": "backend",
"others": [
"perf"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-live-editing-react-components": {
"href": "https://nearform.com/digital-community/live-editing-react-components",
"postType": "blog",
"slug": "digital-community-live-editing-react-components",
"date": "2017-05-31",
"title": "Live Editing React Components for Better Documentation",
"authors": [
"PHIL PLÜCKTHUN"
],
"content": [
"As stated in a previous post about documentation, \"Your Docs And You\", the best response to well made docs is simply, \"I get it!\"\n\nWhen writing documentation, it's sometimes easy to forget that someone on the other end will actually be reading your docs. Writing docs can be as difficult as writing a book (albeit, most of the time much shorter). If the docs are too long and dull, you run the risk of the reader losing interest, but if they're too short and prioritize brevity over solid information, they won't end up being helpful.\n\nWhen looking at React projects on GitHub, you can see that the most common approach to documentation is to add some descriptions, a couple of code examples, and one or more demos. With the addition of docs on the API of a library or project, this delivers a fairly complete picture to aspiring users.\n\nWhile these resources are invaluable for learning how single parts of a React project work, it's tough to get a big picture overview of how it feels to use. People learn and become familiar with something tremendously faster when they can simply use and play around with it. So why can't we add a feature enabling this idea in our docs?\n\nA playground full of components\n\nTo solve this problem, Formidable introduced component-playground. It allows you to render editable code side alongside of a demo, and updates the demoed React components in real time.\n\nUsing this project has allowed us to build docs that immediately give you feedback on how a project's components behave, what they look like, and most importantly what it feels like to use the library.\n\nBy enabling users to immediately use a library, we prevent users from becoming frustrated by unpractical, dull docs.\n\nSpreading the live-editing love\n\nThe next step beyond adding live, editable demos to docs, was to add live, editable React code to every project that might need it. This idea subsequently led to the creation of a leaner alternative to component-playground.\n\nSay hi to component-playground's little brother, react-live: a production-focused playground for live editing React components.\n\nReact Live is built to fulfill the additional requirements that arise when adding a live edit feature to a project.\n\nIt is built modularly, so it comes with separate components that are similar to the ones that make up component-playground. Namely an editor, a preview, an error view, and a provider.\n\nIn a production-focused web app that users load every day, it is important for React Live to load as quickly as possible. That’s why it's built using Buble and Prism, bringing its size down to just 65kB gzipped. It's also server-side renderable!\n\nIt is easily and highly styleable, and thus easy to integrate into every possible environment that you might want to use it in.\n\nThese new improvements make it possible to use React Live creatively for new features, some of which we haven’t even thought of yet.\n\nFor example, want to make the homepage itself editable in your next docs? Consider it possible! styled-components docs, the first website using React Live in production, is doing just that.\n\nWe're excited to see what you might come up with! Ping us on Twitter to show us what you’ve created with React Live."
],
"categories": {
"primary": "frontend",
"others": [
"design"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-what-you-need-to-know-about-node-8": {
"href": "https://nearform.com/insights/what-you-need-to-know-about-node-8",
"postType": "blog",
"slug": "insights-what-you-need-to-know-about-node-8",
"date": "2017-05-31",
"title": "What you need to know about Node 8",
"authors": [],
"content": [
"A quick summary of Node 8 features\n\nJames M Snell, Colin Ihrig Node 8 is a major release with a lot of exciting features and is is the next release line to enter Long Term Support (LTS). In this post, we want to give you a quick summary of what it includes. For a more extensive description, please see our announcement post . npm 5.0.0 npm@5 is a large upgrade which has significantly improved performance in almost all common situations and is more robust and fault-tolerant. V8 5.8 V8 5.8 has major improvements in performance and developer facing APIs.It is guaranteed to have forwards ABI compatibility with V8 5.9 and V8 6.0. It also helps set up a pending transition to the new TurboFan + Ignition compiler pipeline. Node.js API (N-API) For Node.js developers who use or create native add-ons, the new experimental Node.js API (N-API) will allow native add-ons to be compiled once on a system and used across multiple versions of Node.js. The new virtual machine agnostic Application Binary Interface (ABI) also enables native addons to work against Microsoft’s Chakra-Core. async_hooks The experimental async_hooks module diagnostics API allows developers a means of monitoring the operation of the Node.js event loop, tracking asynchronous requests and handles through their complete lifecycle. WHATWG URL parser This URL API was implemented around the WHATWG (Web Hypertext Application Technology Working Group ) URL Standard. It matches the URL implementation and API available in modern Web Browsers. Long Term Support Node.js 8 will enter Long Term Support (LTS) in October 2017 Other Important Changes\n\nBuffer changes – A number of important changes have been made to the Buffer API within Node.js.\nImproved support for Promises – Node.js 8.0.0 includes a new util.promisify() API.\nConsole changes – Making the use of console.log() and the other APIs safer.\nStatic Error Codes – We have started the process of assigning static error codes to all errors generated by Node.js.\nRedirecting Process Warnings – Processing warnings such as deprecations may now to redirected to a file\nStream API improvements – For users of the Stream API, new standard mechanisms for destroying and finalising Stream instances have been added\nDebugger changes – node-inspect has been integrated directly into the Node.js runtime. Additionally, the V8 Inspector debugger is a fully supported feature\nExperimental inspector JavaScript API – Provides new ways of leveraging the debug protocol to inspect running Node.js processes.\n\nA quick overview of what’s new in #Nodejs 8 Click To Tweet\n\nNeed Node experts for your next project? Contact us to see how we can help!"
],
"categories": {
"primary": "backend",
"others": [
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-real-world-redux-saga-patterns": {
"href": "https://nearform.com/digital-community/real-world-redux-saga-patterns",
"postType": "blog",
"slug": "digital-community-real-world-redux-saga-patterns",
"date": "2017-06-07",
"title": "Javascript Power Tools Part III: Real-world redux-saga Patterns",
"authors": [
"MATT HINK"
],
"content": [
"In the past two articles, we've talked a lot about redux-saga in the abstract, without much concern for real-world applications. Now that we’re equipped with new knowledge, we're ready to jump in and start putting the pieces back together.\n\nFirst, we'll take a look at a pattern for structuring behavior in single-page applications using redux-saga and redux-little-router, and then we'll build a saga that implements the business logic for a basic form.\n\nPairing redux-saga with redux-little-router\n\nI really love redux-little-router, and I'm not just saying that because it's a Formidable project. When you're already reading from the Redux store to access state and dispatching Redux actions to modify state, it feels very elegant to interact with the browser location the same way. However, when used in conjunction with redux-saga, we gain an additional benefit: the ability to trigger behaviors in response to browser location changes. Why is this beneficial?\n\nWell, first off, it separates the business logic associated with route changes from the view lifecycle. When I'm building a React application, I typically prefer to keep my React components as stateless and declarative as possible, and shoehorning business logic into React components makes that difficult at best.\n\nSecondly, in a large project, it typically behooves us to avoid running too many concurrent sagas at once because it can quickly become difficult to determine the ramifications of dispatching a particular action. If we're monitoring for route changes from within redux-saga, we can ensure that the only sagas running at any given time are those relevant to the current route.\n\nWith this approach, each route becomes a 'mini-application', which starts up when the user navigates to it and shuts down when they navigate away. Let's take a look at how we could implement this.\n\nFirst, we'll start out with a rootSaga which will be the entry point for our entire application.\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n}\njsCopy to clipboard\n\nNext, we'll put together a configureStore function which attaches both redux-little-router and redux-saga to our Redux store.\n\nconst REDUCERS = {\n todos: (state = {}, action) => state,\n debug: (state, action) => action,\n};\n\nconst ROUTES = {\n '/' : {},\n '/todos' : {},\n '/todos/new' : {},\n '/todos/:id' : {},\n '/todos/:id/edit' : {},\n};\n\nexport const configureStore = (initialState = {}) => {\n const {\n reducer : routerReducer,\n enhancer : routerEnhancer,\n middleware : routerMiddleware,\n } = routerForBrowser({ routes: ROUTES });\n\n const sagaMiddleware = createSagaMiddleware();\n\n const store = createStore(\n combineReducers({\n ...REDUCERS,\n router: routerReducer,\n }),\n initialState,\n compose(\n routerEnhancer,\n applyMiddleware(\n sagaMiddleware,\n routerMiddleware,\n ),\n )\n );\n\n sagaMiddleware.run(rootSaga);\n\n const initialRouterState = store.getState().router;\n store.dispatch(initializeCurrentLocation(initialRouterState));\n\n return store;\n};\njsCopy to clipboard\n\nThe first thing to notice here is the initializeCurrentLocation action we're dispatching during initialization. Since we've already started our rootSaga, we can take that action right away.\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n\n const action = yield take(\"ROUTER_LOCATION_CHANGED\");\n const location = action.payload;\n\n console.log(\"Your current location is:\", location);\n}\njsCopy to clipboard\n\nLet's go a little further, and make our saga take any \"ROUTER_LOCATION_CHANGED\" action.\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n\n while (true) {\n const action = yield take('ROUTER_LOCATION_CHANGED');\n const location = action.payload;\n console.log(\"Your current location is:\", location);\n }\n}\njsCopy to clipboard\n\nTo help us make our code more functional, redux-saga provides a helper called takeEvery which does something somewhat similar.\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n\n yield takeEvery('ROUTER_LOCATION_CHANGED', function* (action) {\n const location = action.payload;\n console.log(\"Your current location is:\", location);\n });\n}\njsCopy to clipboard\n\nThere's a problem here, though. Suppose the inner saga was performing some long-running action (emulated by a 10-second delay in the example below). What would happen if the user started navigating around the site quickly?\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n\n yield takeEvery('ROUTER_LOCATION_CHANGED', function* (action) {\n const location = action.payload;\n yield delay(10000); Wait 10 seconds, for some reason\n console.log(\"Your current location is:\", location);\n });\n}\njsCopy to clipboard\n\nYou guessed it. This is Race-condition Central, population: you. Fortunately, redux-saga provides a helper called takeLatest, which ensures that only one saga is running at a time by cancelling the previously running saga when a new action comes in. Let's make the change, and add some exception handling so we can see this behavior in vivo.\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n\n yield takeLatest('ROUTER_LOCATION_CHANGED', function* (action) {\n const location = action.payload;\n try {\n yield delay(10000);\n console.log(\"Your current location is:\", location);\n } finally {\n if (yield cancelled()) {\n console.log(\"Fine, fine! Your location WAS\", location);\n }\n }\n });\n}\njsCopy to clipboard\n\nSo, we've now built a saga with the following properties:\n\nStarts a new saga whenever a location change occurs\nCancels the saga associated with the previous location change\nUses the current location to perform effects\n\nFrom here, we'll take advantage of the third bullet point to branch into different sagas based on the current route. First, let's extract that inline saga into the module scope and give it a name.\n\nfunction* navigationSaga(action) {\n const location = action.payload;\n console.log(\"Your current location is:\", location);\n}\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n yield takeLatest('ROUTER_LOCATION_CHANGED', navigationSaga);\n}\njsCopy to clipboard\n\nNow, let's create some stubs for the different behavior we want to perform at each route.\n\nfunction* navigationSaga(action) {\n const location = action.payload;\n switch (location.route) {\n case '/': {\n break;\n }\n case '/todos': {\n break;\n }\n case '/todos/:id': {\n break;\n }\n case '/todos/new': {\n break;\n }\n case '/todos/:id/edit': {\n break;\n }\n default: {\n break;\n }\n }\n}\n\nexport default function* rootSaga() {\n console.log(\"Starting up the root saga!\");\n yield takeLatest('ROUTER_LOCATION_CHANGED', navigationSaga);\n}\njsCopy to clipboard\n\nThis seems like it could get really messy before long. And what if someone forgets a break statement? Let's keep refactoring.\n\nconst SAGA_FOR_ROUTE = {\n '/' : function* homeSaga() {},\n '/todos' : function* listTodosSaga() {},\n '/todos/:id' : function* showTodoSaga() {},\n '/todos/new' : function* newTodoSaga() {},\n '/todos/:id/edit' : function* editTodoSaga() {},\n};\n\nfunction* navigationSaga(action) {\n const location = action.payload;\n const saga = SAGA_FOR_ROUTE[location.route];\n\n if (saga) {\n yield call(saga, location);\n }\n}\n\nexport default function* rootSaga() {\n yield [\n takeLatest(\"ROUTER_LOCATION_CHANGED\", navigationSaga),\n ];\n}\njsCopy to clipboard\n\nWonderful! In addition to being less verbose, redux-saga can log our sagas by name if they are cancelled due to a ROUTER_LOCATION_CHANGED (in development mode, anyways).\n\nNow, each route-specific saga can react to being started and stopped completely on its own. For instance, if we were to implement editTodoSaga, it would perhaps look something like this.\n\nfunction* editTodoSaga(location) {\n const { id } = location.params;\n\n try {\n // Some behaviors which happen once when the user navigates to this route\n yield put(startedEditingTodo(id));\n yield call(ensureTodoExistsLocally, id);\n\n // Start up some long-running behaviors tied to this saga's lifetime.\n yield all([\n fork(takeEvery, \"BLAH\", handleBlahSaga),\n fork(someOtherLongRunningSaga),\n ]);\n\n } finally {\n // Whether we finished naturally or got cancelled by our parent because the\n // route changed, clean up after ourselves before exiting for good.\n yield put(finishedEditingTodo(id));\n }\n}\njsCopy to clipboard\n'Create Todo' form\n\nAlong those lines, let's take a look at a \"mini-application\" we could implement. Specifically, we'll write an implementation for the newTodoSaga associated with the /todos/new route above.\n\nWe'll assume that somewhere in the view layer there's a form the user's filling out and eventually submitting - we want to implement just the business logic for it. Let's write a quick outline of that.\n\nWait for the user to submit the form\nPerform client-side validation\nIf client-side validation fails, show an error message and start over\nSend the form data to the server\nIf successful, show the user their new to-do\nIf not successful, show an error message and start over\n\nSounds easy enough. Let's start out with an empty saga.\n\nfunction* newTodoSaga() {\n}\njsCopy to clipboard\n\nFirst things first: we need to wait for the user to submit a form. We'll assume that the view layer will dispatch an action called SUBMIT_TODO_FORM with the form data as payload.\n\nfunction* newTodoSaga() {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n}\njsCopy to clipboard\n\nNext, let's write a quick validation function...\n\nconst validateForm = (formData) => {\n if (formData.name === '') {\n throw new Error('\"name\" field must not be blank!');\n }\n}\njsCopy to clipboard\n\n...and invoke it like so.\n\nfunction* newTodoSaga() {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n yield call(validateForm, formData);\n}\njsCopy to clipboard\n\nAt this point, the user can submit their form for validation, but there are two problems: if their input fails validation, we'd like to tell them why, and naturally, we'd like to allow them to resubmit the form.\n\nLet's handle these issues separately.\n\nFirst, we need to allow the user to retry form submission. The easiest way to do this is with a loop which will not exit until their input passes validation.\n\nfunction* newTodoSaga() {\n\n while (true) {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n\n try {\n yield call(validateForm, formData);\n break;\n } catch (err) {\n continue;\n }\n }\n}\njsCopy to clipboard\n\nNote that the continue statement is not strictly necessary, but I consider it good form in this circumstance. It explicitly indicates the expected flow of this procedure, rather than expecting the reader to notice that the loop will start over. We'll see why this is important in a moment.\n\nWe've handled both the valid and invalid cases, so now let's assume that we have a showErrorNotification action creator which dispatches some action indicating that the application should show an error notification.\n\nfunction* newTodoSaga() {\n while (true) {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n\n try {\n yield call(validateForm, formData);\n break;\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n }\n}\njsCopy to clipboard\n\nBy the time we reach the break statement, we know we have valid form data, so let's send it to an API endpoint. We are going to make an assumption that we have a function called createTodo which somehow does this for us. (I've had success with these sorts of thin layers over the Fetch API.)\n\nBecause this API call could fail for all kinds of reasons, we need to make sure we handle those cases by using a simple try-catch block, just like we did before. (Notice that we've removed the break statement.)\n\nfunction* newTodoSaga() {\n while (true) {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n\n try {\n yield call(validateForm, formData);\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n\n try {\n yield call(createTodoApi, formData);\n break;\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n }\n}\njsCopy to clipboard\n\nNotice that our earlier continue statement has now become crucial to the correct behavior. If we hadn't been so conscientious, this would have caused a bug and probably made us feel kinda dumb for missing it.\n\nAnd yes, I can hear you saying \"but this is PROCEDURAL!\" to which I say, \"business logic is inherently procedural, and expressing it as such makes our intent clearer.\" That being said, it is easy to let code like this get out of hand, so it’s important to stay disciplined and keep it focused.\n\nBusiness logic code should tell a story. Like a good storyteller, it should draw focus to the important details and elide over the minute ones. That is, unless we ask for them.\n\nLooking at this code, we see the story:\n\nWait for a SUBMIT_TODO_FORM action\nCall validateForm with action.payload.formData\nIf an error occurs, notify the user and start over\nOtherwise, proceed to the next step\nPOST the form data to the server\nIf an error occurs, notify the user and start over\n\nLooks like we've stuck to the business logic pretty well so far. Now then, the only thing left is to show the user their new to-do. Since we're using redux-little-router, we'll dispatch a Redux action which routes them to the \"show todo\" page.\n\nfunction* newTodoSaga() {\n while (true) {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n\n try {\n yield call(validateForm, formData);\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n\n try {\n const todo = yield call(createTodoApi, formData);\n yield routerPush(`/todos/${todo.id}`);\n break;\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n }\n}\njsCopy to clipboard\n\nThis is good enough, but let's take a short aside to talk about saga structure and separation of responsibilities. This saga's primary responsibility is validating and submitting form data to the server. We know we want to redirect the user's browser after all that happens, but with the way things are currently structured, the sequence of events isn't all that obvious. At a high level, we're doing this:\n\nWhen the user successfully creates a new todo,\nShow the user the newly created todo\n\nLet's separate these two concerns completely.\n\nfunction* awaitSuccessfulTodoCreation() {\n while (true) {\n const action = yield take('SUBMIT_TODO_FORM');\n const formData = action.payload;\n\n try {\n yield call(validateForm, formData);\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n\n try {\n const todo = yield call(createTodoApi, formData);\n return todo;\n } catch (err) {\n yield put(showErrorNotification(err.message);\n continue;\n }\n }\n}\n\nfunction* newTodoSaga() {\n const todo = yield call(awaitSuccessfulTodoCreation);\n\n yield put(showSuccessNotification(\"Congratulations! You created a new todo.\"));\n\n yield routerPush(`/todos/${todo.id}`);\n}\njsCopy to clipboard\n\nNow there's no question as to whether or not todo has been created. We observe that the only 'escape route' from awaitSuccessfulTodoCreation is the return statement, which can only happen after a successful todo creation. So if control returns to newTodoSaga, we're certain that it's time to show a success message and send the user on their way.\n\nHowever, also note what happens if the user navigates away from /todos/new during this process: redux-saga will cancel newTodoSaga. Moreover, if newTodoSaga is still blocking on awaitSuccessfulTodoCreation, the cancellation will propagate \"downward\" to that as well, cleanly exiting the whole shebang without causing any more effects.\n\nEither way, we now have a saga that is responsible for doing one thing only. We can think about this part of the application in isolation, and that goes a long way towards wrangling in the complexity of larger single-page-applications.\n\nConclusion\n\nThis about wraps up our series on redux-saga. I hope you've learned a lot and that you're able to make use of some of these ideas in your own projects! Questions, comments, and suggestions are welcome - I'm available on Twitter at @mhink1103. Let me know what you'd like to see next in the Javascript Power Tools series.\n\n(Also, thanks to all the rad folks at Formidable for their contributions and suggestions- especially Becca Lee, without whom this would all still be a jumbled mess of .md files on my laptop.)"
],
"categories": {
"primary": "frontend",
"others": [
"product"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-why-react-native-is-best-for-native-apps": {
"href": "https://nearform.com/digital-community/why-react-native-is-best-for-native-apps",
"postType": "blog",
"slug": "digital-community-why-react-native-is-best-for-native-apps",
"date": "2017-06-14",
"title": "Why React Native is the Best Choice for Making Native Apps",
"authors": [
"CARLOS KELLY"
],
"content": [
"Companies often eschew building native apps, thinking hybrid solutions will be easier and more cost effective to build. They end up shoving a not-so-responsive web app into some kind of native web frame, which ultimately leads to terrible user experience. Many companies opt to have two separate native teams for iOS and Android instead, but that strategy can be expensive and often comes with its own challenges. React Native, on the other hand, lets you build performant native apps using the same technologies your web team is already using. I’ve been a native app developer for many years and this is why I think React Native has become the best choice for native app development.\n\nModern, developer-friendly tooling\n\nThe React Native CLI lets you start a new native app project that will work on both iOS and Android. These tools are great since they let you start building your app right away, rather than spending a lot of time working on setup and configuration before getting started. The tools are similar to Xcode or Android Studio project templates, so if you’re familiar with those, it’s easy to get started. One of my favorite features of React Native app development is live reload. The tooling for both web and native apps supports live reloading of changes, which makes testing changes faster and eliminates the need to recompile and redeploy onto a device. Another advantage of live reload is that your application state is maintained. This is great for testing changes to complex flows without having to recreate the steps each time. Tools like CodePush make it easy to push updates to your React Native app without having to go through the iTunes Connect or Google Play submission processes.\n\nBuild native apps quicker with code sharing\n\nI have worked on several projects where there have been three client apps—web, iOS native, and Android native. More often than not, the development of each app diverges and some become more complete than others. With a React-based platform, your web and native apps can share more code used for state management, API requests, and GraphQL queries or mutations. This can help reduce inconsistencies and bugs across the board since logic doesn’t need to be duplicated across three separate applications. For example, all of your state and request logic could be built as a separate package that both the web and React Native apps consume. React Native also lets you unify your development team since the same set of technologies is used on all platforms. Since React works the same way on web as it does on native, any developer familiar with React and JavaScript can create UI components for any platform. Your React Components render to native UI elements specific to each platform, creating apps that are more performant than web-based ones.\n\nThe JavaScript ecosystem\n\nIt’s really easy to add third-party modules to your native app in React Native – just yarn add whatever. With React Native your app has access to the entire JavaScript ecosystem of modules. Unlike native 3rd party frameworks, there are no additional package managers required, no compiler flags to add, and no worry about target SDK compatibility.\n\nConsiderations\n\nAs with every technology, there are tradeoffs to the ease of building native apps with React Native. On mobile devices, JavaScript runs on a single thread, meaning calculation-intense operations might be a better fit for a native module where multiple threads can be leveraged. To write performant native modules, you will need to understand Objective-C as well as Java, and have a good understanding of how platform APIs operate. That said, React Native is constantly being updated and has under-the-hood performance improvements rolling out regularly, so this issue may lessen over time.\n\nFinal Thoughts\n\nReact is already a fantastic way to build UIs for the web and components are a great way to conceptualize your UI as a function of some data. React Native builds upon React's philosophy of \"Learn once, write anywhere,\" making it easy for React web developers to build native apps. Composable unified UI codebases, instant app updates, and better development tooling make React Native the better way to make native apps. If you have any questions about React Native or mobile development in general, find me on Twitter at @carlos_paelinck.\n\nAnd if you want to check out a great example of a React Native app for data visualization, Victory UI Explorer by Angela Nicholas is one of my favorite Formidable projects."
],
"categories": {
"primary": "mobile",
"others": [
"frontend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-latest-news-from-nearform": {
"href": "https://nearform.com/insights/latest-news-from-nearform",
"postType": "blog",
"slug": "insights-latest-news-from-nearform",
"date": "2017-07-10",
"title": "Latest news from nearForm",
"authors": [],
"content": [
"Today, we are delighted to tell you that successful Irish tech entrepreneur John McQuillan will be joining the NearForm board. John has a long and distinguished background in technology with his sale of Irish travel company OpenJaw being his most recent success. John brings a deep understanding of the needs of the enterprise, and many years experience in the development of large-scale software solutions. He is deeply committed to NearForm, which has the potential to become a prototype for technology companies globally. As a programmer John has been involved in the Node community for a number of years, and many of you may already have met him at our events. Welcome on board John!\n\nNearForm has seen continued strong growth this year and we are excited to announce that there are now more than 100 NearFormers, and growing! John McQuillan joins the board at a key moment in our history as we scale up to meet and excel at the challenges of this growth.\n\nIn other changes, one of our founders Richard Rodger has decided to spin out of NearForm in order to pursue some opportunities that have arisen due to the publication of his most recent book The Tao of Microservices.\n\nThankfully Richard is not going too far and will continue to promote NearForm. We'd like to thank Richard for his contribution to NearForm and wish him well in the future. We have no doubt that whatever he decides to do it will be good for the South East and enrich the tech eco-system here even further!\n\nIn other NearForm news we are back in Lyrath for NodeConfEU later in the year. With the calibre of speakers already confirmed, it is shaping up to be our best NodeConfEU lineup ever. This year we have the most diverse gender mixture of any conference we have been involved in. The quality of talks and the diversity of speakers are a testament to the rapidly-maturing Node eco-system.\n\nWe have been developing complex full-stack solutions for global organisations since 2011. Our early embrace of Node.js and JavaScript-Everywhere has resulted in fundamental changes to how software is built. It is a privilege for an independent company from the South-East of Ireland to be setting out the blueprint for the next generation of successful sustainable companies in the open-source world.\n\nWe have been continuing to add amazing people to the NearForm family, and we are looking forward to pushing a major update to our website and online resources in the coming weeks.\n\nOh, BTW NearForm is hiring"
],
"categories": {
"primary": "oss",
"others": [
"work",
"product",
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-react-components-living-style-guides-overview": {
"href": "https://nearform.com/insights/react-components-living-style-guides-overview",
"postType": "blog",
"slug": "insights-react-components-living-style-guides-overview",
"date": "2017-07-10",
"title": "An overview and comparison of various tools and approaches to create living style guides in context of React components development",
"authors": [],
"content": [
"An in-depth look at React Components Style Guides\n\nReact components style guides are a type of \"living pattern library\", where the components are rendered using production code and enable interactivity and live configuration.\n\nFor many years style guides have been used as a primary platform of communication between designers and developers working on any complex project. Usually it has been a separate resource that breaks down the user interface to organized building blocks and serve as a reference to locate and define design patterns for the project.\n\nThe problem with this approach is that we have to maintain the code in two different places: the guide itself and the actual production codebase. And unfortunately the style guide consistency is the first to be sacrificed when it comes to hotfixes and tight deadlines.\n\nThis approach comes in handy on every stage of the project and for anybody in a cross-functional team.\n\nIt allows designers to easily stay abreast of the existing state of UI elemens\nIt allows develeopers and QA engineers to use it as a playground to develop, mantain and test components in isolation\nIt allows product owners to have a global overview of the design an implentation\n\nThe larger the team and the more complex the codebase, the more beneficial it is to have one source of reference when working on the UI components. This is especially true when it comes to modern frontend development, such as React applications, that can easily end up with hundreds or thousands of UI components and complex hierarchies.\n\nLet's have a look at some of the most interesting projects that provide the tools to easily create and configure living style guides out of your existing React codebase.\n\nStorybook\n\nDocs & demo: https://storybook.js.org/ Github: https://github.com/storybookjs/storybook\n\nStorybook is the most popular pattern library creation tool. It has become even more popular since the release of version 3.0 which added support for Webpack 2.0 and snapshot testing.\n\nInitial setup is quite straightforward and consists of the creation of definitions (stories) for each component with the corresponding state in a similar manner to the visual test cases:\n\nimport React from 'react';\nimport { storiesOf } from '@storybook/react';\n\nimport MyComponent from './MyComponent';\n\nstoriesOf('MyComponent', module)\n .add('default', () =>
// Start the server\n server.start((err) => {\n if (err) {\n console.error(err)\n process.exit(1)\n }\n })\n})\ntextCopy to clipboard\n\nThis creates logs on all our routes of the format:\n\n{\"pid\":11518,\"hostname\":\"local-server\",\"level\":30,\"time\":1499413842512,\"msg\":\"server started\",\"created\":1499413842353,\"started\":1499413842502,\"host\":\"localhost\",\"port\":1337,\"protocol\":\"http\",\"id\":\"local-server:11518:j4tkad9t\",\"uri\":\"https://localhost:1337\",\"address\":\"127.0.0.1\",\"v\":1}\ntextCopy to clipboard\n\nThere are two additional ways of logging available to pass down extra information:\n\n// the logger is available in server.app\nserver.app.logger.warn('Pino is registered')\ntextCopy to clipboard\n// and through Hapi standard logging system\nserver.log(['subsystem'], 'standard logging way for accessing it')\ntextCopy to clipboard\n\nTo see more examples of pino output:\n\nclone our pino branch\nrun npm install\nrun npm run watch\nin another tab run curl localhost:1337\nCaveat Emptor: Pino & Hapi & TypeScript\n\nPino also provides server.logger().info(‘some info’) as a form of logging with Hapi. However, if you have a TypeScript project then you are restricted to the type definitions supplied by @types/hapi, and they do not allow this form of logging yet.\n\nLog aggregation with Event Hubs\n\nNow we have a logger working, but a feed on a local machine is not that useful. We need to be able to get this information into a dashboard.\n\nHere we need to take a step back and consider our architecture. We are deploying an app in the cloud, i.e., on a VM or container. Presumably, there are multiple instances of your app, and their other instances of, e.g., databases. There will also be other services if you are doing a micro-services deploy. You might also be collecting client-side or IoT data. Connecting all of these to a dashboard in parallel might overwhelm it. It makes more sense to connect to a log aggregation service. Event Hubs is the obvious candidate since this is part of our series on Azure.\n\nMicrosoft describes Event Hubs as:\n\n“Azure Event Hubs is a highly scalable data streaming platform and event ingestion service capable of receiving and processing millions of events per second. Event Hubs can process and store events, data, or telemetry produced by distributed software and devices. Data sent to an event hub can be transformed and stored using any real-time analytics provider or batching/storage adapters. With the ability to provide publish-subscribe capabilities with low latency and at massive scale, Event Hubs serves as the “on ramp” for Big Data.”\n\nTo collect our pino logs, we will provide a Web App and an Event Hub. Then pipe the logs from the Web App to the Event Hub:\n\nEvent Hubs can be used even if part of your architecture is on another cloud or own infrastructure. We will make use of this during development below to test the connection from our local machine to an Event Hub.\n\nThat said, it is still necessary for us to connect a dashboard by subscribing it to our Event Hub. We will do so in the next article. Note that when using Event Hub there is a delay of 10 minutes when viewing data on its dashboard. Keep this in mind as you build your own application.\n\nProvision an Event Hub\n\nIn our previous article we demonstrated how to provision a Web App. Assuming you have an Azure and a deployment user, then this simple bash script will get you started:\n\n#! /bin/bash
GROUP_NAME=nearGroup\nLOCATION=westeurope\nPLAN_NAME=nearPlan\nSKU=B1\nAPP_NAME=pino007
CREATE_GROUP=\"az group create -n $GROUP_NAME --location $LOCATION\"\necho $CREATE_GROUP\n$CREATE_GROUP
CREATE_PLAN=\"az appservice plan create -n $PLAN_NAME -g $GROUP_NAME --sku $SKU\"\necho $CREATE_PLAN\n$CREATE_PLAN
CREATE_WEBAPP=\"az webapp create -n $APP_NAME -g $GROUP_NAME -p $PLAN_NAME\"\necho $CREATE_WEBAPP\n$CREATE_WEBAPP
SET_WEBAPP_SOURCE=\"az webapp deployment source config-local-git -n $APP_NAME -g $GROUP_NAME --query url --output tsv\"\necho $SET_WEBAPP_SOURCE\n$SET_WEBAPP_SOURCE\ntextCopy to clipboard\n\nPlease remember to replace APP_NAME with a unique name of your choice. Note that while we could deploy a Web App for free, Event Hubs requires at least the Basic 1 (BI) plan.\n\nIn case you have not done it before set your Azure git repository as a target.\n\ngit remote add azure https://@pino007.scm.azurewebsites.net/ath.git\ntextCopy to clipboard\n\nProvisioning Event Hubs is possible using the Azure CLI. But, that is kind of tricky since we need to use templates . To simplify things for this example, we will use the web interface .\n\nUnder https://portal.azure.com/#create/Microsoft.EventHub create a (unique) namespace. We used pinoEventHubNS (which will be accessible via, pinoEventHubNS.servicebus.windows.net hence why it needs to be unique.):\n\nThen create eventHub called pinoEventHub :\n\nHere you can see the Event Hub pinoEventHub linked under the namespace pinoEventHubNS :\n\nThen create Shared Access Policy called sendPinoEvent :\n\nCongratulations you have provisioned a simple Web App infrastructure, including a log ingestor. More can be done to fine tune Event Hubs provisioning, but for now this suffices.\n\nSend your first event\n\nTo be able to send your first event to an Event Hub, all we need to do is create a Shared Access Signature (SAS).\n\nFirst, you will need the data from Shared Access Policy under CONNECTION STRINGPRIMARY KEY , eg,\n\nEndpoint=sb://pinoeventhubns.servicebus.windows.net/;\nSharedAccessKeyName=sendPinoEvent;\nSharedAccessKey=;\nEntityPath=pinoeventhub\ntextCopy to clipboard\n\nUsing Microsoft’s example as basis we created a simple function :\n\nfunctioncreateSignature (uri, ttl, sapk) {\n const signature = encodeURIComponent(uri) +'\\n'+ ttl\n const signatureUTF8 = utf8.encode(signature)\n const hash = crypto.createHmac('sha256', sapk)\n .update(signatureUTF8)\n .digest('base64')\n return encodeURIComponent(hash)\n}\ntextCopy to clipboard\n\nWhere the ttl is the expiry date of the SAS in Unix time. And the uri is simply https://pinoeventhubns.servicebus.windows.net/pinoeventhub .\n\nArmed with your SAS, you can now send an event:\n\ncurl -sl -w \"%{http_code}\" -H 'Authorization: SharedAccessSignature sr=http%3A%2F%2Fpinoeventhubns.servicebus.windows.net%2Fpinoeventhub&sig=&se=&skn=sendPinoEvent' -H 'Content-Type:application/atom+xml;type=entry;charset=utf-8' --data '{ \"event\": \"hello world\" }' https://pinoeventhubns.servicebus.windows.net/pinoeventhub/messages\ntextCopy to clipboard\n\nWhich should return 201 , denoting a successful POST .\n\nPiping stdout to an Event Hub\n\nWe can now successfully post to an Event Hub, now all we need is to send all the Pino logs to such a posting mechanism. Since Pino logs to stdout, all we need is something like node yourapp.js | eventhub .\n\nWhat we have is a stream of output from pino. First, we can break those into discrete events using split2 . Then, POST these lines in batches (for better performance) to an Event Hub with a writable stream in object mode using https .\n\nSomething similar to:\n\nconst writable = new Writable({\n objectMode: true,\n writev: function (lines, done) {\n const events = lines\n .map(line => {\n return `{\"UserProperties\":${line}}`\n })\n .join(',')
const req = https.request(options, function () { done() })\n req.write(`[${events}]`)\n req.end()\n },\n write: function (line, enc, done) {\n const req = https.request(options, function () { done() })\n req.write(line)\n req.end()\n },\n})
pump(split2(), writable)\ntextCopy to clipboard\n\nWe have captured this in an npm module called pino-eventhub . You have already installed this as part of your download, and we can test this on your local machine using:\n\nnode build/index.js | ./node_modules/.bin/pino-eventhub -s pinoeventhubns -e pinoeventhub -n sendPinoEvent -a -x\ntextCopy to clipboard\n\nTo see it working:\n\nrun curl localhost:1337\nopen https://portal.azure.com\nclick on pinoEventHubNS (or your equivalent Event Hub namespace)\nwait 10 minutes, and you will see a blip under incoming messages\nThe alternative: AMQP\n\nAlternatively, AMQP could be used to send the logs to an Event Hub. We stuck to https for simplicity.\n\nDeploying to Web App\n\nWe have our basic Web App setup done already. However, to keep in line with Twelve-Factor App development, we want to simplify our npm run start command to node build/index.js | pino-eventhub . We have setup our npm module to take environment variables, so this can be done using the Azure CLI:\n\naz webapp config appsettings set -n pino007 -g nearGroup --settings \\\n PINO_EVENT_HUB_NAMESPACE=pinoeventhubns \\\n PINO_EVENT_HUB=pinoeventhub \\\n PINO_SHARED_ACCESS_POLICY_NAME=sendPinoEvent \\\n PINO_SHARED_ACCESS_SIGNATURE= \\\n PINO_SAS_EXPIRY=\ntextCopy to clipboard\n\nNow all we need to do is:\n\ngit push azure pino:master\naz webapp browse -n pino007 -g nearGroup\ntextCopy to clipboard\n\nAgain, you can go to your Event Hubs dashboard on the Azure Portal to see your events arrive.\n\nAlternative: Application Insights\n\nInstead of using Pino with Event Hubs, we could’ve opted for a solution like Application Insights (AI).\n\nAccording to Microsoft:\n\n“Application Insights is an extensible Application Performance Management (APM) service for web developers on multiple platforms. Use it to monitor your live Web App. It will automatically detect performance anomalies. It includes powerful analytics tools to help you diagnose issues and to understand what users actually do with your app. It’s designed to help you continuously improve performance and usability.”\n\nAll the functionality AI ties into looks great. AI , however, tightly couples to your application through monkey-patching. At the moment AI is instrumented for Bunyan, Console, MongoDB, MongoDB-Core, Mysql and Redis. However, for this application, we are looking at the lightest/fastest possible logging solution. Hence, Event Hubs made more sense for us.\n\nConclusion\n\nBuilding on our previous article , we wanted to be able to collect the log files from our application. We showed that it is easy to provision an Event Hub on the Azure portal, using Pino as our logger. After creating credentials (using the tool we provided), your Web App can stream to an Event Hub from a single-line command.\n\nIn the next article in the series, we will select from the logs collected in an Event Hub, and represent them visually."
],
"categories": {
"primary": "cloud",
"others": [
"backend",
"data"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-get-ready-a-new-v8-is-coming-node-js-performance-is-changing": {
"href": "https://nearform.com/insights/get-ready-a-new-v8-is-coming-node-js-performance-is-changing",
"postType": "blog",
"slug": "insights-get-ready-a-new-v8-is-coming-node-js-performance-is-changing",
"date": "2017-07-27",
"title": "Get ready: A new V8 is coming, Node.js performance is changing.",
"authors": [],
"content": [
"How the performance characteristics of v8's Turbofan will affect the way we optimize\n\nReviewers: Franziska Hinkelmann and Benedikt Meurer from the V8 team.\n\n**Update: Node.js 8.3.0 has shipped V8 6.0 with Turbofan .\n\nSince its inception Node.js has depended on the V8 JavaScript engine to provide code execution in the language we all know and love. The V8 JavaScript engine is a JavaScript VM that was written by Google for the Chrome browser. From the beginning, a primary goal of V8 was to make JavaScript fast, or at least - faster than the competition. For a highly dynamic, loosely typed language this is no easy feat. This piece is about the evolution of V8 and JS engines' performance.\n\nA central piece of the V8 engine that allows it to execute JavaScript at high speed is the JIT (Just In Time) compiler. This is a dynamic compiler that can optimize code during runtime. When V8 was first built the JIT Compiler was dubbed FullCodegen. Then, the V8 team implemented Crankshaft, which included many performance optimizations that FullCodegen did not implement. Edit: FullCodegen was the first optimizing compiler of V8, thanks to Yang Guo for reporting. As an outside observer and user of JavaScript since the 90's, it has seemed that fast and slow paths in JavaScript (whatever the engine may be) were often counter-intuitive, the reasons for apparently slow JavaScript code were often difficult to fathom.\n\nIn recent years Matteo Collina and I have focused on finding out how to write performant Node.js code. Of course, this means knowing which approaches are fast and which approaches are slow when our code is executed by the V8 JavaScript engine.\n\nNow it's time for us to challenge all our assumptions about performance because the V8 team has written a new JIT Compiler: Turbofan.\n\nStarting with the more commonly known \"V8 Killers\" (pieces of code which cause optimization bail-out - a term that no longer makes sense in a Turbofan context) and moving towards the more obscure discoveries Matteo and I have made around Crankshaft performance, we're going to walk through a series of microbenchmark results and observations over progressing versions of V8.\n\nOf course, before optimizing for V8 logic paths, we should first focus on API design, algorithms and data structures. These microbenchmarks are meant as indicators of how JavaScript execution in Node is changing. We can use these indicators to influence our general code style and the ways we improve performance after we've applied the usual optimizations.\n\nWe'll be looking at the performance of these microbenchmarks on V8 versions 5.1, 5.8, 5.9, 6.0 and 6.1.\n\nTo put each of these versions into context: V8 5.1 is the engine used by Node 6 and uses the Crankshaft JIT Compiler, V8 5.8 is used in Node 8.0 to 8.2 and uses a mixture of Crankshaft and Turbofan.\n\nCurrently, 6.0 engine is due to be in Node 8.3 (or possibly Node 8.4) and V8 version 6.1 is the latest version of V8 (at the time of writing) which is integrated with Node on the experimental node-v8 repo at https://github.com/nodejs/node-v8 . In other words, V8 version 6.1 will eventually be in some future version of Node, likely Node.js 9.\n\nLet's take a look at our microbenchmarks and on the other side, we'll talk about what this means for the future. All the microbenchmarks are executed using benchmark.js , and the values plotted are operation per second, so higher is better in every diagram.\n\nThe try/catch problem\n\nOne of the more well-known deoptimization patterns is use of try/catch blocks.\n\nIn this microbenchmark we compare four cases:\n\na function with a try/catch in it (sum with try catch)\na function without any try/catch (sum without try catch)\ncalling a function within a try block (sum wrapped)\nsimply calling a function, no try/catch involved (sum function)\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/try-catch.js\n\nWe can see that the existing wisdom around try/catch causing performance problems is true in Node 6 (V8 5.1) but has significantly less performance impact in Node 8.0-8.2 (V8 5.8).\n\nAlso of note, calling a function from within a try block is much slower than calling it from outside a try block - this holds true in both Node 6 (V8 5.1) and Node 8.0-8.2 (V8 5.8).\n\nHowever, for Node 8.3+ the performance hit of calling a function inside a try block is negligible.\n\nNevertheless, don't rest too easy. While working on some performance workshop material, Matteo and I found a performance bug , where a rather specific combination of circumstances can lead to an infinite deoptimization/re-optimisation cycle in Turbofan (this would count as a \"killer\" - a pattern that destroys performance).\n\nRemoving properties from objects\n\nFor years now, delete has been off-limits to anyone wishing to write high-performance JavaScript (well at least where we're trying to write optimal code for a hot path).\n\nThe problem with delete boils down to the way V8 handles the dynamic nature of JavaScript objects and the (also potentially dynamic) prototype chains that make property lookups even more complex at an implementation level.\n\nThe V8 engine's technique for making property objects high speed is to create a class in the C++ layer based on the \"shape\" of the object. The shape is essentially what keys and values a property have (including the prototype chain keys and values). These are known as \"hidden classes\". However, this is an optimization that occurs on objects at runtime, if there's uncertainty about the shape of the object V8 has another mode for property retrieval: hash table lookup. The hash table lookup is significantly slower. Historically, when we delete a key from the object subsequent property access will be a hash table lookup. This is why we avoid delete and instead set properties to undefined which leads to the same result as far as values are concerned but may be problematic when checking for property existence; however it's often good enough for pre-serialization redaction because JSON.stringify does not include undefined values in its output ( undefined isn't a valid value in the JSON specification).\n\nNow, let's see if the newer Turbofan implementation addresses the delete problem.\n\nIn this microbenchmark we compare three cases:\n\nserializing an object after an object's property has been set to undefined\nserializing an object after delete has been used to remove an object's property which was not the last property added.\nserializing an object after delete has been used to remove the object's most recently added property.\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/property-removal.js\n\nIn V8 6.0 and 6.1 (not yet used in any Node releases), deleting the last property added to an object hits a fast path in V8, and thus it is faster, even, than setting to undefined . This is excellent news, as it shows that the V8 team is working towards improving the performance of delete . However, the delete operator will still result in a significant performance hit on property access if a property other than the most recently added property is deleted from the object. So overall we have to continue to recommend against using delete .\n\nEdit: in a previous version of the post we concluded that delete could and should be used in future Node.js releases. However Jakob Kummerow let us know that our benchmarks only triggered the last property accessed case. Thanks Jakob Kummerow!\n\nLeaking and arrayifying arguments\n\nA common problem with the implicit arguments object available to normal JavaScript functions (as opposed to fat arrow functions which do not have arguments objects) is that its array-like but not an array.\n\nIn order to use array methods or most array behaviour, the indexing properties of the arguments object have been copied to an array. In the past JavaScripters have had a propensity towards equating less code with faster code. Whilst this rule of thumb yields a payload-size benefit for browser-side code, the same rule can cause pain on the server side where code size is far less important than execution speed. So a seductively terse way to convert the arguments object to an array became quite popular: Array.prototype.slice.call(arguments) . This calls the Array slice method passing the arguments object as the this context of that method, the slice method sees an object that quacks like an array and acts accordingly. That is, it takes a slice of the entire arguments array-like object as an array.\n\nHowever, when a function's implicit arguments object is exposed from the context of the functions (for instance, when it's returned from the function or passed into another function as in the case of Array.prototype.slice.call(arguments) ) this typically causes performance degradation. Now it's time to challenge that assumption.\n\nThis next microbenchmark measures two interrelated topics across our four V8 versions: the cost of leaking arguments and the cost of copying arguments into an array (which is subsequently exposed from the function scope in place of the arguments object).\n\nHere are our cases in detail:\n\nExposing the arguments object to another function - no array conversion (leaky arguments)\nMaking a copy of the arguments object using the Array.prototype.slice tricks (Array prototype.slice arguments)\nUsing a for loop and copying each property across (for-loop copy arguments)\nUsing the EcmaScript 2015 spread operator to assign an array of inputs to a reference (spread operator)\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/arguments.js\n\nLet's take a look at the same data in line graph form to emphasise the change in performance characteristics:\n\nHere's the takeaway: if we want to write performant code around processing function inputs as an array (which in my experience seems fairly common), then in Node 8.3 and up we should use the spread operator. In Node 8.2 and down we should use a for loop to copy the keys from arguments into a new (pre-allocated) array (see the benchmark code for details).\n\nFurther in Node 8.3+ we won't be punished for exposing the arguments object to other functions, so there may be further performance benefits in cases where we don't need a full array and can make do with an array-like structure.\n\nPartial application (currying) and binding\n\nPartial application (or currying) refers to the way we can capture state in nested closure scopes.\n\nFor instance:\n\nfunction add (a, b) {\n return a + b\n}\nconst add10 = function (n) {\n return add(10, n)\n}\nconsole.log(add10(20))\n\n\nHere the a parameter of add is partially applied as the value 10 in the add10 function.\n\nA terser form of partial application was made available with the bind method since EcmaScript 5:\n\nfunction add (a, b) {\n return a + b\n}\nconst add10 = add.bind(null, 10)\nconsole.log(add10(20))\n\n\nHowever, we would typically not use bind because is demonstrably slower than using a closure.\n\nThis benchmark measures the difference between bind and closure over our target V8 versions, with direct function calls as the control.\n\nHere are our four cases:\n\na function that calls another function with the first argument partially applied (curry)\na fat arrow function that calls another function with the first argument partially applied (fat arrow curry)\na function that is created via bind that partially applies the first argument of another function (bind)\na direct call to a function without any partial application (direct call)\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/currying.js\n\nThe line graph visualization of this benchmark's results clearly illustrates how the approaches converge in later versions of V8. Interestingly, partial application using fat arrow functions is significantly faster than using normal functions (at least in our microbenchmark case). In fact it almost traces the performance profile of making a direct call. In V8 5.1 (Node 6) and 5.8 (Node 8.0-8.2) bind is very slow by comparison, and it seems clear that using fat arrows for partial application is the fastest option. However bind speeds up by an order of magnitude from V8 version 5.9 (Node 8.3+) becoming the fastest approach (by an almost negligible amount) in version 6.1 (future Node).\n\nThe fastest approach to currying over all versions is using fat arrow functions. Code using fat arrow functions in later versions will be as close to using bind as makes no odds, and currently it's faster than using normal functions. However, as a caveat, we probably need to investigate more types of partial application with differently sized data structures to get a fuller picture.\n\nFunction character count\n\nThe size of a function, including its signature, the white space and even comments can affect whether the function can be inlined by V8 or not. Yes: adding a comment to your function may cause a performance hit somewhere in the range of a 10% speed reduction. Will this change with Turbofan? Let's find out.\n\nIn this benchmark we look at three scenarios:\n\na call to a small function (sum small function)\nthe operations of a small function performed inline, padded out with comments (long altogether)\na call to a big function that has been padded with comments (sum long function)\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/function-size.js\n\nIn V8 5.1 (Node 6) sum small function and long all together are equal. This perfectly illustrates how inlining works. When we're calling the small function, it's as if V8 writes the contents of the small function into the place it's called from. So when we actually write the contents of a function (even with the extra comment padding), we've manually inlined those operations and the performance is equivalent. Again we can see in V8 5.1 (Node 6) that calling a function that is padded with comments which take it past a certain size, leads to much slower execution.\n\nIn Node 8.0-8.2 (V8 5.8) the situation is pretty much the same, except the cost of calling the small function has noticeably increased; this may be due to the smushing together of Crankshaft and Turbofan elements whereas one function may be in Crankshaft the other may be in Turbofan causing disjoints in inlining abilities (i.e. there has to be a jump between clusters of serially inlined functions).\n\nIn 5.9 and upwards (Node 8.3+), any size added by irrelevant characters such as whitespace or comments has no bearing on the performance of the function. This is because Turbofan uses the functions AST ( Abstract Syntax Tree node count to determine function size, rather than using character count as in Crankshaft. Instead of checking byte count of a function, it considers the actual instructions of the function, so that from V8 5.9 (Node 8.3+) whitespace, variable name character count, function signatures and comments no longer factors in whether a function will inline. Notably, again, we see that the overall performance of functions decreases.\n\nThe takeaway here should still be to keep functions small. At the moment we still have to avoid over-commenting (and even whitespace) inside functions. Also if you want the absolute fastest speed, manually inlining (removing the call) is consistently the fastest approach. Of course, this has to be balanced against the fact that after a certain size (of actual executable code) a function won't be inlined, so copy-pasting code from other functions into your function could cause performance problem. In other words, manual inlining is a potential footgun; it's better to leave inlining up to the compiler in most cases.\n\n32-bit integers vs integers stored in doubles\n\nIt's rather well known that JavaScript only has one number type: Number .\n\nHowever, V8 is implemented in C++ so a choice has to be made on the underlying type for a numeric JavaScript value.\n\nIn the case of integers (that is, when we specify a number in JS without a decimal), V8 assumes that all numbers fit into 32 bits - until they don't. This seems like a fair choice since in many cases a number is within the -2147483648 - 2147483647 range. If a JavaScript (whole) number exceeds 2147483647 the JIT Compiler has to dynamically alter the underlying type for the number to a double (a double-precision floating point number) - this may also have a potential knock-on effects with regards to other optimizations.\n\nThis benchmark looks at three cases:\n\na function handling only numbers in the 32-bit range (sum small)\na function handling a combination of 32-bit and double numbers (from small to big)\na function handling only double numbers (all big)\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/numbers.js \n\nWe can see from the graph that whether it's Node 6 (V8 5.1) or Node 8 (V8 5.8) or even some future version of Node this observation holds true. Operations using Numbers (integers) greater than 2147483647 will cause functions to run between a half and two-thirds of the speed.\n\nSo, if you have long numeric IDs - put them in strings.\n\nIt's also quite noticeable that operations with numbers in the 32-bit range have a speed increase between Node 6 (V8 5.1) and Node 8.1 and 8.2 (V8 5.8) but slow significantly in Node 8.3+ (V8 5.9+). However, operations over double numbers become faster in Node 8.3+ (V8 5.9+). It's likely that this genuinely is a slow-down in (32-bit) number handling rather than being related to the speed of function calls or for loops (which are used in the benchmark code). Edit: updated thanks to Jakob Kummerow and Yang Guo and the V8 team, for both accuracy and precision of the results.\n\nIterating over objects\n\nGrabbing all of an object's values and doing something with them is a common task and there are many ways to approach this. Let's find out which is fastest across our V8 (and Node) versions.\n\nThis benchmark measures four cases for all V8 versions benched:\n\nusing a for-in loop with a hasOwnProperty check to get an object's values (for in)\nusing Object.keys and iterating over the keys using the Array reduce method, accessing the object values inside the iterator function supplied to reduce (Object.keys functional)\nusing Object.keys and iterating over the keys using the Array reduce method, accessing the object values inside the iterator function supplied to reduce where the iterator function is a fat arrow function (Object.keys functional with arrow)\nlooping over the array returned from Object.keys with a for loop, accessing the object values within the loop (Object.keys with for loop)\n\nWe also benchmark an additional three cases for V8 5.8, 5.9, 6.0 and 6.1\n\nusing Object.values and iterating over the values using the Array reduce method, (Object.values functional)\nusing Object.values and iterating over the values using the Array reduce method, where the iterator function supplied to reduce is a fat arrow function (Object.values functional with arrow)\nlooping over the array returned from Object.values with a for loop (Object.values with for loop)\n\nWe don't bench these cases in V8 5.1 (Node 6) because it doesn't support the native EcmaScript 2017 Object.values method. Code: https://github.com/davidmarkclements/v8-perf/blob/master/bench/object-iteration.js\n\nIn Node 6 (V8 5.1) and Node 8.0-8.2 (V8 5.8) using for - in is by far the fastest way to loop over an object's keys, then access the values of the object. At roughly 40 million operations per second, it's 5 times faster than the next closest approach which is Object.keys at around 8 million op/s.\n\nIn V8 6.0 (Node 8.3) something happens to for - in and it cuts down to one quarter the speed of former versions, but is still faster than any other approach.\n\nIn V8 6.1 (the future of Node), the speed of Object.keys leaps forward and becomes faster than using for - in - but no where near the speed of for - in in V8 5.1 and 5.8 (Node 6, Node 8.0-8.2).\n\nA driving principle behind Turbofan seems to be to optimize for intuitive coding behavior. That is, optimize for the case that is most ergonomic for the developer.\n\nUsing Object.values to get values directly is slower than using Object.keys and accessing the values in the object. On top of that, procedural loops remain faster than functional programming. So there may be some more work to do when it comes to iterating over objects.\n\nAlso, for those who've used for - in for its performance benefits it's going to be a painful moment when we lose a large chunk of speed with no alternative approach available. Note: the for-in loops performance has been fixed in V8, see https://benediktmeurer.de/2017/09/07/restoring-for-in-peak-performance/ for more details. This change will likely be integrated by Node 9.\n\nObject allocation\n\nWe allocate objects all the time so this is a great area to measure.\n\nWe're going to look at three cases:\n\nallocating objects using object literals (literal)\nallocating objects from an EcmaScript 2015 Class (class)\nallocating objects from a constructor function (constructor)\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/object-creation.js\n\nObject allocation has the same cost in all the test V8 versions, apart from classes in Node 8.2 (V8 5.8), which are slower than the rest. This is due to the mixed Crankshaft/Turbofan nature of V8 5.8, and it will be resolved in Node 8.3 with V8 6.0. Edit: Jakob Kummerow noted in https://disq.us/p/1kvomfk that Turbofan can optimize away the object allocation in this specific microbenchmark, leading to incorrect results so this article has been edited accordingly.\n\nObject allocation elimination\n\nWhile collating our results for this article, we came across a really neat optimization that Turbofan brings to a certain category of object allocation. Originally we mistook this for all object allocation, but thanks to input from the V8 team we've come to understand the circumstances which this optimization relates to.\n\nIn the previous Object allocation benchmarks, we assign a variable, set it to null and then reassign the variable a bunch of times to avoid triggering the special optimization case which we're going to look at now.\n\nIn this benchmark we look at three cases:\n\nallocating objects using object literals (literal)\nallocating objects from an EcmaScript 2015 Class (class)\nallocating objects from a constructor function (constructor)\n\nHowever, the difference is, the reference to the object is not overwritten with additional object allocations, and the object is passed to another function which performs a task on the object.\n\nLet's take a look at the results!\n\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/object-creation-inlining.js \n\nNotice how V8 6.0 (Node 8.3) and 6.1 (Node 9) yield a massive jump in speed for this case, over 500 million operations per second, mainly because nothing is really happening once Turbofan applies the optimization. In this particular scenario, Turbofan is able to optimize away the object allocation because it's able to determine that subsequent logic can be executed without requiring the existence of the actual object.\n\nThe benchmark code still doesn't fully demonstrate what it takes to trigger this optimization (because it's more about what's not there than what is) and the conditions it takes for this optimization to be applied are complex.\n\nHowever, one condition which we know will definitely not allow for the object to be optimized away by Turbofan is as follows:\n\nThe object must not outlive the function it was created in. That means, there should be no reference to that object after every function from that point in the stack has completed. The object can be passed to other functions, but if we add that object to a this context, or assign it to an outer scoped variable, or add it to another object that lives on after the stack has finished the optimization cannot be applied.\n\nThe ramifications for this could be pretty cool, however, it's difficult to predict all the conditions where such an optimization occurs. Nevertheless, it may yield useful speedups when the complex set of conditions are met. Edit: Thanks to Jakob Kummerow and the rest of the V8 team for helping us discover the underlying reasons for this particular behaviour. As part of this research, we discovered a performance regression in the new GC of V8, Orinoco. if you are interested checkout https://v8project.blogspot.it/2016/04/jank-busters-part-two-orinoco.html and https://bugs.chromium.org/p/v8/issues/detail?id=6663\n\nPolymorphic vs monomorphic code\n\nWhen we always pass the same type of argument into a function (say, we always pass a string), we are using that function in a monomorphic way.\n\nSome functions are written to be polymorphic. We can think of a polymorphic function as a function that accepts different types in the same argument position . For instance, a function may accept either a string or an object as its first argument. However in this context when we say \"type\", we don't just mean string versus number versus object, we mean object shape (although JavaScript types actually count as different object shapes as well)\n\nThe shape of an object is defined by its properties and values. For instance, in the following snippet, obj1 and obj2 are the same shape but obj3 and obj4 are different shapes to the rest:\n\nconst obj1 = { a: 1 }\nconst obj2 = { a: 5 }\nconst obj3 = { a: 1, b: 2 }\nconst obj4 = { b: 2 }\n\n\nProcessing objects with different shapes through the same code can make for nice interfaces in some circumstances but tends to have a negative impact on performance.\n\nLet's see how monomorphic and polymorphic cases do in our benchmarks.\n\nHere we investigate two cases:\n\na function where we process objects with different properties (polymorphic)\na function where we process objects with the same properties (monomorphic)\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/polymorphic.js \n\nThe data visualized in our graph shows conclusively that monomorphic functions outperform polymorphic functions across all V8 versions tested. However, the performance of the polymorphic function improves from V8 5.9+ (which means it improves from Node 8.3 which uses V8 6.0).\n\nPolymorphic functions are very common through the Node.js codebase, and they provide a great deal of flexibility through the APIs. Thanks to this improvement around polymorphic interaction, we may see a degree of improved performance in more complex Node.js applications.\n\nIf we're writing code that needs to be optimal, that is a function that will be called many times over, then we should commit to call functions with the parameters with the same \"shape\". On the other hand, if a function is only called once or twice, say an instantiating/setup function, then a polymorphic API is acceptable. Edit: Thanks Jakob Kummerow for providing a more reliable version of this benchmark.\n\nThe debugger keyword\n\nFinally, let's talk about the debugger keyword.\n\nBe sure to strip debugger statements from your code. Stray debugger statements destroy performance.\n\nWe're looking at two cases:\n\na function that contains the debugger keyword (with debugger)\na function that does not contain the debugger keyword (without debugger)\nCode: https://github.com/davidmarkclements/v8-perf/blob/master/bench/debugger.js \n\nYep. Just the presence of the debugger keyword is terrible for performance across all V8 versions tested.\n\nThe without debugger line noticeably drops over successive V8 versions, we'll talk about this in the Summary .\n\nA real-world benchmark: Logger comparison\n\nIn addition to our microbenchmarks, we can take a look at the holistic effects of our V8 versions by using benchmarks of most popular loggers for Node.js that Matteo and I put together while we were creating Pino .\n\nThe following bar chart represents the time taken to log 10 thousand lines (lower is better) of the most popular loggers in Node.js 6.11 (Crankshaft):\n\nWhile the following is the same benchmarks using V8 6.1 (Turbofan):\n\nWhile all of the logger benchmarks improve in speed (by roughly 2x), the Winston logger derives the most benefit from the new Turbofan JIT compiler. This seems to demonstrate the speed convergence we see among various approaches in our microbenchmarks: the slower approaches in Crankshaft are significantly faster in Turbofan while the fast approaches in Crankshaft tend to get a little slower Turbofan. Winston, being the slowest, is likely using the approaches which are slower in Crankshaft but much faster in Turbofan whereas Pino is optimized to use the fastest Crankshaft approaches. While a speed increase is observed in Pino, it's to a much lesser degree.\n\nSummary\n\nSome of the benchmarks show that while slow cases in V8 5.1, V8 5.8 and 5.9 become faster with the advent of full Turbofan enablement in V8 6.0 and V8 6.1, the fast cases also slow down, often matching the increased speed of the slow cases.\n\nMuch of this is due to the cost of making function calls in Turbofan (V8 6.0 and up). The idea behind Turbofan was to optimize for common cases and eliminate commonly used \"V8 Killers\". This has resulted in a net performance benefit for (Chrome) browser and server (Node) applications. The trade-off appears to be (at least initially) a speed decrease for the most performant cases. Our logger benchmark comparison indicates that the general net effect of Turbofan characteristics is comprehensive performance improvements even across significantly contrasting code bases (e.g. Winston vs Pino).\n\nIf you've had an eye on JavaScript performance for a while, and adapted coding behaviours to the quirks of the underlying engine it's nearly time to unlearn some techniques. If you've focused on best practices, writing generally good JavaScript then well done, thanks to the V8 team's tireless efforts, a performance reward is coming.\n\nThis article was co-written by David Mark Clements and Matteo Collina , and it was reviewed by Franziska Hinkelmann and Benedikt Meurer from the V8 team.\n\nAll the source code and another copy of this article could be found at https://github.com/davidmarkclements/v8-perf The raw data for this article can be found at: https://docs.google.com/spreadsheets/d/1mDt4jDpN_Am7uckBbnxltjROI9hSu6crf9tOa2YnSog/edit?usp=sharing Most of the microbenchmarks were taken on a MacBook Pro 2016, 3.3 GHz Intel Core i7 with 16 GB 2133 MHz LPDDR3, others (numbers, property removal, polymorphic, object creation) were taken on a MacBook Pro 2014, between the different Node.js versions were taken on the same machine. We took great care in assuring that no other programs were interfering. Will O"
],
"categories": {
"primary": "perf",
"others": [
"backend",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-azure-real-time-insights-and-visualisations": {
"href": "https://nearform.com/insights/azure-real-time-insights-and-visualisations",
"postType": "blog",
"slug": "insights-azure-real-time-insights-and-visualisations",
"date": "2017-07-31",
"title": "Azure real-time insights and visualisations",
"authors": [],
"content": [
"This is the 3rd article in our Azure series. We have been building an architecture that will work for a simple node application but will also scale to micro-services and/or IoT, and then into a fully fledged solution platform running on Azure. In the previous two articles, we focused on rapid application development on Microsoft Azure , and how to collect fast Pino logs in an Azure Event Hub . To complete the first part of our series, we will investigate how we can process and gain insights from the data we are generating.\n\nReal-time insights: Stream Analytics\n\nThis post assumes you have a Web App that pipes logs into an Event Hub. (If not instructions are included in the previous article .) Now we want to be able to gain real-time insights from that data.\n\nAzures Stream Analytics processes data in real-time. In Microsoft's words:\n\nUsing Stream Analytics, you can examine high volumes of data flowing from devices or processes, extract information from the data stream, and look for patterns, trends, and relationships. Based on what’s in the data, you can then perform application tasks. For example, you might raise alerts, kick off automation workflows, feed information to a reporting tool such as Power BI, or store data for later investigation.\n\nWith Stream Analytics we want to be able to extract information from our log stream, eg,\n\nQuery response times\n90th, 95th and 99th percentile response times\nWarnings on a 404 requests\nAlerts on Denial-of-Service attacks\n\nFor this article, we will simply look at query response times.\n\nReal-time dashboard: Power BI\n\nThe last piece of our minimal architecture would be to visualise the information extracted from our logs. Azures Power BI Embedded allows the creation of a real-time dashboard by embedding components in your application. However, for the simplicity of the article, we will be using Power BI where a dashboard is ready for use.\n\nCaveat Emptor: Even though the Power BI dashboard uses real-time data, it has to be manually refreshed to show it.\n\nInitial setup\n\nTo start, please provide a basic application with logging, as per Collect fast Pino logs in an Azure Event Hub .\n\nProvision Power BI\n\nFirst, we want to set up our Power BI account, because Stream Analytics would need it as a target output.\n\nPower Bi UI is not completely integrated into Azure UI yet. Do not set up your Power BI account from within Azure. Instead, open https://powerbi.microsoft.com/en-us/get-started/ Click Try Free (in the middle-left of the page) under Power BI Cloud collaboration and sharing, then register with your existing Microsoft login.\n\nNow we want to create a workspace from the menu on the left:\n\nNow we can proceed to create a workspace named pinoPowerBI :\n\nRegister your app on Azure Active Directory\n\nSome of the online advice says that it is a requirement to register your app on Azure Active Directory (AAD). However, this only relevant if you want to embed Power BI into your app. We will be viewing our graphs directly on https://powerbi.microsoft.com/ , so we can ignore this.\n\nProvision Stream Analytics\nCreate an input and an output\n\nMicrosoft uses the words input and source, and output and sink interchangeably. In this article, we will stick to input and output.\n\nGo back to https://portal.azure.com/ . Search for Stream Analytics jobs :\n\nAdd a New Stream Analytics Job , named responseTimesAll :\n\nThis should result in a success screen like this:\n\nBefore we create the input. Go create an Event Hub policy under pinoEventHub called listenPinoEvents with listen privileges:\n\nNow, click on Inputs under Job Topology :\n\nAdd a new Input and connect it to listenPinoEvents with alias inputFromPinoEventHub :\n\nClick on Outputs under Job Topology , and choose to add a new Output :\n\nName the output alias dashboard , and choose Power BI under Sink :\n\nNow click authorize, and sign in with the Power BI user you enabled earlier:\n\nNow we can create the Output named dashboard , in Group Workspace pinoPowerBI , with Dataset Name pino and Table Name `responseTimes :`\n\nNow we are all set to connect input and output.",
"Real-time query language\n\nThe real power of Stream Analytics is the Stream Analytics Query Language . We will only be doing a simple example, but with this language queries can done as data is received from the input.\n\nTo create our query we will select Query under Job Topology :\n\nWe want to create a time-series graph showing response times of queries against time. So, we need to SELECT responseTime and time . We take those values from the input inputFromPinoEventHub and send them into the output dashboard , using FROM inputFromPinoEventHub and INTO dashboard .\n\nHowever, Pino formats time in unix time, and Power BI needs it in ISO 8601 format. In Stream Analytics every event has an associated time stamp. We can override that time stamp and convert our time value into ISO 8601 format, using TIMESTAMP BY DATEADD(millisecond, time, 1970-01-01T00:00:00Z) . Lastly we need to change our SELECT to pick the event time stamp we have overwritten System.Timestamp AS time .\n\nLastly, we want to filter for log results that actually have a responseTime defined, which WHERE (responseTime > 0) will do for us.\n\nResulting in this query:\n\nBefore we save this as done, we can test the query by uploading test data:\n\nIf you run the test, you should get some positive results:\n\nWe can now save our query, and then start the Stream Analytics job (this might take a couple of moments). Once, the running sign is up, refresh your Web App url. Then you should see an event:\n\nThat single event we witnessed before created a pino dataset in Power BI. Go ahead and create a couple more data points by refreshing your web page.\n\nVisualize the data\n\nNow, we leave https://portal.azure.com/ and return to https://powerbi.microsoft.com/ . Expand tour pinoPowerBi workspace. If it is not visible, select it under Workspaces :\n\nNow, under DATASETS , click on the pino , and select a line graph.\n\nTo populate our graph, we drag time under Axis . And, we drag responsetimes under Values :\n\nTo view our graph simply maximise it:\n\nTo keep this graph save it to a report.\n\nCaveat Emptor: Time Zones\n\nPay attention to time zone being displayed. I've noticed 3 time-zones being used in graphs across Azure and Power BI: The time-zone I'm in, the time-zone my company is in, and GMT +0.\n\nAzure Resource Manager templates\n\nWe provisioned Event Hubs and Stream Analytics using the Azure Portal point-and-click interface because we wanted a first-time or prototyping user to be up-and-running with Azure as quickly as possible. As your product or usage matures though, you would want to describe your infrastructure in code. This is what Azure Resource Manager templates are for. These templates can be harnessed from the Azure CLI , to deploy, update, or delete your resources in a single action.\n\nConclusion\n\nPower BI and Azure might not be fully integrated on a UI level yet, but the back end integration works great. Stream Analytics real-time insights combined with Power BIs embedded visualization functionality allows opportunities to present business value metrics to management in custom apps.\n\nHere ends our series on setting up a basic Node.js app using Azure. Next, we will look at using Docker to provision a Node.js app on Azure Web Apps."
],
"categories": {
"primary": "cloud",
"others": [
"data"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-time-travel-with-postgresql-on-amazon-rds": {
"href": "https://nearform.com/insights/time-travel-with-postgresql-on-amazon-rds",
"postType": "blog",
"slug": "insights-time-travel-with-postgresql-on-amazon-rds",
"date": "2017-08-11",
"title": "nearForm open sources a temporal tables implementation for PostgreSQL",
"authors": [],
"content": [
"View historical data with this PostgreSQL extension\n\nAt NearForm we prefer PostgreSQL for building relational databases. In a recent project, we built a feature to allow users to freely operate on data, whilst being able to see previous versions and, in case of necessity, rollback to a certain point in time.\n\nThere are multiple applications for this, e.g. think of a text editor that creates a snapshot every time you hit save. If you make a mistake or are unsatisfied with recent changes, you can navigate through all the previous versions and restore to a previous point in time.\n\nOr just think of Git.\n\nPostgreSQL to the rescue\n\nWouldn't it be great to solve this problem directly at the database level? Maybe being able to query data like it was yesterday or last year?\n\nWell, turns out that SQL:2011 , a relatively recent iteration of the sql standard, includes improved support for temporal databases, allowing the user to store historical data and easily query through it.\n\nUnfortunately, no databases fully support the new standard yet.\n\nSince we know and love PostgreSQL and it's our go-to choice for relational databases, we searched for an alternative supported by it and found the temporal_tables extension. It works through a trigger that on insert, update or delete stores a copy of the data in a separate table.\n\nIt's a very nice extension and works well but, unfortunately, it's not available on Amazon RDS or other hosted solutions.\n\nSo we decided to create a drop-in replacement written in PL/pgSQL to be fully portable to all PostgresSQL installations, and today we are proud to open source and share it with you: check it out on our github .\n\nFor a tutorial on using temporal_tables check the project readme or this article by Clark Dave If you need help building relational databases for your business, get in touch with us"
],
"categories": {
"primary": "data",
"others": [
"oss"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-introducing-electron-webpack-dashboard": {
"href": "https://nearform.com/digital-community/introducing-electron-webpack-dashboard",
"postType": "blog",
"slug": "digital-community-introducing-electron-webpack-dashboard",
"date": "2017-08-16",
"title": "Introducing Electron Webpack Dashboard",
"authors": [
"KEN WHEELER"
],
"content": [
"Everything is magic, except to the magician.\n\nOne year ago yesterday, I released webpack-dashboard. It ended up getting a lot more popular than I thought it would! As it turns out, everyone wants to pretend like they are working at NASA.\n\nShortly after releasing it, and selling it under the guise of it feeling like working at NASA, I came to find out that NASA actually has their own dashboard lib, and it's a lot better than mine. Check it out here: https://github.com/nasa/openmct.\n\nOne interesting thing about their dashboard though, is that it is web based. So that, coupled with a TON of frustration trying to make the terminal dashboard work on the incredible variety of terminal environments, led me to create an Electron, web-tech based version. Think less 50's NASA, and more Westworld.\n\nI tweeted some teasers and people seemed pretty excited, so I stayed up until 3:30am the last few nights and got a beta version ready to put out. Today, that beta is yours:\n\nFeatures\n\nLets start with the main display:\n\nLog - The first panel is the webpack log panel where you can see build ouput.\nOperations - The operations panel shows circular build progress, error counts, build status, total module size and total asset size.\nModules - The modules panel shows the modules used in the bundle and their sizes.\nAssets - The assets panel shows the assets produced by the build, with the JavaScript bundles displaying minified and gzipped sizes, along with warnings for bundle size limits.\nProblems - The problems panel shows you problems such as version skews or duplicates.\n\nNext up, we have an eye icon in the top right. Clicking that brings you to the Visualization panel.\n\nThis allows you to take a graphical look at your bundle composition, and see where bloat may be coming from.\n\nCheck out the dashboard in action:\n\nFuture\n\nOne obvious goal for the future is to get this thing out of beta. That said, the switch to web based technology opens the door to faster iteration and really impressive features. One goal is to provide a wider set of best practices and common errors to validate against.\n\nIn the next few versions leading up to 1.0, expect: better and more visualizations; Lodash specific warnings; warnings that go beyond size, version skews, and duplicates; and better support for different configurations.\n\nGive it a try now at:\n\nhttps://github.com/FormidableLabs/electron-webpack-dashboard\n\nThanks\n\nBig shouts out to Jon Cooper for the design and Tyler Thompson for the additional plugin logic.\n\nAlso, a wholehearted shout out to Ramin Djawadi, composer of the Westworld soundtrack, which has powered this release."
],
"categories": {
"primary": "frontend",
"others": [
"backend",
"oss"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-how-design-sprints-enable-organisations-to-build-better-products": {
"href": "https://nearform.com/insights/how-design-sprints-enable-organisations-to-build-better-products",
"postType": "blog",
"slug": "insights-how-design-sprints-enable-organisations-to-build-better-products",
"date": "2017-08-16",
"title": "nearForm facilitates 5 days Concept Sprints to define the Minimum Viable Product.",
"authors": [],
"content": [
"The 5 day design sprint\n\nThe challenges of creating a new product are numerous and the path to success is rarely straightforward. Choosing the right process and partner to support you along this journey is a critical strategic decision. In our experience design sprints help our customers identify and implement outstanding products. At NearForm we work hand-in-hand with companies, helping them overcome some of their biggest challenges. For example; limited budget, resources, or customer knowledge. Our approach is simple. We focus the product design process on the user feedback loop. As Winston Churchill once said: Facts are better than dreams.\n\nFocus, focus, focus\n\nSuccess in new developments comes from concentrating on the core value of your product. But formalising this can be hard.\n\nThe five-day Design Sprint is one of the best ways to achieve this. At the beginning of a project, all stakeholders are gathered around the table. Business, design, technical stakeholders and, later in the week, the user. All are united in the pursuit of one unique endeavour: Defining the Minimum Viable Product (MVP).\n\nAs a partner, NearForm provides clients with not just the strategy but also the skillset that will be the backbone of your project. Whether its mobile, design, front- or back-end expertise, we build the team for your success.\n\nThis design sprint process originates from Google Ventures . It has allowed companies such as Medium, Slack or Nest to build bigger, better and faster solutions.\n\nA 5 Day design sprint\n\nHeres a sample breakdown of how it works : Monday is a day for the product owner to expose the problem they are trying to solve. This is also the time to discuss it with the experts around the table. Personas representing user groups are designed and a journey map created. Tuesday is a space for solution exploration. Pros and cons of existing solutions, ideation workshop, sketching out everything is strongly encouraged. Wednesday is the time to review the previous days ideas, rank concepts and sketch the main concept. On Thursday the testable prototype is built, technical roadmap defined and the elevator pitch finalised. The prototype is usually built in InVision. Friday is the day to test the prototype with potential customers, end users. It's the time your assumptions face reality.\n\nThe Invision prototype is one of the deliverables of the Design Sprint Week.\n\nThe persona created at the beginning of the design sprint helps stakeholders think from the audience's point of view.\n\nSo what makes a successful design sprint?\n\nHere are a few learnings from our own experience that contribute to a successful design sprint.\n\n“Free up your calendar We recognize that five days is a big commitment. However, this is the most efficient way to do a design sprint. It ensures constant progress, holistic discussions and avoids misalignments. It's also a great team building exercise.”\n\n“Concentrate on your audience and outcomes, not on features Quite often, the discussion starts around a feature set. This is a mistake, as features are solutions to problems that haven't been identified. We always take a step back to the customer needs which the solution is trying to solve. Throughout the design sprint, the users' points of view serve as the reference.”\n\n“Recognize and minimize assumptions During the design sprint, assumptions are made. It can be around users technical skills, their availability to use a product, their environment, etc. Most of these are assumptions that must be recognised as such. Regular testing with users will help you validate it... or not.”\n\nThe insights gained during the design sprint are countless for all stakeholders. The Minimum Viable Product is defined, and with it, the viability, feasibility and desirability of the project. All in all, the perfect foundation for a successful product and team collaboration. Need help with product development? Contact us to see how we can help!"
],
"categories": {
"primary": "product",
"others": [
"design"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-sharing-react-components-with-lerna": {
"href": "https://nearform.com/insights/sharing-react-components-with-lerna",
"postType": "blog",
"slug": "insights-sharing-react-components-with-lerna",
"date": "2017-08-18",
"title": "A brief look into the Monorepo architecture and how to make that liveable day-to-day",
"authors": [],
"content": [
"All the code for this article can be found here",
"Lerna - Component Sharing in React done right!\n\nAt NearForm, we've mostly become accustomed to creating well-architected applications with components in React. We've been doing it for quite some time, but we see the same stumbling block over and over: how do you effectively share front-end components across various projects within an organisation?\n\nWhere do you start? Do you create a repository for each component? One repo for everything? A toolkit?\n\nFor me, this is where the monorepo approach steps in. It's a convenient way to manage your small distributable pieces of code without the overhead of many repositories and the setup involved with that. That's not to say that a monorepo doesn't have its own drawbacks but we're able to alleviate the majority of that pain with a tool called Lerna .\n\nWhat is Lerna?\n\n“Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.”\n\nOkay but what does that really mean?\n\nIn short, we have one repository with all our code. We set up a containing repository for all our packages to live in which allows us to manage a single setup at the top. By setup I mean testing, linting, build processes, continuous integration and the like.\n\nThis allows our packages to stay extremely small and manageable and any change in our setup can just be reflected in a single place.\n\nGetting started with Lerna\nInstalling Lerna\n\nTo get started let's initialise Lerna and NPM.\n\nnpm install -g lernagit init lerna-repo && cd lerna-repo\ntextCopy to clipboard\n\nNow let's initialise Lerna with lerna init --independent . This will create a lerna.json and package.json file. In this case were using Lerna in independent mode which allows us to individually manage the version of each package.\n\nThere is also a default mode called fixed/locked which uses a single version number in the lerna.json file to publish packages when they've been updated, this method is what babel use in their Lerna setup and is less involved than manually bumping versions for each changed package.\n\nThe mode you choose will come down to personal preference and what you're using the Monorepo for. Independent mode allows me to control each package as I choose and prevents me releasing major versions unnecessarily, this might not be a concern depending on what you're publishing.\n\n“Use this if you want to automatically tie all package versions together. One issue with this approach is that a major change in any package will result in all packages having a new major version.”\n\nBasic React Setup\n\nWe're going to use Lerna as a home for a set of React components wed like to share. For this were going to add a few common libraries like Babel, Storybook and Jest in, a copy of our package.json is below:\n\n{\n \"devDependencies\": {\n \"@storybook/react\": \"^3.0.0-alpha.4\",\n \"babel-cli\": \"^6.24.1\",\n \"babel-jest\": \"^20.0.3\",\n \"babel-plugin-transform-es2015-modules-commonjs\": \"^6.24.1\",\n \"babel-preset-env\": \"^1.5.1\",\n \"babel-preset-react\": \"^6.24.1\",\n \"enzyme\": \"^2.8.2\",\n \"glob-loader\": \"^0.3.0\",\n \"jest\": \"^20.0.0\",\n \"lerna\": \"^2.0.0-rc.4\",\n \"react\": \"^15.5.4\",\n \"react-dom\": \"^15.6.1\",\n \"react-test-renderer\": \"^15.6.1\"\n },\n \"jest\": {\n \"resetModules\": true,\n \"testMatch\": [\n \"**/?(*.)(spec).js?(x)\"\n ],\n \"testPathIgnorePatterns\": [\n \"node_modules/\"\n ]\n }\n}\ntextCopy to clipboard\n\nNow we add in a .babelrc file:\n\n{\n \"presets\": [\"env\", \"react\"],\n \"env\": {\n \"test\": {\n \"plugins\": [\n \"transform-es2015-modules-commonjs\"\n ]\n }\n }\n}\ntextCopy to clipboard\n\nWe're now going to add some basic Storybook configuration to allow it to find our packaged stories using glob-loader . Create a directory called .storybook . Within that we need to create two files: config.js\n\nimport React from 'react'\nimport { configure } from '@storybook/react'
function loadStories () {\n require('glob-loader!./stories.pattern')\n}
configure(loadStories, module)\ntextCopy to clipboard\n\nstories.pattern\n\n../packages/**/*.story.js\ntextCopy to clipboard\nOur First Package\n\nWe're ready to start putting packages into our Lerna repo. Start by creating a packages directory which is the default Lerna directory. This can be changed in the lerna.json configuration if you want a different name for your packages.\n\nTo save time, clone down our example repository . Open the examples directory and copy our test-package into your packages directory. If you've got some extra time, feel free to create your own package similar to ours!\n\nThe example package contains a small React component with a Jest test and a Storybook story.\n\nBootstrapping\n\nSo an obvious drawback of having all our packages in one repository is having to go into each package and npm install as the Monorepo grows this would quickly become unmanageable and time-consuming. Lerna has various built-in commands to ease the drawbacks and bootstrap is one of them!\n\nSimply run lerna bootstrap at the top level and Lerna will automatically install the required dependencies for each package as well as resolving internal dependencies by creating symlinks.\n\nStorybook, Jest\n\nNow that we're set up and have an example package to test with, we can start to see the benefits of Lerna. Let's add some scripts into our package.json :\n\n\"start\": \"NODE_ENV=development start-storybook -p 9001 -c .storybook\",\"test\": \"jest --verbose\"\ntextCopy to clipboard\n\nOnce you've added the scripts, try running each of them! Storybook should automatically detect any stories within our packages allowing us to have a live documentation of our components for everyone to see. Equally, Jest will do the same, looking in our packages for any tests that need running, we can even add coverage with a few lines of code.\n\nBuilding\n\nAnother great feature of Lerna is the exec command. Essentially Lerna allows us to run a command within each of our packages. In our case this is really useful for building our components ready to be published onto npm.\n\nHeres an example of how we go about compiling each of our React components:\n\nlerna exec --parallel -- babel src -d dist --ignore spec.js,story.js\ntextCopy to clipboard\n\nDemonstrating this on a single package is probably fairly underwhelming but as the Monorepo grows you can easily see the benefit of being able to do this in a single line of code. Again if your build process changes it's in one place.\n\nBut I want a different build process for each package? - Simple, just use lerna exec npm run build and define a custom build script within each repositories package.json !\n\nPublishing\n\nSo, we've got a great setup that's all managed in one place, we have a great developer environment but the final piece of the puzzle is using publish .\n\nPublish is another great time and hassle saver, it automatically pushes up all of your packages to Git and NPM, specifically Lerna documents it by doing the following:\n\nRun the equivalent of lerna update to determine which packages need to be published.\nIf necessary, increment the version key in lerna.json.\nUpdate the package.json of all updated packages to their new versions.\nUpdate all dependencies of the updated packages with the new versions, specified with a caret (^).\nCreate a new git commit and tag for the new version.\nPublish updated packages to npm.\nSummary\n\nIf the post hadn't already given it away, I'm really onboard with Lerna. I see the huge benefits of having a Monorepo and all our code in a single place and, from a developer perspective, it feels like a great step towards making our own lives easier when it comes to maintaining everything.\n\nThe Monorepo architecture combined with Lerna resolves the majority, if not all, of the drawbacks that come with maintaining and using a Monorepo and its something I'm keen to see become more commonplace.\n\nInterested in React? You might find these articles interesting\nHow to use GraphQL in React using hooks\nAnimation in React\nSay Hello to React Browser Hooks\nForget Everything you Learned about React – Hooks Rock!"
],
"categories": {
"primary": "frontend",
"others": [
"devops",
"product"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-develop-in-minishift-deploy-to-openshift": {
"href": "https://nearform.com/insights/develop-in-minishift-deploy-to-openshift",
"postType": "blog",
"slug": "insights-develop-in-minishift-deploy-to-openshift",
"date": "2017-08-22",
"title": "Quickly deploy your local Minishift app to Openshift on AWS",
"authors": [],
"content": [
"We have previously shown the benefits of using Minishift as a local development environment. In this post, we'll walk you through installing OpenShift Origin on AWS and then use it to deploy our Minishift demo project. If you haven't run through our Minishift Demo, do that now before proceeding. Our demo is quick to get up and running and is required to deploy the demo to an OpenShift cluster on AWS.\n\nProvisioning the OpenShift Cluster on AWS\n\nWe will provide you with an easy and quick setup of OpenShift Origin, but more details see REDHAT AWS DEPLOYING OPENSHIFT CONTAINER Before starting, you will need to a registered domain added to Route53 as a Hosted Zone. You may purchase a registered domain through AWS. You will also need an AWS account with admin rights to EC2 and S3. From the REDHAT AWS DEPLOYING OPENSHIFT CONTAINER documentation:\n\nThe deployment of OpenShift requires a user that has the proper permissions by the AWS IAM administrator. The user must be able to create accounts, S3 buckets, roles, policies, Route53 entries, and deploy ELBs and EC2 instances. It is helpful to have delete permissions in order to be able to redeploy the environment while testing.\n\nQuick Installation\n\nTo quickly provision an Openshift Origin cluster, we have included a Vagrantfile. Vagrant will spin up a virtual machine running CentOS 7 with already installed all required dependencies to run Ansible playbooks successfully. A Greenfield installation of Openshift Origin will start in the AWS Account defined by the exported variables.\n\nGithub App for OAuth\n\nWe use GitHub authentication for logging into OpenShift. You will need to create an OAth App in GitHub using your Hosted Zone from the Route53 setup.\n\nTo setup GitHub OAuth: Go to OAuth applications -> register new application Enter Name URL: needs to match https://openshift-master.HostedZone Callback URL needs to match is https://openshift-master.HostedZone/oauth2callback/github\n\nRegister application creates the clientID and secret that you will need to use these in the environment variables in the next step.\n\n. See here for more details on setting up an app on GitHub Quick Openshift Origin Setup\n\ngit clone https://github.com/nearform/openshift-ansible.git\nexport environment variables\ncd to reference-architecture/aws-ansible\nrun make\n\nThe provisioning of OpenShift Origin takes about 50 minutes to run and should end with a list of snapshots. [INFO] Getting list of all snapshots ==> default: 2017-07-17 23:06:55 [INFO] Found 4 snapshots ==> default: 2017-07-17 23:06:55 [INFO] Processing snapshot 1 of 4 total snapshots ==> default: 2017-07-17 23:06:55 [INFO] Processing snapshot 2 of 4 total snapshots ==> default: 2017-07-17 23:06:56 [INFO] Processing snapshot 3 of 4 total snapshots ==> default: 2017-07-17 23:06:56 [INFO] Processing snapshot 4 of 4 total snapshots ==> default: 2017-07-17 23:06:56 [INFO] Completed processing all snapshots ==> default: 2017-07-17 23:06:56 [INFO] Graffiti Monkey completed successfully! OpenShift-Ansible Version The openshift/openshift-ansible repository is a development branch. We have targeted the stable branch release-1.5. You can change the release version in nearform/openshift-ansible On line 97 of reference-architecture/aws-ansible/Vagrantfile change the release to the desired version. Note: the OpenShift-Ansible release must match the version used in OpenShift Origin. cd openshift-ansible && git checkout release-1.5\n\nEC2 Instances\n\nWhen provisioning is successful, you will have nine instances running on EC2. Three masters, three infrastructure nodes, two application nodes, and one bastion server.\n\nBelow is the purpose for each type as described in RedHat AWS DEPLOYING OPENSHIFT CONTAINER :\n\nBastion Server: the bastion server in this reference architecture provides a secure way to limit SSH access to the AWS environment. The master and node security groups only allow for SSH connectivity between nodes inside of the Security Group while the bastion allows SSH access from everywhere. The bastion host is the only ingress point for SSH in the cluster from external entities. When connecting to the OpenShift Container Platform infrastructure, the bastion forwards the request to the appropriate server. Connecting through the bastion server requires specific SSH configuration. The .ssh/config is outlined in the deployment section of the reference architecture guide.\nMaster Nodes: maintains the cluster's configuration, manages nodes in its OpenShift cluster. The master assigns pods to nodes and synchronizes pod information with service configuration. The master is used to define routes, services, and volume claims for pods deployed within the OpenShift environment.\nInfrastructure Nodes: The infrastructure nodes are used for the router and registry pods. These nodes could be used if the optional components Kibana and Hawkular metrics are required. The storage for the Docker registry that is deployed on the infrastructure nodes is S3 which allows for multiple pods to use the same storage. AWS S3 storage is used because it is synchronized between the availability zones, providing data redundancy.\nApplication Nodes: The Application nodes are the instances where non-infrastructure based containers run. Depending on the application, AWS specific storage can be applied such as an Elastic Block Storage which can be assigned using a Persistent Volume Claim for application data that needs to persist between container restarts. A configuration parameter is set on the master which ensures that OpenShift Container Platform user containers will be placed on the application nodes by default.\nOpenshift Web Console\n\nThe OpenShift web console can now be accessed using the URL https://openshift-master.PUBLIC_HOSTED_ZONE ( the URL that was setup in your GitHub application.\n\nBrowser Warning\n\nSince we are using a self-signed certificate, the browsers are going to throw an error, You can bypass this for testing by clicking advanced and continue. Learn how to define your certificates to use in the Ansible playbook here\n\nContinue by logging into Openshift using your Github credentials.\n\nOops, something went wrong.\n\nYou may start the deployment scripts but fail during the OpenShift provisioning. In this case, use Cloud Formation and delete the stack label openshift-infra then try again.\n\nDeploying the Minishift demo\n\nNow we have an Openshift Cluster running on AWS, and we used Minishift for our local development. Now we are ready to deploy our hello-server app to the OpenShift cluster.\n\nSetup GIT Deploy Keys The git repo will need to be accessible using your deploy keys. Detail instructions on setting up deploy keys found at https://developer.github.com/v3/guides/managing-deploy-keys - Run this command below to match the SSH key filename used in the deploy script: ssh-keygen -t rsa -b 4096 -C \"your_email@example.com\" -f \"deploy\" - Edit the scripts/create-openshift-project.sh and replace username with your GitHub account name.GIT_REPO=git@github.com:username/minishift-demo.git - Copy your public key (deploy.pub) to the Key text box in the GitHub Deploy Keys for the minishift-demo repo the add Key.\n\nInstall and login with the CLI\n\nThe Minishift installation downloads the Openshift Origin CLI called oc and logged into your local minishift environment. We will need to point oc to your AWS hosted Openshift cluster. - Go to you OpenShift web console found at https://openshift-master.example.com/console/ - Once there, click Command Line tools next to your login:\n\nCopy your login command with the token to clipboard:\nRun the login command on your local terminal. Will look like below: oc login https://openshift-master.your_hostname --token=your-token\nCreate Openshift Project\n\ncd into the scripts directory and run ./create-openshift-project.sh to create the demo project on the Openshift cluster in AWS.\n\nThis script takes openshift/openshift-demo.yaml - a template file that defines all the resources needed to run the application and passes it into the oc cli tool (pointing to the AWS openshift cluster) along with some additional parameters. The values in the YAML file get passed into the OpenShift cluster where the resources get created.\n\nYou should see a Success message when the script finishes. To deploy the demo app execute oc start-build hello-server which will build and deploy the hello-server in openshift.\n\nWhen the demo project has successfully deployed, visit the openshift web console again, and you will see the Demo Project\n\nPipeline Build\n\nOpenShift uses the Jenkins pipeline plugin to execute the build and deploys.\n\nIn the openshift/openshift-demo.yaml, we defined a jenkins pipeline strategy that can build and deploy your project on a push to master.\n\nWe will need to copy the GitHub webhook URL from Openshift to GitHub.\n\nCopy the GitHub Webhook URL from Pipeline configuration to the clipboard. In GitHub in your minishift-demo repo, go to settings and click Webhooks -> Add WebHook then: - Paste in the WebHook URL into the payload text box - Select application/json as content type and click add webhook.\n\nNow, any code change pushed to your master repo will trigger a build and deploy on OpenShift using the Jenkins pipeline.\n\nThanks for going through the Openshift deployment. There are more details in the nearform/openshift-ansible README and we look forward to your comments and feedback below.\n\nAll the source code and another copy of this article could be found at: - https://github.com/nearform/openshift-ansible = https://github.com/nearform/minishift-demo"
],
"categories": {
"primary": "cloud",
"others": [
"devops",
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-securing-docker-containers-on-aws": {
"href": "https://nearform.com/insights/securing-docker-containers-on-aws",
"postType": "blog",
"slug": "insights-securing-docker-containers-on-aws",
"date": "2017-08-23",
"title": "Create an AWS Ubuntu AMI for securing docker containers",
"authors": [],
"content": [
"Secure Docker Containers\n\nIn 2015 the Docker team set out to consolidate all the information around best practices for securing docker containers.\n\nYou can find references to the CIS Docker Community Edition Benchmark version 1.1.0 and Docker's own white paper Introduction to Docker Security on the topic here .\n\nThe links above get you more directly to the assets. Just an FYI, the CIS benchmark will cost you an email address to access the download.\n\nOn most projects at NearForm we are deploying our solutions within Docker containers.\n\nThere are tasks that are repeated on each project to secure and harden off those deployments, and we built this packer template to produce a quick and easy way for you to spin up an AWS AMI that passes the Docker-Bench-Security script.\n\nThe Docker-Bench-Security repo is a work product of the above-mentioned consolidation efforts by the Docker team.\n\nTo accomplish the building of this AMI we use Packer , which is an easy way to automate the creation of your images. It supports multiple providers (i.e. AWS, Digital Ocean) and allows you to document in a repo how your images were built and modifications that were done to them over time.\n\nPassing Docker-Bench-Security\n\nThe work to get this AMI passing the Docker-Bench-Security was not a small task, but a critical task of any development organization.\n\nMinimizing the attack surface of an application we deploy into the wild will always return on investment.\n\nWhether that return comes from keeping an active attacker from escalating privileges and damaging our craft or simply letting you sleep at night knowing that you've done everything you can to secure your environments, give it a spin and feel free to raise issues or PRs if you find ways to improve upon the work.\n\nTL;DR\n\nJust give me an AMI I can work with\n\nThe Work\n\nTo get things started we are using Packer to create an ebs-backed AMI with the underlying OS being Ubuntu 17.0.4.\n\nWe are also installing Docker CE for Ubuntu with the supporting packages suggested by the Docker Team.\n\nOut of the box, our base Ubuntu AMI is failing the following tests in the Docker-Bench-Security .\n\nWe'll go through each and document how we passed each test.\n\n[WARN] 1.1 - Ensure a separate partition for containers has been created\n[NOTE] 1.2 - Ensure the container host has been Hardened\n[WARN] 1.5 - Ensure auditing is configured for the Docker daemon\n[WARN] 1.6 - Ensure auditing is configured for Docker files and directories - /var/lib/docker\n[WARN] 1.7 - Ensure auditing is configured for Docker files and directories - /etc/docker\n[WARN] 1.8 - Ensure auditing is configured for Docker files and directories - docker.service\n[WARN] 1.9 - Ensure auditing is configured for Docker files and directories - docker.socket\n[WARN] 1.10 - Ensure auditing is configured for Docker files and directories - /etc/default/docker\n[WARN] 1.12 - Ensure auditing is configured for Docker files and directories - /usr/bin/docker-containerd\n[WARN] 1.13 - Ensure auditing is configured for Docker files and directories - /usr/bin/docker-runc\n[WARN] 2.1 - Ensure network traffic is restricted between containers on the default bridge\n[WARN] 2.5 - Ensure aufs storage driver is not used\n[WARN] 2.8 - Enable user namespace support\n[WARN] 2.11 - Ensure that authorization for Docker client commands is enabled\n[WARN] 2.12 - Ensure centralized and remote logging is configured\n[WARN] 2.13 - Ensure operations on legacy registry (v1) are Disabled\n[WARN] 2.14 - Ensure live restore is Enabled\n[WARN] 2.15 - Ensure Userland Proxy is Disabled\n[WARN] 4.1 - Ensure a user for the container has been created\n[WARN] 4.5 - Ensure Content trust for Docker is Enabled\n[WARN] 4.6 - Ensure HEALTHCHECK instructions have been added to the container image\n[INFO] 4.7 - Ensure update instructions are not use alone in the Dockerfile\n[INFO] \\\\\\\\\\\\\\* Update instruction found: [node:8]\n[INFO] 4.9 - Ensure COPY is used instead of ADD in Dockerfile\n[INFO] \\\\\\\\\\\\\\* ADD instruction found: [node:8]
[INFO] 5 - Container Runtime\n[WARN] 5.2 - Ensure SELinux security options are set, if applicable\n[WARN] 5.10 - Ensure memory usage for container is limited\n[WARN] 5.11 - Ensure CPU priority is set appropriately on the container\n[WARN] 5.14 - Ensure 'on-failure' container restart policy is set to '5'\n[WARN] 5.25 - Ensure the container is restricted from acquiring additional privileges\n[WARN] 5.28 - Ensure PIDs cgroup limit is used\n[INFO] 5.29 - Ensure Docker's default bridge docker0 is not used\ntextCopy to clipboard\n1.1 - Ensure a separate partition for containers has been created\n\nTo pass 1.1 we had to do a few tasks that included modifying our underlying packer template and then scripting a few changes for the AMI build.\n\nThe volume that is defined in the ami_block_device_mappings will be setup and used as the mount for all things Docker. (i.e. images, containers).\n\nFew notes on this block device: - it's currently set for 1GB in size, which is not ideal for anything but testing. (i.e. \"volume_size\": 1 ) - \"device_name\" : \"/dev/sdf\" gets renamed hence the different name in our ./scripts/volumn.sh From my research this appeared to be because of the specific AMI we are using, but to dig deeper on the warning from AWS and reasoning behind them read more The actual work of mounting and configuring the disk for operation is completed in ./scripts/volumn.sh .\n\n1.2 Ensure the container host has been Hardened\n\nThe underlying host of the AMI we create IS NOT hardened and the Docker-Bench-Security DOES NOT verify the host's settings.\n\nYou could easily switch the AMI ID that we are using in our example with one from CIS Benchmarks and start fresh with a hardened host and a secure docker implementation.\n\n1.5 - 1.13 Auditing\n\nThese tests are passed with the installation and configuration of auditd , which is all done within auditd.sh .\n\nWith our settings we are watching for writes and changes to the attributes of each item. more info Each test is listed above the setting for reference.\n\n2.1 - 2.15 Docker daemon configuration\n\nWe accomplish passing these test with the creation of /etc/docker/daemon.json from ./files/daemon.json .\n\n2.1 - Ensure network traffic is restricted between containers on the default bridge\n\nThis warning seems to contradict 5.29, which we implment a solution for alter on in this post.\n\n2.5 - Ensure aufs storage driver is not used\n\nSince we created our new parition with ext4 we are able to use the preferred storage-driver which is overlay2. - Docker CE on Ubuntu storage drivers - supported backing filesystems\n\n\"storage-driver\": \"overlay2\"\ntextCopy to clipboard\n2.8 - Enable user namespace support\n\n\"The best way to prevent privilege-escalation attacks from within a container is to configure your containers applications to run as unprivileged users.\" source Using default means our user will be dockremap versus something that may make more sense in your environment.\n\n\"userns-remap\": \"default\"\ntextCopy to clipboard\n2.11 - Ensure that authorization for Docker client commands is enabled\n\nWe do not implement changes (i.e we still get a WARN result) for this since it requires generation of keys and certificates that will be unique to your environment.\n\nYou can read more on the details of how to configure the docker daemon and how to protect the docker socket\n\n2.12 - Ensure centralized and remote logging is configured\n\nWe are writing everything to syslog, but you could easily configure aws-logs or another end point that you actively work with for logging.\n\n\"log-driver\": \"syslog\",\n\"log-opts\": { \"syslog-address\": \"udp://1.2.3.4:1111\" }\ntextCopy to clipboard\n2.13 - Ensure operations on legacy registry (v1) are Disabled\n\nThis is possibly a legacy configuation since support *\"for the v1 protocol to the public registry was removed in 1.13\"*.\n\n\"disable-legacy-registry\": true\ntextCopy to clipboard\n2.14 - Ensure live restore is Enabled\n\nThis setting allows your containers to continue running when there are issues that cause the docker daemon to be down. more info\n\n\"live-restore\": true\ntextCopy to clipboard\n2.15 - Ensure Userland Proxy is Disabled\n\"userland-proxy\": false\ntextCopy to clipboard\n4.1 - Ensure a user for the container has been created\n\nThis warning is looking for you to run your processes within the container as seperate users.\n\nWith userns-remap enabled for our containers this seems a bit excessive since the container is restricted to a lower privileged user.\n\nThis could be a wrong assumption, but restricting users that run your process inside the container could have other adverse effects. This should be evaluated and configured based on your applications needs.\n\nThis test was passed by modifying the Dockerfile to include adding a new user and group.\n\nYou can confirm your settings are in place with the following.\n\n$ docker inspect --format 'User={ {.Config.User} }' flamboyant_yonath\nUser=app:app
$ docker exec -it flamboyant_yonath /bin/bash
app@188099606d58:~/my-project$\ntextCopy to clipboard\n4.5 - Ensure Content trust for Docker is Enabled\n\nWe take care of this by setting the environment variable DOCKER_CONTENT_TRUST for all users in ./scripts/docker-setup.sh .\n\nNote that if you build an image locally there are known issues with Docker Content Trust disallowing your action with an authorization failure (i.e. 401).\n\nYou can use the --disable-content-trust flag to bypass the setting we created in our docker setup script\n\n$ docker run --disable-content-trust example:1.0.0\ntextCopy to clipboard\n\nDocker Content Trust has no concept of trusting something built locally #25852\n\n4.6 - Ensure HEALTHCHECK instructions have been added to the container image\n\nAdding the following lines to the Dockerfile gets us past this test. This checks the url every 20s and fails after 3s.\n\nHEALTHCHECK --interval=20s --timeout=3s \\\nCMD curl -f https://localhost:8000/ || exit 1\nexample:1.5.1 Up 2 seconds (healthy) 8000/tcp fervent_kilby\nexample:1.5.0 Up 7 minutes (unhealthy) 8000/tcp wonderful_tesla\ntextCopy to clipboard\n4.7, 4.9 Commands UPDATE && ADD\n\nIt appears the node:8 image has a few items that docker-bench-security does not like.\n\nRunning docker history on the image we build with our example will show that those commands are getting into our images based on FROM node:8\n\n5.10, 5.11 Memory && CPU\n\nBy default, a docker container has no resource constraints.\n\nYou can set specific runtime flags for both Memory and CPU based on your applications needs. more info I had no success with the proposed flags and passing the tests that docker-bench-security runs for them.\n\n5.14 - Ensure 'on-failure' container restart policy is set to '5'\n\nThis can be accomplished by adding the flag --restart to your docker run command as below:\n\ndocker run -d --disable-content-trust --restart on-failure:5 example:1.5.1\ntextCopy to clipboard\n5.25 - Ensure the container is restricted from acquiring additional privileges\n\nThis can be accomplished by adding the flag --security-opt to your docker run command\n\ndocker run -d disable-content-trust restart on-failure:5 security-opt=\"no-new-privileges:true\" example:1.5.1\ntextCopy to clipboard\n5.28 - Ensure PIDs cgroup limit is used\n\nI've yet to find a solution that works and passes the test.\n\nThe runtime flag --pids-limit does not appear to have any effect on the configuration of a container.\n\n5.29 - Ensure Docker's default bridge docker0 is not used\n\nIf you are interested in using a bridge other than docker0 you can build your own bridge\n\nRequirements\nPacker\nAWS account - free tier will work with our examples\nSecurity Credentials from your AWS account\nSetup\nSetup packer\n$ git clone git@github.com:nearform/devops.git\nCreate a file named variables.json in the root of the example repo from #2. Add your access and secret keys.\n\nIf you want to use a different region in AWS, change the aws_region and find replace the ubuntu_source_ami with the AMI ID that has an instance type of hvm-ssd from Ubuntu Your File should look something like this:\n\n{\n\"aws_secret_key\": \"< INSERT YOUR SECRET KEY >\",\n\"aws_access_key\": \"< INSERT YOUR ACCESS KEY >\",\n\"aws_region\": \"us-east-1\",\n\"ubuntu_source_ami\": \"ami-cb1d41b0\"\n}\ntextCopy to clipboard\nBuild the AMI After executing the command below you should see an AMI in the AWS Console \n$ packer build -var-file=variables.json base-image-us-east-template.json\ntextCopy to clipboard\n\nOur example template uses packer's amazon-ebs builder.\n\nWe use a builder and two provisioners in our template to accomplish the following:\n\nCreate /etc/docker/daemon.json\nCreate and mount the volume that will be used to store our Docker assets\nInstall Docker CE\nClone Docker-Bench-Security\nInstall and configure auditd\nRun an Instance of Your AMI\n\nOnce packer has finished building your AMI, you can log into the AWS Console and launch an instance from the AMI.\n\nYou will only need to configure network related items (i.e. subnet) and the rest you should be able to take the defaults.\n\nOnce your instance is up and running ssh into it and you should find docker-bench-security in the home directory of the ubuntu user.\n\nDo the following to run and verify the AMI passes the test we've documented above.\n\n$ cd docker-bench-security\n$ sudo sh ./docker-bench-security.sh\n\n*note*: sudo is used in step 2 due to checks against files and directories that require elevated privileges.\n\nRun a Container in your Instance\n\nIf you would like to build a docker image and run a container based on that image you can include ./files/app/ in your AMI build, which will give you a simple hapi app to run the Docker-Bench-Security tests against.\n\nTo include these files add the following to the provisioners section of the template\n\n{\n \"type\": \"shell\",\n \"inline\": [\"mkdir ./app/\"]\n},\n{\n \"type\": \"file\",\n \"source\": \"./files/app/\",\n \"destination\": \"./app/\"
}\ntextCopy to clipboard\nMore content about Docker\nStatic Analysis of Docker Image Vulnerabilities with Clair\nHow to run a Public Docker Registry in Kubernetes\nBuilding Docker images in Go"
],
"categories": {
"primary": "cloud",
"others": [
"security",
"devops",
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-architecting-electron-applications-for-60fps": {
"href": "https://nearform.com/insights/architecting-electron-applications-for-60fps",
"postType": "blog",
"slug": "insights-architecting-electron-applications-for-60fps",
"date": "2017-08-24",
"title": "A deep-dive into optimizations that should be made to achieve smooth rendering, and a low resource footprint for Electron on all platforms",
"authors": [],
"content": [
"Improving the performance of Electron\n\nElectron is a native desktop environment that combines the NodeJS runtime with the Chromium browser. This creates a powerful combination of technologies that allows creating cross-platform applications at a fraction of the cost of other methods.\n\nDespite the many benefits Electron is lauded for, performance often isn't one of them. In this post, we'll dive deep into optimizations that can (and should) be made to achieve smooth rendering and a low resource footprint for Electron on all platforms.\n\nWe'll discuss how to optimize the performance of booting & rendering Electron app, and various tools to help you debug performance problems.\n\nBooting\n\nBefore applications can respond to user input, they generally go through an unresponsive period where they are booting up. To provide a snappy user experience, this phase should be reduced to a bare minimum.\n\nSome optimizations are already done when compiling ; for example, resolving require() calls ahead of time, so fs.readFileSync() does not occur while booting. This means that startup performance whilst developing may not always reflect performance in production.\n\nUsing DOMContentLoaded\n\nBooting itself is defined by a few phases:\n\nloading - load the resources into the Electron runtime\ninterpreting - convert the static resources into executable code\nexecuting - execute the code\n\nTo achieve high performance while booting, the goal should be to reduce the amount of the work that needs to be done. To quote Dominic Tarr : \"Software performance is losing weight not building muscles\" .\n\nProbably the most efficient technique to improve boot performance is to wait for the 'DOMContentLoaded' event before executing any non-UI critical code. This event fires after the initial set of code is done executing, and before any timers start resolving. 'DOMContentLoaded' has the drawback that if a listener is attached after the event has already fired, it will never fire. To circumvent this, it's recommended to use the document-ready package. This checks the ready state on the page before attaching the listener and fixes this problem altogether.\n\nPrerendering\n\nBuilding Electron applications often have the same performance considerations as building websites. More specifically: what works well for websites often also works well for Electron. One of the optimizations that are particularly interesting to improve boot times is prerendering.\n\nPrerendering is the practice of taking a JavaScript application and compiling it to static HTML. This means that when booting up the application, the browser process can start painting things on the screen without needing to first interpret and execute JavaScript. This should significantly improve perceived startup performance, making applications feel way snappier.\n\nAs illustrated in this image, it's only after the JavaScript has loaded that the browser can paint the actual UI - causing the first useful UI to show up around the 900ms mark.\n\nBut if we prerender the HTML and serve it directly, the UI is created about half a second earlier. And on top of that, once the JavaScript has been executed, the second paint is virtually instant because no DOM nodes have to be invalidated. Pretty sweet!\n\nRendering\n\nAfter the application has finished booting, it is ready to respond to outside input. This phase is commonly referred to as the main loop. Optimizations during this phase are usually geared towards reducing resource usage and scheduling actions efficiently.\n\nCompiling UI code\n\nTo achieve high runtime performance, it's common to compile browser applications and apply transforms using tools such as browserify . For example, when using template strings to create DOM nodes , you'll want to compile it to static document.createElement()calls instead of reparsing HTML on every call. Though this is an example, there are plenty of optimizations that can drastically improve front-end performance.\n\nBut because these compilation tools are usually built to make Node code work in the Browser, they don't necessarily work well for environments such as Electron that can run code directly. Particularly when dealing with Native Addons (e.g. C++ ) things can get hairy, and debugging them is never fun.\n\nThe solution to this problem is fairly straightforward. By separating Browser code from Node code, the browser code can directly be targeted. The most efficient way of doing this, is to use require() for Browser code, and window.require() for Node code:\n\n// This code will be compiled by Browserify\nvar foo = require('./bar')
// This code is ignored by Browserify, but picked up at runtime by Electron\nvar beep = window.require('./boop')\ntextCopy to clipboard\n\nNote: before arriving at window.require() we've experimented with various different approaches. Among others there have been attempts to add C++ support to Browserify, creating separate Node / Browser processes in Electron with shared memory, and variations on the two. In hindsight ,it was fun to have tried these approaches, but window.require() is about as good as it gets.\n\nExplicit scheduling of background tasks\n\nThe Browser's event loop is different from Node's event loop because it is primarily concerned with providing a smooth visual experience for humans. In practice ,this means that it must be able to render 60 frames per second, and every frame (or \"tick\" in Node-speak) has a budget of ~16ms .\n\nEvery frame in the browser is roughly resolved as follows:\n\nAll tasks in the microtasks queueresolve.\nAll calls to setTimeout()whose timer has expired resolve.\nAll queued calls to window.requestAnimationFrame()resolve. If requestAnimationFrame() is called during this step, it's queued for the next frame.\nAt this point the browser expects all UI-related tasks to have been resolved, and starts calculating layout, and painting layers on the screen.\nWhen painting has finished, and there's likely time left on the frame, the browser starts something called the \"idle period\". During this phase all queued callbacks from priorwindow.requestIdleCallback() calls are resolved.\n\nNote: calls to requestIdleCallback() may resolve at a much slower rate (e.g.every 10 seconds) if the window is in the background. From the APIs we mentioned above, window.requestIdleCallback() is probably the most exciting. It allows prioritization of tasks within a single process! This allows us to deprioritize everything that isn't essential to rendering UI; and allows breaking up CPU intensive tasks into chunks that resolve over multiple frames.\n\nBut like with most things, it doesn't come without drawbacks. Because all callbacks are resolved during the idle period, it requires carefully checking the time remaining at the start of each call. If there's not enough time remaining on the tick, the callback should be re-queued onto the next tick. Luckily the on-idle takes care of all that for you:\n\nvar onIdle = require('on-idle')
onIdle(function () {\n console.log(\"hello from the idle period\")\n})\ntextCopy to clipboard\nDebugging\n\nSo far we've talked about what you can do to improve the performance of applications. But up front knowledge is only half the work - catching warnings, and acting on them is equally important. Note: At the time of writing, some of these features we'll be discussing here rely on the beta release Electron 1.7, which uses Chrome 58. The Electron beta can be installed from npm as electron@1.7.\n\nToggle verbose mode in console\n\nProbably the easiest way of catching performance regressions is by setting the log level to verbose through the new log level dropdown in the console. Once enabled, performance regressions will emit actionable warnings.\n\nBrowser Performance API\n\nThe final tool in the toolbox we'll be covering today is the DOM Performance API . This API contains all sorts of information about the Browser's performance, but more importantly, it allows creating measuring the time elapsed between two points, and displays them in the devtool's timeline.\n\nNow before we continue, it's worth mentioning Node's new async_hooks API available in its current form without flags since Node 8.2. Because we're discussing Electron, we get to choose which API we use to create custom performance entries. Although Node's API is potent, the Browser's API is more friendly to use, and has the benefit that it integrates directly with the DevTools.\n\nTo create a new Performance Measure on the Performance Timeline, performance.measure(name, firstMark, secondMark) should be called to measure the time spent between two calls to performance.mark(name) . Once a measure has been created, it can be seen on the Performance Timeline under the\"User Timing\" dropdown.\n\nA simpler way of creating these marks is by using the nanotiming module:\n\nvar nanotiming = require('nanotiming')
var timing = nanotiming('my-timing.my-loop')\nvar i = 1000\nwhile (--i) console.log(i)\ntiming()\ntextCopy to clipboard\n\nSometimes it can be useful to act on PerformanceEntries . For example: you might want to send them back to a server for later inspection, or log them out during debugging to catch performance problems early.\n\nWhile there are several APIs that allow retrieving Performance Entries, the PerformanceObserver API is by far the most powerful. However, one of the downsides of using it is that it only starts emitting events after the observer has been created. To react to these events, we recommend using the on-performance module. Not only does it retrieve all Performance Entries once it's attached, it also clears them from the Browser's internal timing buffer so new events can keep flowing in without overflowing the buffer.\n\nvar onPerformance = require('on-performance')
onPerformance(function (entry) {\n console.log('entry: ', entry.entryType, entry)\n})\ntextCopy to clipboard\nWrapping up\n\nIn this post, we've touched on the different aspects of Electron's performance, how to design code in such a way that it can be optimized, and discussed various APIs that can help with improving performance.\n\nLet us know what you think in the comments below, or drop Nearform or Yosh a line on Twitter. Cheers!\n\nNeed Node experts for your next project? Contact us to see how we can help!"
],
"categories": {
"primary": "perf",
"others": [
"frontend",
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-how-to-build-a-real-time-node-js-performance-monitoring-system": {
"href": "https://nearform.com/insights/how-to-build-a-real-time-node-js-performance-monitoring-system",
"postType": "blog",
"slug": "insights-how-to-build-a-real-time-node-js-performance-monitoring-system",
"date": "2017-08-28",
"title": "Real-time monitoring of Node.js with Kibana, Elasticsearch and nearForm modules",
"authors": [],
"content": [
"Performance monitoring of a Node.js app: Step by step instructions\n\nIn this post, we are going to walk through how to build your own real-time performance monitoring of a NodeJS app and how this can be used to detect issues on live servers.\n\nWe will be using technologies like Elasticsearch, Kibana, Docker Compose, and several NodeJS modules developed here at NearForm.\n\nWe show how you can use Docker Compose to run locally or on AWS and we have also created a Vagrantfile that will spin up all the necessary resources if you wish to run it locally on a hypervisor like VirtualBox.\n\nQuick Setup\n\nLike our other posts that cover several repos, we have created a quick and easy setup for the resources needed to walk through this tutorial locally.\n\nWe have defined the services needed in a Dockerfile. Follow the steps below to spin up resources needed:\n\nIf you have Docker Compose installed locally or a cloud provider like AWS then use docker-compose:\n\ngit clone https://github.com/nearform/slow-rest-api.git\n cd slow-rest-api\n git checkout stats-demo\n cd stats-demo\n docker-compose up\n\nVerify\n\nInstallation will take about 5 minutes to finish but once done you should be able to bring up Kibana in your browser at https://localhost:5601/\n\nWhat it looks like after successful installation.\n\nNote: Running Kibana and Elasticsearch on a different major version is not supported .\n\nIf you have a mismatched version, you will see an error like below:\n\nNode Modules Needed:\n\nWe open sourced a few modules for the real time dashboard, and Ill go over each one.\n\nA link to all resources is at the end of this post.\n\nStats:\n\nWe needed a tool to generate statistics about our running processes, so to do this, we developed the Stats module.\n\nThis simple module is used to create a simple event emitter object that emits an event called stats. It contains useful information about the current host machine, the node.js process itself, Garbage Collection within the process and information about how long it is taking your event loop to empty itself and move onto its next iteration.\n\nYou can also configure this to add tags to your stats, and it generates a unique id per emitter so you can track down stats for a particular emitter.\n\nClick here to learn more: https://github.com/nearform/stats\n\nSTATS-TO-ELASTICSEARCH:\n\nThe next module, Stats-to-elasticsearch, makes it very easy for you to send your stats from in process to your elasticsearch server. T\n\nhis module instantiates the stats event emitter for you, so you simply use this module, and you're ready to go.\n\nYou can even preload your node.js process with this module, requiring no changes to your codebase.\n\nClick here to learn more: https://github.com/nearform/stats-to-elasticsearch\n\nCREATE-STATS-DASHBOARD:\n\nNow that you have your stats sent to your elasticsearch server you can install the dashboard for Kibana to visualize it.\n\nTo do this, we created the module create-stats-dashboard that makes it very easy to install the dashboard. When you have this module installed through npm, you can run the command create-stats-dashboard to create the dashboard.\n\nIt has support configuration env variables to hook it up to any elasticsearch server for creating your dashboard.\n\nClick here to learn more: https://github.com/nearform/create-stats-dashboard\n\nSlow API Troubleshooting\n\nNext, we want to show you how you can use these tools to identify performance issues on a running server.\n\nRun docker ps command to see elasticsearch, Kibana, and two slow-rest-api containers on port 3000 and 3001.\n\nDocker ps will show you the 4 processes running:\n\nNow open Kibana in your browser and click dashboard. You will see that it needs a dashboard created.\n\nInstall dashboard\n\nWe have created a model for generating the dashboard in Kibana.\n\nJust use npm install and then run create-stats-dashboard.\n\nnpm i -g @nearform/create-stats-dashboard\n\nCreate-stats-dashboard\n\n\nNow go to Kibana localhost/5601 and click dashboard, you should see the dashboard like below:\n\nThen click Node-process stats and:\n\nYour dashboard now can show real time what is happening with your app. However, since there is no load, it's not going to show you much.\n\nLets add load using Autocannon.\n\nnpm i -g autocannon\n\nnpm i -g concurrently\n\nconcurrently \"autocannon -f -c 50 localhost:3000/a\" \"autocannon -f -c 50 localhost:3000/b\" \"autocannon -f -c 50 localhost:3000/c\"\n\n\nIn Kibana, you should start to see the average event loop delay is increasing caused by the slow rest api.\n\nYou can also drill into the slow process and dig into the logs to investigate a specific process.\n\nAnd look closer at the event-loop delay:\n\nConclusion\n\nThanks for going through the real-time monitoring demo.\n\nThere are more details in the readme of each of the modules we covered.\n\nWe hope you found this helpful.\n\nNearForm Open Source Repos:\n\nhttps://github.com/nearform/stats\nhttps://github.com/nearform/stats-to-elasticsearch\nhttps://github.com/nearform/import-kibana-dashboard\nhttps://github.com/nearform/create-stats-dashboard\nhttps://github.com/nearform/slow-rest-api\nhttps://github.com/mcollina/autocannon\n\nVagrant real-time stats monitor install: https://github.com/nearform/realtime-stats-monitor Docker: https://docs.docker.com/compose/install/ Elasticsearch install: https://www.elastic.co/guide/en/elasticsearch/reference/2.4/_installation.html Kibana install: https://www.elastic.co/downloads/kibana\n\nAdditional resources on Node.js performance\n\nClinic.js rises to the top in diagnosing Node.js performance issues 5 benefits of Node.js for Digital Enterprise The Cost of Logging & Node.js Performance: Tech Talk Video Tuning Node.js app performance with Autocannon and 0x Sematext Guest Post: PINO – The Fastest Node.js Logger for Production Hapi: A Performance deep-dive Speeding up React SSR: Announcing ESX"
],
"categories": {
"primary": "perf",
"others": [
"backend",
"cloud"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-create-a-cross-platform-choo-electron-desktop-application-with-one-command": {
"href": "https://nearform.com/insights/create-a-cross-platform-choo-electron-desktop-application-with-one-command",
"postType": "blog",
"slug": "insights-create-a-cross-platform-choo-electron-desktop-application-with-one-command",
"date": "2017-08-31",
"title": "Today we're announcing the release of create-choo-electron, a code generator that allows you setup Electron projects with no effort.",
"authors": [],
"content": [
"Choo Electron quick overview\n\nToday we're announcing the preview release of create-choo-electron : a code generator that allows you to create your next Electron application in mere minutes.\n\nIt's similar to its sibling create-choo-app . But instead of creating a new web app, create-choo-electron creates a fresh Electron application. It comes with a build system, a full main process, and a Choo application in a renderer process. In short: everything you need to get started right away.\n\nTo always get the latest version of create-choo-app run it with npx(1) :\n\n$ npx create-choo-electron\n\n\nBecause this is a preview release, there are still some details being polished. Most notably we're currently rewriting the Bankai compiler for frictionless builds during development and production.\n\nWrapping up\n\nAnd that's it. We hope you enjoy the first release of create-choo-electron! Let us know what you think in the comments below, or drop nearForm or Yosh a line on Twitter. Cheers!\n\nNeed Node experts for your next project? Contact us to see how we can help!"
],
"categories": {
"primary": "frontend",
"others": [
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-tuning-node-js-app-performance-with-autocannon-and-0x": {
"href": "https://nearform.com/insights/tuning-node-js-app-performance-with-autocannon-and-0x",
"postType": "blog",
"slug": "insights-tuning-node-js-app-performance-with-autocannon-and-0x",
"date": "2017-09-06",
"title": "Featuring two open source tools developed by Matteo Collina and David Mark Clements of nearForm.",
"authors": [],
"content": [
"Improving your Node.js app's performance\n\nFeaturing two open source tools developed by Matteo Collina and David Mark Clements of NearForm.\n\nPerformance improvements either on a database or a web app require good tools. They should be easy to use and help you identify the issue quickly.\n\nIn this post, I am going to introduce two tools developed at NearForm that do just that to help improve your Node.js Web App. Then we will troubleshoot a RESTful API.\n\nThe Tools\n\nAutocannon is an HTTP/1.1 benchmarking tool developed by our very own Matteo Collina . It is written in Node.js and can work on Linux, Mac, and Windows. It supports HTTP and HTTPS.\n\nTo install globally run:\n\nnpm -i autocannon -g\ntextCopy to clipboard\n\n0x ( pronounced zero ex ) was written by our very own David Mark Clements .\n\nIt will profile and generate an interactive flame graph for a Node.js process in a single command. It works on Linux and Mac\n\nTo install globally:\n\nnpm -i 0x -g\ntextCopy to clipboard\nNode.js Version\n\nIn this post, we are going to target Node.js version 6.\n\nUsing Node.js Version Manager , run the commands below:\n\nnvm install v6\nnvm use v6\ntextCopy to clipboard\nSetup App\n\nWe are going to use our Slow Rest API to demo both autocannon and 0x. From our previous post, Building a real-time dashboard , we know there is a bottleneck based on the Kibana dashboard below, and in this post, we'll dive into what we can find.\n\nTo setup:\n\ngit clone https://github.com/nearform/slow-rest-api.git\ncd slow-rest-api\nnode index\ntextCopy to clipboard\n\nIn another terminal: npm test\n\nGenerating a flame graph\n\nWe need to make sure we're using either Linux or Mac and we have super user access since it uses dtrace or perf.0x index. 0x may ask us for sudo access, this is necessary since the underlying tracing tool (dtrace on Mac or perf on Linux) requires root access. In another terminal, as soon as possible, we run: npm test .\n\nWe can now hit CTRL-C in the first terminal where we ran 0x.\n\nThe 0x process will begin to generate the flame graph in the form of an HTML file and then provide the link to that file.\n\nWhen we copy this link into a browser we should see a flame graph similar to the following:\n\nReading Flame Graphs:\n\nEach box in the graph represents a function and the different colors represents the amount of time the function is on CPU.\n\nA dark red box is a hot function which represents more time was observed on top of stack which would be blocking the event loop. We will then follow the child functions below to see the code path that was taken.\n\nTroubleshoot a performance issue:\n\nWhen we run npm test, we are using autocannon to add load to the server.\n\nThe script looks like this: autocannon -d 20 -c 100 https://localhost:3000/a & autocannon -d 20 -c 100 https://localhost:3000/b & autocannon -d 20 -c 100 https://localhost:3000/c From our demo above, we see the request per sec is at 252.\n\nLet's see if we can improve the Req/Sec.\n\nNow let's take a look at the frame graph.\n\nThe reddest box at the top points to etag.js line 23 which is reading through a cache object using the Array forEach method:\n\nWe can use the flame graphs to follow the code path.\n\nIn the above graph, we see the route https://localhost:3000/a from our autocannon is the slowest.\n\nSo from here we can troubleshoot that route and follow it to determine our fix to the bottleneck.\n\nSince the forEach is a hot point in our flame graph, lets rewrite it using a for loop to see if that improves performance.\n\nLets rewrite the loop as follows:\n\nAfter running autocannon on the change, we see our Requests have improved to 386 on average and throughput 507 kb per sec which is better but still not great.\n\nWhile this is better, we still see that request to the /a route is still hot, causing a bottleneck in the app.\n\nFollowing the code path we also see the Buffer.from and Buffer.toString are also red.\n\nLets try to remove the cache and see what happens.\n\nNow when we run autocannon, we get a big improvement on Requests per second.\n\nIt is now at 1867 on average compared to the initial run of 254.\n\nOur throughput is 2.59 MB per second on average.\n\nAnd our flame graph looks like this:\n\nBy removing the etagCache check and generating the etag each time, we have shown this is better for performance.\n\nThe reason is due to the synchronous forEach, for loop, and using Buffer.from and Buffer.toString() in our cache object which all showed up as red in our flame graph.\n\nWe are also reading in a large JSON file each time we check the cache. It is an unnecessary optimization that did not take into account the size that the cache could become later (usually in production), causing a bottleneck.\n\nTypically, implementing cache gives your application better performance but since this was poorly implemented. Using caching slowed down the performance and the fix was to remove the cache.\n\nIn our case, the poorly implemented cache caused the bottleneck. It is also worth noting that adding a cache is ideal.\n\nIt's better to go higher on the stack using something like Nginx which specializes in caching.\n\nWe have shown how to use autocannon to easily add load to your app and 0x to visualize the stack trace to identify potential bottlenecks in the event loop.\n\nThese easy to use tools can help you measure your app performance and visually find bottlenecks.\n\nNearForm Open Source Repos:\n\nAutoCannon: https://github.com/mcollina/autocannon 0x: https://github.com/davidmarkclements/0x Slow Rest API: https://github.com/nearform/slow-rest-api"
],
"categories": {
"primary": "oss",
"others": [
"perf",
"backend"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-building-an-offline-first-website": {
"href": "https://nearform.com/digital-community/building-an-offline-first-website",
"postType": "blog",
"slug": "digital-community-building-an-offline-first-website",
"date": "2017-09-27",
"title": "Building an Offline-Only Website",
"authors": [
"CHRIS BOLIN"
],
"content": [
"This summer I released a small personal project: Offline. It might be the only website that you forces you offline to view it. (I know, it’s weird.)\n\nThe project’s seed was planted a few years ago when I was having a really hard time concentrating while writing my thesis. I would routinely get lost on the internet, jumping from one Wikipedia page to the next, constantly engaged, but never productive. I got desperate for progress, so I finally tried simply disconnecting my WiFi. It worked. When I got a sudden urge to Google something or check Facebook I was met with a choice: did I really want to be distracted? Did I really want to admit defeat and reconnect?\n\nI was able to write my thesis. And I kept the trick: if I really need to concentrate, I would try completely disconnecting.\n\nThis summer I decided to take it a step further: could I make a website that forced readers into that same disconnected and focused state? The \"offline-first” movement in web development has been gaining momentum (following in the footsteps of the related “mobile-first” trend). Maybe those same principles could be taken further, from offline-first to offline-only.\n\nIn the end I pieced together a small offline-only page. Its logic is pretty simple:\n\ndownload all the content\ndon’t display the content; instead, prompt the user to disconnect\nwait for a disconnection, then display the content\nif the page is refreshed while disconnected, fetch a cached local copy\n\nThe first step is trivial. The initial request returns HTML that, in turn, requires some scripts. I used React because I’m familiar with it, but any framework could be used to achieve this (including no framework at all). The important detail is that the all of the page’s contents are loaded into memory, but rendered only when the user is offline.\n\nNext we have to detect if the user is offline. The navigator.onLine property is widely supported in modern browsers. But this is just a static value. In order to respond when the user goes offline, we’d either have to periodically check navigator.onLine, or listen for an event. Luckily browsers also provide \"offline\" and \"online\" events.\n\nIn React, this logic can be abstracted into Online and Offline components that only render their children when the browser is online or offline, respectively. I made react-detect-offline to do exactly this. (Check it out and let me know what you think!)\n\nimport { Online, Offline } from 'react-detect-offline';\n\nconst App = () => (\n
\n Only shown when you're online \n Only shown offline (surprise!) \n
\n);\njsCopy to clipboard\n\nThe last step is the most complex and the least obvious. If the user refreshes the page while disconnected, we’d like to load a local copy of the page rather than that try (and fail) to fetch a copy from the server. Enter service workers. Luckily, Create React App comes with full support for service workers right out of the box. I built Offline on top of Create React App and took advantage of service workers with zero extra effort.\n\nPlug all the pieces together and you have a strange little piece of the internet that makes you leave the internet. I hope you enjoyed it (or at least got a laugh)."
],
"categories": {
"primary": "frontend",
"others": [
"mobile"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-deploying-containerized-node-js-applications-with-microsoft-azure": {
"href": "https://nearform.com/insights/deploying-containerized-node-js-applications-with-microsoft-azure",
"postType": "blog",
"slug": "insights-deploying-containerized-node-js-applications-with-microsoft-azure",
"date": "2017-10-02",
"title": "Taking advantage of the new Web App for Containers service offered by Azure to deploy a containerized Node.js application",
"authors": [],
"content": [
"This is the fourth article in our Azure series. So far we have looked at deploying a simple Node.js application to the Azure platform, using Pino and Event Hubs to collect logs , and using Stream Analytics to gain real-time business intelligence .\n\ntextCopy to clipboard\n\nIn this post we are taking advantage of the new Web App for Containers service offered by Azure to deploy a containerized web application to the Azure cloud. We also take a look at deploying our own container registry on Azure and using webhooks to link up a basic continuous deployment pipeline. The figure below shows the deployment architecture we are aiming for.\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nA containerized deployment architecture using Microsoft Azure Web App for Containers.\n\ntextCopy to clipboard\ntextCopy to clipboard\nRequirements\ntextCopy to clipboard\n\nNote: We go into more detail about getting started with Azure in the first post in this series. If you have worked through that post, you will have already laid a lot of the groundwork for what follows. Otherwise, you will just need to follow these instructions.\n\ntextCopy to clipboard\n\nTo start with, you need a Microsoft account to access Azure. Your company administrator can create a login for you, but if you just want to just try out Azure you can sign up as an individual for a free trial at azure.microsoft.com/en-us/free/ .\n\ntextCopy to clipboard\n\nWith an account, you can set up, view and configure Azure products either via the web-based portal or via Azure CLI. We use the Azure CLI for this exercise. CLI installation instructions are available for macOS, Linux and Windows.\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThe Azure portal allows you to interact with, and create, Azure resources through a graphical user interface (GUI).\n\ntextCopy to clipboard\n\nFinally, we need an application to deploy. In the first post of this series, we discussed a sample application architecture using Hapi and Typescript , and we continue to use this as our canary application. You can drop in any Node.js or other application you like. Grab a copy of the sample application using git by running the following command:\n\ntextCopy to clipboard\ngit clone git@github.com:nearform/azure-typescript-hapi.git\ntextCopy to clipboard\ntextCopy to clipboard\ntextCopy to clipboard\nA Proof-Of-Concept Deployment\ntextCopy to clipboard\n\nBefore we do a deep dive, lets walk through a proof-of-concept deployment of a basic NGINX web server Docker image from Docker Hub. This lets us get used to the setup, whilst isolating any potential problems before trying a more complex setup.\n\ntextCopy to clipboard\n\nLog in to the Azure CLI using the following command, substituting your own login details:\n\ntextCopy to clipboard\naz login -u -p\ntextCopy to clipboard\ntextCopy to clipboard\nSetting up a deployment\ntextCopy to clipboard\n\nBefore you deploy an application, there are various organisational and administrative constructs to create. Compared with some other cloud providers this does feel a bit like additional overhead, but there is a case to be made for making explicit setup choices. Undoubtedly the structure is beneficial for more complex deployments. Fortunately, if you have followed through previous blog posts, or already work with Azure you already have most of these in place.\n\ntextCopy to clipboard\n1. Create a Deployment User\ntextCopy to clipboard\n\nThis is separate to your Microsoft account and can be shared with members of your team, CI bots and so on. Check if you already have one registered using the CLI with the following command:\n\ntextCopy to clipboard\naz webapp deployment user show\ntextCopy to clipboard\ntextCopy to clipboard\n\nOtherwise, create one by running:\n\ntextCopy to clipboard\naz webapp deployment user set --user-name --password\ntextCopy to clipboard\ntextCopy to clipboard\n2. Create a Resource Group\ntextCopy to clipboard\n\nCreate (or choose, if you already have one) a resource group. A resource group is a logical grouping of resources associated with a geographical region. You can find more information on the significance of resource groups here . Choose a name and location for your resource group (in this example we use West Europe).\n\ntextCopy to clipboard\n\nList existing resource groups with the following command:\n\ntextCopy to clipboard\naz group list\ntextCopy to clipboard\ntextCopy to clipboard\n\nIf there is not one already created, or you want to create a new one, run the following command:\n\ntextCopy to clipboard\naz group create --name --location \"West Europe\"\ntextCopy to clipboard\ntextCopy to clipboard\n\nTip: You can also register your resource group (as well as other settings encountered later) as the default resource group in the CLI using:\n\ntextCopy to clipboard\naz configure --defaults group=\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis saves you having to specify the group for lots of CLI commands. In our examples below we do not assume any defaults are set and we explicitly specify all settings.\n\ntextCopy to clipboard\n3. Create a Service Plan for your Resource Group\ntextCopy to clipboard\n\nCreate (or re-use) a service plan associated with this resource group. Service plans define the size and features, and therefore cost, of the server farm provisioned. In this example we use a Standard level plan ( --sku S1 ). Create a service plan by running the following command:\n\ntextCopy to clipboard\naz appservice plan create --name --resource-group --sku S1 --is-linux\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: Here we use the is-linux flag. We must have a Linux service plan to use containers and so we have to use at least a Basic level service plan.\n\ntextCopy to clipboard\nDeploy a Containerized Application\ntextCopy to clipboard\n\nNow that all of the infrastructure is in place, lets go ahead and deploy our application. Here we are going to use a simple hello world Docker image from Kitematic (kitematic/hello-world-nginx) which uses NGINX to serve a static site.\n\ntextCopy to clipboard\n\nRun the following command to deploy your application:\n\ntextCopy to clipboard\naz webapp create --resource-group --plan --name --deployment-container-image-name kitematic/hello-world-nginx\ntextCopy to clipboard\ntextCopy to clipboard\n\nNow run the browse command and the CLI opens a browser window showing our freshly minted application.\n\ntextCopy to clipboard\naz webapp browse --resource-group --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nYou should see something like the following:\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nTip: If you do not see the kitematic hello world page, try checking the logs for your webapp. Use the following command to view logs in your terminal:\n\ntextCopy to clipboard\naz webapp log tail --name --resource-group\ntextCopy to clipboard\ntextCopy to clipboard\n\nCongratulations! We have successfully deployed a containerized app to Azure Web Apps.\n\ntextCopy to clipboard\n\nWhilst this demonstrates a proof of concept, it is likely that you will have at least some container images that you do not want to host publicly on Docker Hub. Fortunately, Azure integrates with any Docker container registry. This allows you to choose where you store your container images, for example in a private cloud or on a dedicated CI/CD server. In the next section we take a look at setting up our own private container registry using Azure Container Registry .\n\ntextCopy to clipboard\ntextCopy to clipboard\nCreating a registry with Azure Container Registry\ntextCopy to clipboard\n\nIt is very quick to set up a private Docker registry with Azure Container Registry. We use the service level Managed_Basic as we want to use the webhook support this provides. To create a registry, choose a name and a resource group to associate it with. You can use the same resource group that we used in the previous section. A registry requires a storage account, but one will be created when you create a registry if you do not already have one.\n\ntextCopy to clipboard\n\nRun the following command to create a registry:\n\ntextCopy to clipboard\naz acr create --name --resource-group --sku Managed_Basic\ntextCopy to clipboard\ntextCopy to clipboard\n\nAzure automatically creates an admin account in a disabled state for your repository. Run the following command to enable it:\n\ntextCopy to clipboard\naz acr update -n --admin-enabled true\ntextCopy to clipboard\ntextCopy to clipboard\n\nYou can now access registry credentials with the command:\n\ntextCopy to clipboard\naz acr credential show -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nNow that your registry is set up, check its contents with the following command:\n\ntextCopy to clipboard\naz acr repository list -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nAt this stage you see an empty list, as we have not yet put any images into our registry. Our next task is to create a Docker image, and push it up to our registry, ready to deploy.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSetting up a Containerized Application\ntextCopy to clipboard\n\nWe use the same Node.js app that we used in the first post in this series, but here we will be using the docker branch. Grab a copy of the repository and checkout the docker branch by running:\n\ntextCopy to clipboard\ngit clone git@github.com:nearform/azure-typescript-hapi.git\ngit checkout docker\ntextCopy to clipboard\ntextCopy to clipboard\n\nThe main difference versus previous posts is that we have introduced a Dockerfile. We wont go into too much detail about how Docker works here, as it is out of scope. If you want to use your own Node.js application, and do not currently have a Dockerfile set up, here is a great how-to guide from the Node.js foundation . To make images you will need to have Docker running (you can find installation instructions here ).\n\ntextCopy to clipboard\n1. Log in to the Container Repository\ntextCopy to clipboard\n\nFirst you will need to log in to the container registry you set up in the previous section. The Azure CLI provides a convenience command for logging into Azure Container Registry registries for you. Simply run:\n\ntextCopy to clipboard\naz acr login -n\ntextCopy to clipboard\ntextCopy to clipboard\n2. Make an Image of the Application\ntextCopy to clipboard\n\nMake an image of the application to store in our repository. To do this cd to the application directory, and run the command:\n\ntextCopy to clipboard\ndocker build -t .\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis builds an image of the current directory using the applications Dockerfile, and tags it as image name provided. You can check that your image has built successfully by running the following command:\n\ntextCopy to clipboard\ndocker run -it --rm -p 3000:3000\ntextCopy to clipboard\ntextCopy to clipboard\n\nYou should see your application running on localhost:3000 . In the terminal window use Ctrl-C to terminate your application.\n\ntextCopy to clipboard\n\nNote: If you are not using the sample application, you will need to change the port number (3000) to whichever port your application exposes.\n\ntextCopy to clipboard\n3. Tag the Image\ntextCopy to clipboard\n\nTag this newly created image to associate it with our new remote repository on Azure. Use the following command:\n\ntextCopy to clipboard\ndocker tag .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: Azure Container Registry supports multi level namespaces. We could equally tag our image with .azurecr.io/my/new/image. This allows more complex hierarchical organisation of images.\n\ntextCopy to clipboard\n4. Push the Image to the Remote Repository\ntextCopy to clipboard\n\nRun the following command to push the image to the remote repository (this may take a while on slower connections):\n\ntextCopy to clipboard\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nCheck it is stored in our registry by running the command:\n\ntextCopy to clipboard\naz acr repository list -n\ntextCopy to clipboard\ntextCopy to clipboard\ntextCopy to clipboard\nDeploying Our Application\ntextCopy to clipboard\n\nAt this stage we have created a containerized app, and successfully uploaded it to our newly created container registry. The last thing to do is to deploy our app to our Azure Web App instance (which we created earlier). To do this we must change a few of the settings for our Web App instance to let it know where to find our new container registry.\n\ntextCopy to clipboard\n\nNote: From here we will switch to using the abbreviated options -g for resource group and -n for app name for brevity.\n\ntextCopy to clipboard\n1. Change Container Registry Settings\ntextCopy to clipboard\n\nChange the container registry settings for the Web App instance with the following command:\n\ntextCopy to clipboard\naz webapp config container set -g -n \\\\\n --docker-registry-server-url https://.azurecr.io \\\\\n --docker-custom-image-name .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: Here Azure automatically includes your registry credentials because we are using an Azure registry. If you are using a non-Azure private registry, you will also need to set your credentials with the --docker-registry-server-user and docker-registry-server-password options.\n\ntextCopy to clipboard\n\nTip: If you create a new web app at this stage, you will need to create it with dummy container data (via the --deployment-container-image-name or --runtime options) before you set up a custom container registry. This seems like an oversight in the CLI design. We raised this with Microsoft and they told us that theyre currently working on improving the CLI in this area, as well as better documentation around this feature, so watch this space!\n\ntextCopy to clipboard\n2. Assign a Port\ntextCopy to clipboard\n\nWe tell our Web App instance which port our application exposes, so that this port is exposed to external traffic by Docker. For our sample app, this is port 3000. Run the following command:\n\ntextCopy to clipboard\naz webapp config appsettings set -g -n --settings WEBSITES_PORT=3000\ntextCopy to clipboard\ntextCopy to clipboard\n\nNow with the settings changed, restart your application using the command:\n\ntextCopy to clipboard\naz webapp restart -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nCheck out your newly deployed app with the command:\n\ntextCopy to clipboard\naz webapp browse -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nEt Voila , your newly deployed container is available in your browser.\n\ntextCopy to clipboard\n\nTip: If your browser is showing an error message at this point, try running your Docker image on your local machine. Pull it from your repository with the command:\n\ntextCopy to clipboard\ndocker run -it --rm -p 3000:3000 .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYou can also inspect logs for your application with the command:\n\ntextCopy to clipboard\naz webapp log tail --name --resource-group\ntextCopy to clipboard\ntextCopy to clipboard\n\nTo recap, we carried out the following tasks:\n\ntextCopy to clipboard\nSet up a private container repository with Azure Container Registry,\nPushed our own container images to it,\nUsed those images to deploy our own Azure Web App.\ntextCopy to clipboard\n\nSo far so good! However, most projects these days look to deploy on an increasingly frequent basis, and it would be great to be able to automate this. Thankfully Azure supports various continuous deployment options. In the next section we will look at one in particular: using webhooks with our Azure Container Repository to automate new deployments of our application.\n\ntextCopy to clipboard\ntextCopy to clipboard\nContinuous Deployment with Azure Container Registry\ntextCopy to clipboard\n\nTo set up continuous deployment from our registry, we simply need to set up a push webhook to our Web App instance to let it know a new image is available. The first thing we do is enable the continuous deployment setting for our Web App instance with the command:\n\ntextCopy to clipboard\naz webapp deployment container config --enable-cd true -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThe return value from the above command gives a setting with the name CI_CD_URL which contains a long URI looking something like:\n\ntextCopy to clipboard\nhttps://$:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\ntextCopy to clipboard\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command.\n\ntextCopy to clipboard\n\nTip: Wrap your URI in single quotes as they tend to include a $ character.\n\ntextCopy to clipboard\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem.\n\ntextCopy to clipboard\n\nThe above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ntextCopy to clipboard\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\ntextCopy to clipboard\n\nYour updated app is deployed automatically.\n\ntextCopy to clipboard\n\nTip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\ntextCopy to clipboard\nCleanup\ntextCopy to clipboard\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\ntextCopy to clipboard\naz group delete --name\ntextCopy to clipboard\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\ntextCopy to clipboard\ntextCopy to clipboard\nSummary\ntextCopy to clipboard\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\ntextCopy to clipboard\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\ntextCopy to clipboard\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\ntextCopy to clipboard\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\ntextCopy to clipboard\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular.\n\ntextCopy to clipboard\n\nlt;app-name>:ciHitjfZQ3CyDmTYTraLQKLeeca1H5KwhxukHauG2Ts5yPotP0JTq4EAFJNN@.scm.azurewebsites.net/docker/hook\n\ntextCopy to clipboard\n\nThis is the inbound URI for webhooks and we need it to set up the webhook from our registry. You can always re-fetch this URI later with the command:\n\naz webapp deployment container show-cd-url -g -n\ntextCopy to clipboard\n\nWith this URI go ahead and create a webhook from your registry using the following command. Tip: Wrap your URI in single quotes as they tend to include a $ character.\n\naz acr webhook create -n -r --uri --actions push\ntextCopy to clipboard\n\nNote: If you try and create more webhooks than your plan allows, you might see an unhelpful error message along the lines of 'unicode' object has no attribute 'get'. Running the command again with the --debug flag gives more information on the underlying problem. The above command sets up a push action webhook, which fires when you push new images to your registry. You can read more on Docker webhooks here . Now that you have configured a webhook, any changes you push up to your registry are automatically released to your web app; we have a basic CD pipeline in place. Go ahead and test this by making some changes to your original Node.js app, and then rebuilding and pushing the image using the commands:\n\ndocker build -t .\ndocker tag .azurecr.io/\ndocker push .azurecr.io/\ntextCopy to clipboard\n\nYour updated app is deployed automatically. Tip: If you dont see changes deploying automatically, you can check your webhook is firing, and inspect request and response objects, in the Azure portal.\n\nCleanup\n\nWe have reached the end of our example. With any luck you will now have a containerized application of your choice deploying automatically to Azure. In order to tear down the infrastructure we have used for this example, we can simply delete the resource group that contains it. This will delete all resources associated with that group, and delete any service plans that now have no linked resources. Delete a resource group by running the following command:\n\naz group delete --name\ntextCopy to clipboard\n\nNote: If you are using an existing resource group, you can just delete the specific resources we have created via the Azure portal.\n\nSummary\n\nWe have successfully deployed a containerized application using Azure Web Apps for Containers, and set up a simple continuous deployment pipeline for that application. From this platform we can go on to make use of the other built-in features of Azure Web Apps, such as on-demand vertical and horizontal scaling, and performance testing. However these go beyond the scope of this post.\n\nWe have covered the basics of the Azure CLI commands for interacting with web application containers, and container registries. Full reference documentation is available here . Although we have chosen to use the CLI for this example, we could equally have set up this architecture using the GUI available at the Azure portal .\n\nSupporting custom container deployments is a big step forward for the Azure Web App product. For many projects it represents the sweet spot between a typical platform-as-a-service offering and wrestling with full blown container orchestration.\n\nAs the service has only recently achieved general availability, there are a few rough edges with the documentation and CLI. However our interactions with the Microsoft team make it clear theyre working hard to squash bugs and refine an already top-notch developer experience with Web Apps for Containers.\n\nFinally it's fantastic to see Microsofts continued commitment to open source software in general and Docker in particular."
],
"categories": {
"primary": "cloud",
"others": [
"backend",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-microsoft-and-nearform-announce-alliance-for-node-js-developer-migration-and-support-services-on-azure": {
"href": "https://nearform.com/insights/microsoft-and-nearform-announce-alliance-for-node-js-developer-migration-and-support-services-on-azure",
"postType": "blog",
"slug": "insights-microsoft-and-nearform-announce-alliance-for-node-js-developer-migration-and-support-services-on-azure",
"date": "2017-10-02",
"title": "We are announcing a new alliance to help customers migrate to using Node.js on Azure",
"authors": [],
"content": [
"At Node.js Interactive 2017, starting tomorrow in Vancouver, BC, nearForm and Microsoft are announcing a new alliance to help customers migrate Node.js apps to Azure, and provide enterprise-grade support for them . At NearForm we are bringing our multi-year expertise in architecting, designing and supporting Node.js apps to developers as they adopt Azure and build on top of Microsoft's cloud.\n\nEvery day, more and more Node.js developers choose to build and deploy on top of Azure, and Microsoft continues in its commitment to making investments in the area. The recent announcements of Azure App Service on Linux and Web App for Containers , combined with support for Node.js across Azure Functions , Azure Container Service and pre-provisioned VM images , provide developers a great range of choices to host their Node.js apps and services across several cloud compute models offered by Azure.\n\nNearForm Migration and Developer Support Services\n\nNearForm is the recognized leader in architecting and developing full-stack JavaScript solutions. With world-class teams of software architects, designers, developers, DevOps engineers and Open Source tooling experts, NearForm takes its clients worldwide through a current-architecture review workshop to a fully deployed high-performance new stack on Azure in a short space of time.\n\nAs part of the alliance, Microsoft and nearForm are announcing today the NearForm Migration and Developer Support services , which enables customers to confidently move to the cloud with Node.js on Azure. Beyond the migration of technology, NearForm can help businesses migrate to offering new SaaS capabilities : the NearForm design sprint is an intensive engagement which enables customers to generate a prototype of new business ideas in just five days. Customers can then work with NearForm's Solutions teams to turn the prototype into a production system.\n\nNearForm's Developer Support service ensures that teams have direct access to the most knowledgeable Node.js developers in the industry, including Node.js Core contributors. This level of expertise can guide customers step by step through the process.\n\nCustomers interested in the Migration and Developer Services can request a quote from NearForm directly."
],
"categories": {
"primary": "cloud",
"others": [
"backend",
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-adaptive-uis-with-react-image-palette": {
"href": "https://nearform.com/digital-community/adaptive-uis-with-react-image-palette",
"postType": "blog",
"slug": "digital-community-adaptive-uis-with-react-image-palette",
"date": "2017-10-09",
"title": "Adaptive UIs with react-image-palette",
"authors": [
"BRANDON DAIL"
],
"content": [
"When you have an application driven by rich media, like images, it can be hard to develop a cohesive, universal color scheme. A lot of the time, applications end up defaulting to shades of black and white to avoid clashing with the rich colors in their content. If you have only a few static images, you can work around this by just manually picking the best colors, but when the images are dynamic, it’s almost impossible to know which colors look the best. You can process the image on a server and store the palette in your database, but that means more data to process, store, and send to the client.\n\nThat’s why we’re excited to announce react-image-palette, an easy way to dynamically parse vibrant and accessible color palettes directly in your React application. react-image-palette will process an image using the Canvas API and then generate a color palette based on the most dominant colors, right in the client. No server processing, no data to store, just provide a URL to the image and react-image-palette will take care of the parsing. The colors in the palette are also guaranteed to meet the WCAG contrast standards so you never have to worry about readability. Check out the demo application to see what it can do."
],
"categories": {
"primary": "frontend",
"others": [
"design",
"a11y"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"digital-community-release-webpack-dashboard-and-electron-webpack-dashboard": {
"href": "https://nearform.com/digital-community/release-webpack-dashboard-and-electron-webpack-dashboard",
"postType": "blog",
"slug": "digital-community-release-webpack-dashboard-and-electron-webpack-dashboard",
"date": "2017-10-17",
"title": "Introducing Webpack Dashboard & Electron Webpack Dashboard 1.0",
"authors": [
"CARLOS KELLY",
"KEN WHEELER"
],
"content": [
"“Tang sucks” - Buzz Aldrin\n\nAfter releasing Webpack Dashboard and Electron Webpack Dashboard, we realized that we had hit a sweet spot for developer tooling. Developers wanted to be able to have more transparency and more visual feedback from their build processes (and they liked feeling like they worked at NASA, too). So we gathered our finest minds, opened our editors and got working on a stable release.\n\nThat release is finally here: stable 1.0 releases of Webpack Dashboard and Electron Webpack Dashboard. We have done some major bug fixing, added better documentation, and added a couple of new cool features. If you haven’t tried either before, get started by downloading Electron Webpack Dashboard, or learn how to get Webpack Dashboard running for your project.\n\nSo What’s New?\n\nThe biggest update to Webpack Dashboard is that it’s now fully compatible with Webpack 3. You can now see the sizes, assets, and any problems for each of your modules. It also still works with Webpack 2.\n\nOther fixes and features:\n\nInspectpack no longer runs in minimal mode, and the minimal mode box-sizing issue has been fixed to ensure that panels for status and progress always align.\nThe issue with the logs showing duplicate errors has been fixed.\nYou can now navigate through the main log using your keyboard arrows.\nWe’ve added support for custom hostnames via the plugin config object.\nWhen the NODE_ENV is production and doesn’t run Inspectpack, Webpack Dashboard now shows a warning instead of displaying errors.\n\nElectron Webpack Dashboard builds upon the Webpack Dashboard 1.0 Plugin and now has full compatibility with Webpack 2 and Webpack 3. We have also added better-looking scrollbars, a status bar for the NODE_ENV, and fixes for various visual glitches.\n\nSo What’s Next?\n\nAfter releasing the beta version of Electron Webpack Dashboard, we had a ton of requests asking us to add capability to run multiple instances for different projects. As a result, we’ll be adding support for multiple tabs and port configurations for macOS, and multiple window support for Linux and Windows.\n\nBeyond that, we intend to evolve alongside the Webpack project. We are excited to see what new features we can do cool things with and look into ways that we can provide even more value and reason to open up the dashboard window. Additionally, we are committed to providing data that’s as accurate as possible and will take care to not misrepresent asset/module sizes in our displays. This will be an iterative process, but because your dashboard updates automatically, it will just roll right out to you.\n\nIf you haven’t tried our dashboards yet, give them a spin. We’ve gotten feedback that they have helped companies reduce bundle sizes and unearth bundling issues, so you may reap similar benefits. Worst case scenario, you are just going to look cool AF."
],
"categories": {
"primary": "frontend",
"others": [
"devops"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-benchmarking-apache-kafka-deployed-on-openshift-with-helm": {
"href": "https://nearform.com/insights/benchmarking-apache-kafka-deployed-on-openshift-with-helm",
"postType": "blog",
"slug": "insights-benchmarking-apache-kafka-deployed-on-openshift-with-helm",
"date": "2017-10-19",
"title": "Running Apache Kafka on an OpenShift cluster with Helm and measuring the write throughput of the producer",
"authors": [],
"content": [
"Running Apache Kafka on Open Shift in a Production Environment\n\nWe have previously shown how to deploy OpenShift Origin on AWS. In this post, well walk you through installing Apache Kafka and Apache Zookeeper with Kubernetes Helm. We'll also produce some useful and valuable benchmarks like write throughput and inbound message rate. Our goal is to collect benchmark data to show that Apache Kafka can not only operate inside a container orchestration platform like OpenShift , but can be seriously considered as a valid solution for a production-ready environment.\n\nThis post assumes that you already have some knowledge of Apache Kafka, its use cases and its architecture components, the intro on the official Apache Kafka website is a good place to start if you don't.\n\nPrerequisites\nHelm\n\nWe use Helm to deploy applications on the cluster. Helm is a tool for managing Kubernetes charts. A chart is a set of preconfigured Kubernetes resources - it's a little like a package manager (e.g. apt/yum/Homebrew/etc) for Kubernetes. Helm allows us to have a reproducible, controlled and customisable installation of OpenShift resources in our Kubernetes cluster.\n\nThe reason why we chose Helm is because it's very powerful, it's actively supported by the community around Kubernetes, and it's bundled with several features that the internal OpenShift template mechanism doesn't have.\n\nThe Helm architecture is made up of two main parts:\n\nA client which runs locally.\nA server which runs in the cluster.\n\nYou need to first install Helm locally on your machine, see installation notes here . Once the client has been installed locally, it is time to install the server, also knowns as the tiller , in the cluster. When the user runs commands locally, they are delivered to the server to be executed.\n\nConfigure RBAC\n\nOpenshift Origin comes with Role Based Access Control (RBAC) enabled. It is used to configure policies and roles which allow access to the components of the cluster. We need to assign to the tiller the proper role in order to create/update/delete resources in the cluster.\n\nThe following instructions will install the tiller in the kube-system namespace. This is a namespace where only users with admin role have the privileges to operate. If we want to give our GitHub user access to the kube-system namespace, we need to connect to one of the master nodes via the bastion host and run: oadm policy add-cluster-role-to-user cluster-admin where username is your GitHub user name.\n\nThe username now has the cluster-admin role and can execute any operations against any namespaces in the cluster. Connect to command-line and copy to the clipboard the command: oc login https://openshift-master.os.nearform.net --token= Once executed on the local machine this will login the user to the cluster and it is then possible to execute operations with admin level via the oc command against the cluster.\n\nNext, create the serviceaccount tiller within the namespace kube-system : oc create serviceaccount tiller -n kube-system From the master node run: oadm policy add-cluster-role-to-user cluster-admin -z tiller -n kube-system Finally run locally: helm init --service-account tiller --tiller-namespace kube-system Doing so, the tiller will be installed in the kube-system namespace and it will have privileges to operate against all the namespaces in the entire cluster. Disclaimer : Note that with this approach tiller has been installed under the kube-system and it has cluster-admin privileges. It can execute all actions against every resource across the entire cluster. Running it could be potentially dangerous!\n\nAnother approach is to install tiller in a project that we already created. In this case we are by default admin of the project. We can then install tiller in the project and assign to it the admin role. In this way, it can create/read/update/delete resources only in that namespace.\n\nArchitecture\n\nIt is important to understand the general architecture of the Apache Kafka deployment.\n\nThe basic concepts in Apache Kafka are producers and consumers . Producers publish messages to topics, and consumers consume from topics. The brokers are Kafka nodes which store messages for the consumers, ready to be consumed at the rate of the consumer. The brokers run in StatefulSets .\n\nAs we need to persist Kafka message storage, it is necessary to enforce this by using proper Kubernetes resources in the cluster. By definition, pods deployed by using a Deployment object are ephemeral and they dont have any sort of identity. They can be destroyed and replaced without any issue. Different StatefulSets are resources that allow that every single pod maintain an unique, sticky identity and associate each of them to an unique different volume which stores the data of the application. We do this in OpenShift by using Stateful Resources which allows us to persist message state.\n\nIn the same pod but in a separate container, we run Prometheus JXM Exporter . This approach provides a high level of separation and encapsulation. It is a common microservices pattern called Sidecar . The exporter is composed of two main components:\n\na collector that configurably scrapes and exposes Apache Kafka mBeans of a JMX target\nan HTTP Server that receives the scraped metrics and exposes them via an API in a Prometheus compatible format\n\nWe can then consume these metrics from Prometheus. It is just required to set up the HTTP server address of the Prometheus JMX Exporter in the Prometheus configuration file. It is beyond the scope of this article to explain this, for more information please read the Prometheus documentation .\n\nThe choice of running Prometheus JMX Exporter to extract metrics has been done by taking into consideration different aspects such as the grade of integration of the current monitoring solutions in an environment like Kubernetes, the simplicity of deploying, configuration interface and, last but not least, support from the community. Prometheus JMX Exporter allows us to define which metrics we want to expose and track them down in Prometheus with the added advantage that the latter comes out of the box with great support for Kubernetes internal cluster metrics.\n\nThe broker message stores are backed by external network volumes, this gives us better read and write performance for Kafka, as well as data resiliency. We'll talk about volumes in the next section.\n\nFinally, no Apache Kafka deployment is complete without ZooKeeper . ZooKeeper is the glue that holds it all together, and is responsible for:\n\nelecting a controller (Apache Kafka broker that manages partition leaders)\nrecording cluster membership\ntopic configuration\nquotas (0.9+)\nACLs (0.9+)\nconsumer group membership (removed in 0.9+)\nDeployment of Apache Kafka\n\nThe installation of Apache Kafka and Zookeeper can now be executed by using Helm. We are going to provision the Apache Kafka cluster in a new OpenShift project.\n\nLets create a new project: oc new-project kafka --display-name=\"Apache Kakfa\" --description=\"Apache Kafka cluster using Stateful Sets and Zookeper\" This will create a new project and consequently a new namespace.\n\nBefore launching the installation, it is necessary to set some important properties in the cluster settings. For limiting and securing the cluster, OpenShift doesnt allow running containers which have low uid and guid numbers. It is needed to set the Security Context constraints (SCCs) in this way we can run Zookeeper as privileged containers.\n\nRun the following from the master machine: oc edit scc anyuid and edit the file by adding at the end:\n\nusers:\n- system:serviceaccount:kafka:default\ntextCopy to clipboard\n\nThe service account default can now run containers under the kafka project/namespace with any uid.\n\nLast step before spinning up the installation, is to set up the wanted values in the Helm configuration. The file kafka.yaml is under the values folder and it defines the configuration used in our scenario. The values defined there are more than enough to spin up a fully working cluster. Under the route property it is necessary to setup the application_domain url in the form One common mistake is to skip using the Post-it and write directly on the whiteboard. This approach quickly reveals its limits when you need to start moving things around.”\n\n2. Express your ideas\n\nWrite every idea you can come up with on individual Post-it notes. Their size forces you to be concise and straightforward. Construct your ideas following this simple principle: Verb + Short expression (e.g. Review Mark's presentation).\n\n“If it doesnt fit on a note, well, it probably means that your idea is too big. In that case, divide it into smaller ideas. ”\n\nThe more you produce, the merrier.\n\n3. Sort your ideas\n\nOnce you've written all of your ideas down on Post-it notes it is time to organise theme by themes. There are many ways to approach this but Antoine points out 4 in his article.\n\nA. The Eisenhower Principle\n\nSelect two criteria for the purposes of evaluation. For his example Antoine uses Urgent and Important. Divide the whiteboard into 4 sections by drawing two axes, one with an arrow pointing up and another with an arrow pointing to the right, and labeling each with one of your evaluation criteria. The most pressing ideas should be placed in the top right quadrant, while less pressing items should be placed appropriately in the bottom right or top left quadrant. The least pressing items should be placed in the bottom left quadrant.\n\nEisenhower Principle inspired by the late U.S. President.\n\nB. Effort/Impact\n\nMuch like the Eisenhower principle the Effort/Impact approach also divides the whiteboard into 4 quadrants. Draw two axes one with an arrow pointing up labeled Impact and another with an arrow pointing to the right labeled effort. Here the quadrants are different than the previous example. The top left quadrant now should contain the quick wins, while the bottom left and top right quadrants contain less pressing items. The bottom right quadrant now should house the least pressing items as they require the most effort and offer the least amount of impact.\n\nMid to long-term strategic matrix.\n\nC. Pros and cons\n\nExactly what it sounds like. Draw one line down the middle of the whiteboard and write Pros to the left and Cons to the right of the line. Arrange your ideas accordingly.\n\nPros and Cons. A classic. An alternate take on the Pros and Cons method is the Nice to have/Must have model which can help you priortize your ideas.\n\nD. Kanban\n\nThe Kanban method distributes your ideas or tasks according to their current state of completion: not started, in progress, blocked, done. This works well when updated daily.\n\nKanban (literally billboard in Japanese) are often used in Software development.\n\nConclusion\n\nAlthough these methods for the decision making process may be simple it doesn't mean they are simplistic. Sometimes the simplest methods can help a team crack the most complex problems or tackle tough decisions. Use these methods individually or in combination with each other based on your needs.\n\nIf you can master these simple methods and incorporate them into your workflow you will make smarter decisions.\n\nLearn more about design at NearForm."
],
"categories": {
"primary": "product",
"others": [
"design"
]
},
"verticals": {
"primary": "none",
"others": []
}
},
"insights-exploring-react-portals": {
"href": "https://nearform.com/insights/exploring-react-portals",
"postType": "blog",
"slug": "insights-exploring-react-portals",
"date": "2017-10-31",
"title": "A dive into React Portals and exploring some common use cases",
"authors": [],
"content": [
"Portals receive first-class official support in React\n\nWith the recent release of React v16 , there are a number of exciting new features. One of these new features is called Portals . Portals have been a concept in the React community for quite some time 1 and have gained first-class official support. The React docs define a Portal as > Portals provide a first-class way to render children into a DOM node that exists outside the DOM hierarchy of the parent component. To best illustrate why this might be useful and necessary, let's take a look at two examples: - Modals - Legacy Applications\n\nModals\n\nA modal is a UI element that overlays on top of the main window of an application.\n\nLet's build an application with a simple modal 2 to illustrate how this works:\n\nTo start, create a new application using create-react-app . If you have never used create-react-app before, run this command to install:\n\nnpm install -g create-react-app\n\n\nand then:\n\nimport React, { Component } from 'react'\nimport './Modal.css'\n\nclass Modal extends Component {\n render() {\n return this.props.open ? (\n \n
\n
\n
\n {this.props.header} \n this.props.onClose()}\n type=\"button\"\n aria-label=\"close\"\n >\n CLOSE\n \n \n
{this.props.children}
\n
\n
\n
\n Welcome to React \n \n
\n \n this.setState({\n showModal: !showModal\n })}\n >\n Show Modal\n \n\n \n this.setState({\n showModal: false\n })}\n >\n Some Content \n \n
\n
\n
\n Welcome to React \n \n
\n this.setState({ showSidebar: !showSidebar })}\n >\n Toggle Sidebar\n \n
\n
\n \n this.setState({\n showModal: !showModal\n })}\n >\n Show Modal\n \n\n \n this.setState({\n showModal: false\n })}\n >\n Some Content \n \n
\n
\n \n \n
\nhtmlCopy to clipboard\n\nAdding the transform property to the sidebar changed our fixed position from the viewport to the sidebar container! This means the styling of the
\n\nhtmlCopy to clipboard\n\nNext, let's create a new file called src/Portal.js :\n\nimport React from 'react'\nimport ReactDOM from 'react-dom'\n\nimport Modal from './Modal'\n\nclass Portal extends React.Component {\n constructor(props) {\n super(props)\n\n this.rootSelector = document.getElementById('root-modal')\n this.container = document.createElement('div')\n }\n\n componentDidMount() {\n this.rootSelector.appendChild(this.container)\n }\n\n componentWillUnmount() {\n this.rootSelector.removeChild(this.container)\n }\n\n render() {\n return ReactDOM.createPortal(\n
\n Welcome to React \n \n
\n this.setState({ showSidebar: !showSidebar })}\n >\n Toggle Sidebar\n \n
\n
\n
\n this.setState({\n showModal: !showModal\n })}\n >\n Show Modal\n \n\n \n this.setState({\n showPortal: !showPortal\n })}\n >\n Show Portal\n \n\n \n this.setState({\n showModal: false\n })}\n >\n Some Content \n \n\n \n this.setState({\n showPortal: false\n })}\n >\n Some Portal Content \n \n \n
\n...App Content\n
\n\n\n
\n
\n
\n
\n
\n My Portal Modal \n\t\t CLOSE \n \n
\n
Some Portal Content \n \n
\n
\n
\n
component that will display in the header.\nA customer support component that opens a window in the bottom right corner of the viewport.\n\nThe component will need to render to the top header and the component will need to render to a right before the closing