[{"post_id":7448,"post_title":"Algolia + SeaUrchin.IO","post_date":1521007207,"post_date_formatted":"Mar 14th 2018","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-acquires-seaurchin-io\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/2018-03_SeaUrchin-01-360x200.png","time_to_read":2,"content":"Building our Next Wave of Analytics\u00a0\nWe\u2019re introducing an exciting addition to our product family and rolling out new analytics solutions for our customers \nI\u2019m thrilled to announce today our first acquisition: SeaUrchin.IO \u2014 a real-time analytics platform that zeroes in on search insights. This is a key milestone on our path to help our customers improve their user engagement. We\u2019ve been focused on identifying new opportunities for them to create more intuitive, relevant and rewarding experiences. Today\u2019s announcements bring us one step closer on that journey.\nWhy SeaUrchin.IO?\nWe first came across the SeaUrchin.IO team as admirers of their technology. They built a unique platform that surfaced granular insights about how users were engaging with search. They were targeting the exact need we were trying to solve for our customers. We quickly saw that together, we could accomplish so much more for our customers\u2026 and accomplish it faster!\nThanks to the work we\u2019ve already been doing to integrate SeaUrchin\u2019s technology, the acquisition has enabled us to immediately accelerate the development of our analytics solutions. Read more about our new Analytics here.\nWe\u2019re just getting started\nWe\u2019ve had a quite a year of milestones \u2014 we crossed the 40 billion mark for search queries processed monthly, doubled our revenue, team and customers. But as excited as I am about what the team has accomplished, I\u2019m even more excited about the future. Bringing new technology to our team gives us the ability to innovate faster and bring new solutions to our community.\nWe\u2019re on a mission to give product builders the tools to create amazing experiences, so stay tuned for what\u2019s next!","record_index":0},{"post_id":7439,"post_title":"Search Analytics: Gain Insights from User Search Data","post_date":1521007105,"post_date_formatted":"Mar 14th 2018","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supercharging-search-analytics\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/01-2018_Analytics-V2-release-01-360x200.png","time_to_read":1,"content":"By analyzing what your users search for, you can understand what they want and which keywords they use. By looking at how they interact with the search results, you can understand how well your service answers their aspirations.\nMost importantly, through search, you can get invaluable feedback about your users\u2019 intent: feedback that accelerates their digital journey, and therefore your revenue. But that\u2019s only true if you\u2019re actively using the data that they\u2019re giving you through search \u2014if you collect, analyze, and surface this data.\nOur new Analytics\nFor the first time, our Analytics feature will track the search flow in its entirety \u2014\u00a0from beginning to end. We\u2019re not only analyzing the queries that are typed and the filters used, but also the results that are retrieved and how users interact with them. Do people click on the results list? Do they add a product to their cart, or read the article, or watch the video?\nToday, we\u2019re releasing a new Analytics functionality, available via API as well as the newly updated front-end dashboard. It is designed to track click and conversion events following a search.\nWhat\u2019s new \nAnalytics API \nSearches.\u00a0\u00a0Improve visibility into queries and keywords users type, as well as queries that return zero results. Set up Query Rules for one-off problematic queries \u2014\u00a0or adjust your searchable attributes to address systemic issues. \nFilters. Create faster paths to conversion by predefining filters on specific keywords based on the most popular filters users are selecting when searching.\nResults. Improve relevance on every query by using insights into results users see when performing a search. Identify and analyze low quality search results and find problematic queries such as common typos, obscure queries and query reformulations in order to improve content and relevance.\nClick Analytics API\nClicks. Gain insights into what results users are clicking on, what position those results appear in, and the average","record_index":0},{"post_id":7439,"post_title":"Search Analytics: Gain Insights from User Search Data","post_date":1521007105,"post_date_formatted":"Mar 14th 2018","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supercharging-search-analytics\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/01-2018_Analytics-V2-release-01-360x200.png","time_to_read":1,"content":"\u2026 click position of specific search queries\u00a0\nConversions.\u00a0Join search and conversion data in order to understand which search results are converting, and improve search queries that lead to users abandoning \nWhy it\u2019s useful \n\nThere are two main ways search analytics provide value: by giving you actionable business insights, and by helping you improve the relevance of your search engine.\n\nBusiness insights\nBy analyzing the queries that your users are typing, you can get insights that will be useful across your business:\n\n\nImprove your catalog of product or content. For example, an e-commerce platform could realize that a specific product is often asked by customers and that they should add it to their catalogue.\u00a0 A news outlet can learn more about the topics that their audience wants to read more about.\nImprove SEO. If your users are searching for certain words on your website, chances are these words are also the ones they type into Google.\nFuel your merchandising and marketing initiatives. Noticing the popularity of thematic words like \u201chealthy\u201d, \u201ccheap\u201d, or \u201crevolutionary\u201d can drive your future campaigns.\n\nImprove the relevance of the search\n\nAnalytics surfaces easy paths to improving your search relevance:\n\n\nThe query \u201ccouch\u201d returns no result? Perhaps you\u2019re missing the \u201csofa\u201d synonym for \u201ccouch\u201d.\n\nThe Click Rate on the query \u201ctomato\u201d is lower than you\u2019d like? Perhaps your relevance configuration retrieves \u201ctomato soup\u201d higher than actual tomatoes, and you need to improve your relevance strategy.\n\nThe articles most surfaced by your search don't correspond to your best articles? You could update your ranking to showcase these more.\n\nIt\u2019s only the beginning!\nEver since we've started Algolia, our goal has been to focus on making the best possible search engine, and our new Analytics feature is another step on that path. It adds a new dimension to the strong foundation of our engine: textual and business relevance, speed and","record_index":1},{"post_id":7439,"post_title":"Search Analytics: Gain Insights from User Search Data","post_date":1521007105,"post_date_formatted":"Mar 14th 2018","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supercharging-search-analytics\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/01-2018_Analytics-V2-release-01-360x200.png","time_to_read":1,"content":"\u2026 reliability. \n\nWe can\u2019t wait to see how you use Analytics to make meaningful improvements to your search experience and to your business. And we certainly plan to guide you along the way and help you make the most of the insights surfaced by your analytics. \n\nJoin our webinar on April 5 to discover how Search Analytics help drive conversion rates. Also, check out the following resources or contact us for a demo:\n\nInteractive tour\nAnalytics documentation\nClick Analytics documentation\n\nDon\u2019t hesitate to get in touch and share your questions and feedback: shoot us an\u00a0email, tweet to us, comment on this post.","record_index":2},{"post_id":7420,"post_title":"Mobile Search UX \u2013 Part Three: Displaying the Hard Work","post_date":1520955350,"post_date_formatted":"Mar 13th 2018","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-three-seach-results-display\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/03-2018_Search-Mobile-UX-1-360x200.png","time_to_read":6,"content":"In part 1, we\u2019ve depicted the many obstacles around search mobile UX. In part 2, we took a look at the anatomy of mobile search. We tackled the search bar, the search screen, and the search result screen. In this last part of the series, we will focus on the actual results page, specifically looking at what it needs to look like when building a great search experience for your mobile users.\nDesign: real estate is key\nDisplay results the right way\nWhen displaying search results, the end user should be able to pick the results that are right for her. It is therefore important to visually display the information in a way that will resonate with her, while making efficient use of the limited screen space.\nStart by understanding how your users browse your catalog. For items that they select visually \u2014\u00a0 like shoes or clothes \u2014\u00a0you could present search results as enlarged pictures.\n\nOn the other hand, price, distance and ratings can be used to display results for entries that are heavy on specifications, like restaurants.\n\nWhile it is possible to use a mix of both, understanding how your users search will guide you in choosing the right search interface.\nShowing feedback when filtering\nProviding filtering options is omnipresent in advanced search experiences. The issue at hand is mobile screen real estate since it is hard to include both filters and search results in the same view without the screen getting cluttered. Sometimes, we are able to squeeze a few important filtering buttons in a top bar:\n\nBut most of the time, we\u2019ll have to show a dedicated view with all possible filters.\u00a0Good filtering user experiences have one thing in common: they show some sort of feedback when the user applies a filter. For example, Amazon displays a panel that occupies 2\/3 of the screen width. When a filter is applied, the background search result view will update on the fly to showcase the new results:\n\n \nAnother example is the Airbnb app, where there is a button at the","record_index":0},{"post_id":7420,"post_title":"Mobile Search UX \u2013 Part Three: Displaying the Hard Work","post_date":1520955350,"post_date_formatted":"Mar 13th 2018","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-three-seach-results-display\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/03-2018_Search-Mobile-UX-1-360x200.png","time_to_read":6,"content":"\u2026 end of the screen displaying the number of results every time a filter changes.\n\nShow relevant results\nRelevance is of utmost importance on mobile because of its tiny screen which allows for a very limited number of results to be visible. Without relevance, all best practices are for nothing.\nWhile on the desktop you may afford a luxury of not having the most relevant result at the top \u2014\u00a0since the user will eventually find it \u2014\u00a0this is not the case on mobile. Because of the limited real estate, only the top search results are visible to the user, so they\u2019d better be relevant.\n\n \nRelevance is best evaluated by taking into considerations both textual and business ranking rules; you can learn more about it here. \nUsability: respect user's effort\nRecognize the typos\nIt is easy for the user to do typos. This is even more true when typing on mobile due to the limited space between keys. It is therefore important for your search results to have some sort of a typo tolerant mechanism. Anticipating user\u2019s intent and correcting their mistakes will substantially reduce their frustrations.\n\nLet them scan the results\nIn general, people don\u2019t like to read everything that is on the screen, especially when they are browsing vs. knowing exactly what they\u2019re looking for. What they prefer to do is scan for information. It is thus our duty to help them achieve this, and what better way to do this than highlighting.\nHighlighting helps the user understand why they are obtaining specific results or suggestions. \u00a0There are two effective uses of highlighting.\nThe first one is a standard highlighting that matches the query exactly. This is great for the user to understand why her query matched a result, and is best used when showing instant results.\n\nThe second one is inverted highlighting, which highlights everything but the search query. A great scenario for this is query suggestions. In there, the user can easily identify the differences between the different","record_index":1},{"post_id":7420,"post_title":"Mobile Search UX \u2013 Part Three: Displaying the Hard Work","post_date":1520955350,"post_date_formatted":"Mar 13th 2018","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-three-seach-results-display\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/03\/03-2018_Search-Mobile-UX-1-360x200.png","time_to_read":6,"content":"\u2026 suggestions.\n\nEnvironment: speed and connectivity\nThis is an obvious one, but we have to mention it since a good search experience has to meet user\u2019s expectations of speed. Google has set the bar high, and apps must follow.\nOn mobile, users are on the go, often with bad connectivity. The search experience has to react well to these circumstances in the best possible way in order to ensure a smooth user experience.\n\nWhen connectivity is slow, a good idea is to show some progress indicator: spinning wheels, loading icons or progress bars. Another way to manage user\u2019s expectations is to show a skeleton screen with placeholders, as shown in the screenshot above. This gives the user a sense of progress and feedback. A final way to improve the search experience is to implement lazy loading, where you would give priorities to certain types of content over others, and fetch them separately.\nNevertheless, the dreaded case of no network availability will surface at some point to the user, and this needs to be handled properly, beyond just showing a \u201cretry\u201d button. One straightforward way to solve this is to cache the top results offline, and then offer a basic local search experience. A better way is to provide a light full-fledged search engine on the device that is capable of doing most of what the online engine would do. However, this is not for the faint of heart, and requires a lot of skills, resources and time, as we learned building it with Algolia. \nWith this series, we tried to share everything we learned through many years of building search user experiences on mobile. Let us know if we missed anything or if you have any questions or feedback: @guydaher, @algolia\nThis blog post relies heavily on the research done by Lucas Cerdan, Product Manager at Algolia. Big thanks, Lucas!","record_index":2},{"post_id":7391,"post_title":"Bringing Search to Angular","post_date":1519807918,"post_date_formatted":"Feb 28th 2018","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-search-to-angular\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/2018-02_Angular-IS_release-1-360x200.png","time_to_read":4,"content":"Today, we are excited to launch Angular InstantSearch \u2014 our newest library for easily creating search UIs in Angular applications.\nOur mission at Algolia is to make sure any website or application has the best search experience. But search is its own domain of expertise, with specifics concepts, specialized coding skills and UX\/UI best practices. This knowledge is hard to gain and as search is only one part of a project, that's why we want to make sure any developer can easily create a search UI even when the rest of the search puzzle is solved by someone else.\nFrameworks and search UI\nIn 2015 we released our first solution to provide a search UI\/UX library: InstantSearch.js. The goal of InstantSearch.js was to let developers build their search UI faster by providing a set of ready-to-use UI widgets. Since the web framework war was roaring back then, we built a generic (Vanilla JavaScript) solution instead of one tailored to a specific framework.\nBut soon it became clear that we should go a step further if we wanted to provide the best developer experience. When using a specific web framework, a developer expects a certain kind of an API that we can only provide with dedicated InstantSearch versions. That\u2019s why in 2016 we started releasing other \u201cflavors\u201d of InstantSearch like React InstantSearch and Vue InstantSearch.\nAfter several requests from our users, we thought that Angular developers deserved a dedicated InstantSearch library. This InstantSearch flavor, like all our products, has been built in collaboration with our users. We contacted all our current and potential users interested in Algolia + Angular and offered them to join a private chat where they can continually give us feedback on the library. Based on this, we built a first beta; today, Angular InstantSearch is already used in production with great success.\nLet's look at some of the main features of the library.\nBuilt for Angular 4 and 5\nAngular InstantSearch is compatible with Angular 4 and","record_index":0},{"post_id":7391,"post_title":"Bringing Search to Angular","post_date":1519807918,"post_date_formatted":"Feb 28th 2018","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-search-to-angular\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/2018-02_Angular-IS_release-1-360x200.png","time_to_read":4,"content":"\u2026 5 which means that, even if you are still on Angular 4, you can use it and be sure it will work when you upgrade to Angular 5. We are not compatible with Angular 1 and 2 because the changes between Angular 2 and Angular 4 or 5 were too broad. We also thought it was better to provide great features (like server side rendering) to the latest instead of being blocked by older version incompatibility.\n20 ready-to-use widgets\nFrom a good search box to a proper pagination system, we got you covered. This release includes all the widgets you need to build a complete search UI. The widgets can be completely customized, from the DOM they generate to their style and behavior.\n\n<Declarative \/> API\nWith Angular InstantSearch, widgets can be used easily with Angular directives. It's just a matter of writing some HTML-like code to obtain a working search:\nView the code on Gist.\nAll the options are attributes to apply on the widgets directives. Building a search UI can now be achieved in a few minutes while still remaining highly customizable.\n\nCustomization API\nWhile we identified the most common use cases and best practices when designing InstantSearch widgets, some needs just can't be achieved with the sole use of options. That's OK.\nAngular InstantSearch provides an API letting you choose the rendering you want without having to rewrite all the search business logic code. For example, transforming the menu widget that by default renders as a list into a dropdown menu becomes simple.\nFirst-class SPA support\nSingle page applications are able to deliver performant and dynamic experiences to their users by loading and displaying only relevant elements of the UI based on user needs and actions.\nWith Angular InstantSearch, widgets are nothing more than regular DOM elements. They can be dynamically added or removed from the page. For example, you could decide to display only certain type of filters depending of the current query.\nServer-side rendering, of course\nAngular 5 comes","record_index":1},{"post_id":7391,"post_title":"Bringing Search to Angular","post_date":1519807918,"post_date_formatted":"Feb 28th 2018","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-search-to-angular\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/2018-02_Angular-IS_release-1-360x200.png","time_to_read":4,"content":"\u2026 with nice utilities that ease the creation of server-side rendered applications.\nCreating a server-side rendered app is a must-have to improve SEO score on search engines by facilitating web crawlers, especially if you're running an e-commerce business. It's also a good way to speed up the first page display for users.\nThat's why we developed Angular InstantSearch with service-side rendering available from day one.\nTry it out\nTry Angular InstantSearch! We have a getting started tutorial, and also built some demos so you can quickly see how it looks and behaves.\nThis is the first release of Angular InstantSearch, based on your feedback we will enrich it in the coming weeks and months.\nWhat can we do better? What's missing? Reply in comments here, reach out on our community forum or tweet us!\nYou can also comment on Product Hunt and Hacker News.\nCredits\nThis release would not have been possible without some good team work:\n\nMaxime Janton, our Angular champion, coded 90% of Angular InstantSearch\nMarie Thuret, our product manager, ensured that the release is on time and meets our InstantSearch feature requirements\nMatthieu Blandineau, our product marketing manager, made sure we reach every possible user with the news of the release\nThe whole InstantSearch team was here to support Maxime with code and discussions\nS\u00e9bastien Navizet,\u00a0Alexandra Prokhorova and\u00a0Tiphaine Gillet designed the Angular InstantSearch logo and website and\u00a0made the illustrations on this blog post and Product Hunt\nLucas Bonomi took ^ that design and transformed it to HTML and CSS compatible with desktop and mobile\nIvana Ivanovic\u00a0helped us write this blog post\n\nWanna be a part of our next release? We're hiring!","record_index":2},{"post_id":7369,"post_title":"What Can't React Do?","post_date":1519298927,"post_date_formatted":"Feb 22nd 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/what-cant-react-do\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Search-Party-sandbox-01-360x200.png","time_to_read":4,"content":"Algolia Search Party is a monthly Meetup we organize around a common theme that makes fellow devs\u2019 lives easier. Topics have included Laravel, static websites and, this month, React and some cool things you can do with it.\nSince the talks were useful and well received, we are sharing summaries and videos with you.\nCodeSandbox: your IDE in a browser\nIves van Hoorne flew in from the Netherlands to share his work on CodeSandbox. The idea for CodeSandbox came when Yves was trying to help some friends with code, but had no easy way to share projects. Because mentally parsing and running code in their heads was cumbersome and error-prone, Ives decided to invent a collaboration tool.\n\nCodeSandbox UI is a split view. On the one side you have your code, and on the other you have a live preview of the code. Whenever you update the code, the preview automatically reloads. The nice thing about this view is that you can load a full project, require any `npm` dependency and run it directly in the browser.\n\nTechnically, CodeSandbox is written in React and is made of two different apps. One is for handling the code editor, the other is for running the preview. Both apps are running on different subdomains so a live preview cannot hijack your CodeSandbox credentials. Whenever code is updated in the editor, CodeSandbox will compile\/transpile it and send it to the live preview for execution. Any errors generated are then sent back and displayed in a more human-friendly way. As Ives put it: \u201cHumans will be reading the code and humans are visually focused.\u201d UI is just one way that CodeSandbox is improving the developer experience.\nAn example: the Webpack visualization UI built on top of webpack-dashboard. It gives you a clear visual representation of your bundles and their respective sizes. CodeSandbox also goes further than displaying what the error is; it also explains what went wrong and how to fix it. For simple errors like a missing dependency, it even provides a one-click","record_index":0},{"post_id":7369,"post_title":"What Can't React Do?","post_date":1519298927,"post_date_formatted":"Feb 22nd 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/what-cant-react-do\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Search-Party-sandbox-01-360x200.png","time_to_read":4,"content":"\u2026 button to add the dependency.\n\nAlgolia plays a role in the quest to provide the best DX, as we provide the search into all the `npm` packages via the index we built and maintain for Yarn (available for anyone to use). CodeSandbox uses it for discovery of packages as well as autocomplete when adding a `require`. We can\u2019t help but quote Yves: \"I've seen search. Algolia is not search, it's instant results\". Because of that speed, Algolia can be used directly in the UI, and it took Yves only one day to integrate it.\nIves finished his talk with more ideas about how he could use Algolia to improve the DX of CodeSandbox\u2014further including the re-use of the indices in DocSearch to provide contextual documentation help directly from the editor. This would remove one more step for users and allow them to stay in the zone for longer. Another idea would be to improve the error display by automatically searching for the error and displaying the documentation about it. We love the ideas and will work with Ives to make them happen!\nCodeSandbox is open source and the code is available at https:\/\/github.com\/CompuIves\/codesandbox-client.\nLet's code Redux in 15 minutes\nAurore Malherbes, a mobile developer specialized in React Native presented an exercise she does with any new member joining her team: recoding Redux from scratch to understand how it works.\nAurore\u2019s talk was inspired by Dan Abramov\u2019s Redux course on egghead.io. Redux itself is not hard, but to use it correctly one needs to actually understand it \u2014\u00a0and the best way to understand it is to build it.\nAurore\u2019s talk was clear,explaining step-by-step in 15 minutes how the state, subscribe and dispatch all fit together to form the Redux event loop. I won\u2019t be repeating everything Aurore explained in her talk and would instead suggest watching the video with explanations so clear it would do them injustice repeating them \u201con paper\u201d.\n\nSearch UX on React Native\nLast but not the least, our own Marie Thuret\u00a0talked","record_index":1},{"post_id":7369,"post_title":"What Can't React Do?","post_date":1519298927,"post_date_formatted":"Feb 22nd 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/what-cant-react-do\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Search-Party-sandbox-01-360x200.png","time_to_read":4,"content":"\u2026 about unique challenges mobile search presents: screen size, connection speed, low connectivity and more, as well as how to overcome them with our React InstantSearch library\n\n \nInstantSearch packages a set of UI components on top of the Algolia API, that include all the best practices you see above. All those components are open source and extensible, letting you add your own custom logic.\nMarie ended her talk with a live demo of implementing a mobile search using our React-flavored version of InstantSearch. Step by step, from the search bar to the infinite scrolling of results, including the highlight. In less than 5 minutes, you too can have a working search. Check it out!\n\nSpecial thanks \u2026\n...to our speakers and the audience; both made the evening super enjoyable and informative. We hope to see you at one of our community events next time, and meanwhile, stay tuned to this space for interesting talk summaries and more.","record_index":2},{"post_id":7351,"post_title":"Travis Encrypted Variables and External Contributions","post_date":1518511390,"post_date_formatted":"Feb 13th 2018","author_name":"Julien Bourdeau","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6163f2a1af373d957715bcebee188ba3?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/travis-encrypted-variables-external-contributions\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Travis-community-PR-360x200.png","time_to_read":7,"content":"In this article, I\u2019ll explain how we handle community pull requests while avoiding storing Algolia credentials in Travis's secured environment variables. It requires a bit of time, but for a SaaS company maintaining open source projects, we believe it's worth it.\nFirst, it's important to understand that the only way to have a green build for community pull request is to make the credentials publicly available. It sounds like a bad idea, but I will show you how to do it safely.\nNote that this post uses Travis CI as an example because it is what we primarily use at Algolia. The problem and solution are also valid with other build systems, like CircleCi for instance.\nThe problem\nBefore we start, let's summarize the issue we want to solve.\nAlgolia maintains more than 10 open source API clients and most of them run tests against our API. It requires credentials to be able to contact the service, so following Travis's best practices, we store them as encrypted environment variables in our .travis.yml. To test our API clients, we use Algolia API keys, which are not just read-only but will also be used to test insertions in an Algolia index. Thus they can't be made available outside of our own builds (otherwise people might use the keys to create a lot of objects and indexes in our API).\nThe way encrypted environment variables works is that you use the travis gem to add encrypted variables to your .travis.yml. Those keys are then automatically decoded by Travis at build time, but only for builds triggered by contributors with write access to the repository \u2014 not for forks, meaning not for external contributors\u2019 PRs.\nTravis tried to solve this issue by providing a JWT addon that some companies (like Sauce Labs) implemented, but recently, they took a step back and will deprecate the addon in April 2018.\nPending PRs on the Symfony bundle\u00a0\nThe screenshot above illustrates the problem: maintainers' PRs are green and community PRs are red \u2014not because they're doing","record_index":0},{"post_id":7351,"post_title":"Travis Encrypted Variables and External Contributions","post_date":1518511390,"post_date_formatted":"Feb 13th 2018","author_name":"Julien Bourdeau","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6163f2a1af373d957715bcebee188ba3?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/travis-encrypted-variables-external-contributions\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Travis-community-PR-360x200.png","time_to_read":7,"content":"\u2026 something bad, but because in this context, Travis is not able to call Algolia.\nTerrible experience for contributors\nAt Algolia we care a lot about developer experience. Everything \u2014 from our rest API to all the ecosystems we built on top of it \u2014\u00a0is focused on providing the best developer experience for our users. So when our users are willing to contribute to our open source projects and their experience is less than ideal, it can be...frustrating.\nImagine you took some of your free time to fix a bug or refactor something in a library you use and the first feedback you get from the build system is more or less: \u00a0\"Tests are failing for some reason, figure it out yourself\".\n\nYou might look at it and spend some time understanding what's going on. Then, you\u2019d realize there is nothing you can do and you just wasted some of your precious time. In short: this is terrible for contributors and will definitely not encourage them to come back to us again.\nPainful for maintainers\nNot only do we want to provide the best developer experience for our users, but we also want our own developers to have a great maintainer experience.\nAs a maintainer, I'm always hesitant to merge a contribution if I don't see the results of the tests. Sure enough, if only the README.md was modified, it's safe to merge. But in general, it leaves you with a feeling of doing something wrong. What I am used to is to pull the contributor's branch and run the tests locally, but this is really time consuming.\nAnother solution would be to re-submit the same PR by pushing the branch to the original repository, either manually or using a script. But this means that the original PR gets closed, which complicates the process and is seen by contributors as a strange way to handle PR.\nOverall, this was so painful that we thought it was time to invest some time to find a better solution, once and for all.\nFirst try: using temporary keys with limited capabilities\nIn the end, the solution was pretty","record_index":1},{"post_id":7351,"post_title":"Travis Encrypted Variables and External Contributions","post_date":1518511390,"post_date_formatted":"Feb 13th 2018","author_name":"Julien Bourdeau","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6163f2a1af373d957715bcebee188ba3?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/travis-encrypted-variables-external-contributions\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Travis-community-PR-360x200.png","time_to_read":7,"content":"\u2026 straightforward: every build will use temporary credentials that will expire. We still want to avoid malicious developers using these keys, so we added a set of limitations to each key. It means that even if the key is publicly exposed and has write access to Algolia, there is so little one can do in such a limited time that it is simply not worth the effort.\nOn Algolia side, these limitations are different for each repository, but each set of keys:\n\nHas a subset of all the available ACLs\nExpires automatically after a few minutes\nOnly has access to some indices\nHas a limited number of operations per hour\nHas a small nbHits that you can't override\nCan only be used from Travis IP addresses\n\nThis is how we protect our API keys given Algolia API key features. Of course, it will be different for your own API.\nBut wait...we realized that, even with such keys, there's one drawback: in the case of Algolia, you're still able to create API keys, which means that an attacker could fork one API client repository, change the testing code to create keys using our restricted keys, and still escape the protection. Back to square one: we need another layer of protection.\nA proper solution: using an API key dealer\nThe best way we found to solve this \"API keys can create other API keys\" issue was to build a little server that would act as an API key dealer. It's open (no authentication) and holds one configuration per repository (what restrictions to apply to the key). This server is responsible for creating keys on the Algolia side, and giving them back to the build system.\nWe plan to open source the API Key Dealer we built in the coming weeks, and will let you know when it's out.\nBecause it's a publicly available server, anyone is able to generate keys, so we check if the call is legit and originates from Travis's IP ranges. Not only this, but when a key is requested, we sends the original TRAVIS_JOB_ID and we use it to verify that the repository is part of the Algolia organization","record_index":2},{"post_id":7351,"post_title":"Travis Encrypted Variables and External Contributions","post_date":1518511390,"post_date_formatted":"Feb 13th 2018","author_name":"Julien Bourdeau","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6163f2a1af373d957715bcebee188ba3?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/travis-encrypted-variables-external-contributions\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Travis-community-PR-360x200.png","time_to_read":7,"content":"\u2026 and that a job is currently running.\nThe whole process is described in this schema:\n\nThe client\nTo be able to call such an API key dealer from Travis, we needed a small client that could be reused in all our repositories.\nTo do so, we built a small Go script that compiles into one binary file.\nThis binary is downloaded before each build, so we can easily update it on all the repositories.\n\nIf you want more details, read the whole .travis.yml file.\nThe small binary will:\n\nAssess if temporary credentials are necessary\nCall the API key dealer\nPrint a comment (for help and debug purposes)\nExport env\n\nChallenge: testing advanced admin permissions\nDespite having a temporary key with limited capabilities, there might be permissions you cannot give to a public key: managing team members or changing account credentials for instance. The only solution for this is to skip the tests requiring admin permissions in external contributions.\nThe client is then responsible for determining if the credentials should be grabbed from the env variables (pull request from organization member) or from the API key dealer (pull request from external contributor). So yes, you still need to leave the master credentials in the encrypted env vars of Travis.\n\nIs it really worth it?\nAs you can see, we spent some time designing this system so finding out if this was time well invested is a healthy question to ask.\nThe answer: it all depends on your own ecosystem and what experience you want to offer to your contributors. If one day someone decides that they want to take some of their free time to help improve our API clients, we believe they should have the best possible developer experience. Our libraries don't yet have hundreds of external contributors but we want to reach that milestone and the only way to get there is to show respect to every contributor, no matter their level of engagement or skill set.\nI hope you enjoyed this blog post and that it encourages you to, if you get inspired,","record_index":3},{"post_id":7351,"post_title":"Travis Encrypted Variables and External Contributions","post_date":1518511390,"post_date_formatted":"Feb 13th 2018","author_name":"Julien Bourdeau","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6163f2a1af373d957715bcebee188ba3?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/travis-encrypted-variables-external-contributions\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/02-2018_Travis-community-PR-360x200.png","time_to_read":7,"content":"\u2026 contribute to our open source projects.\nThanks to Vincent Voyer,\u00a0Josh Dzielak, Ivana Ivanovic and Tiphaine Gillet for helping me with this article.","record_index":4},{"post_id":7323,"post_title":"Introducing Query Suggestions: Making Autocomplete Search Experiences Right","post_date":1517561606,"post_date_formatted":"Feb 2nd 2018","author_name":"Lucas Cerdan","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/10b0bc7dd6c81097ef921b3d292c7f0b?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/introducing-query-suggestions-better-autocomplete\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Query-suggestion-release-360x200.png","time_to_read":4,"content":"As-you-type autocomplete is a well known search pattern that many of the best websites offer. Providing suggestions helps the user enter longer queries. For mobile users, it reduces the amount of typing required to execute a search. But, it's not just about typing less. It\u2019s far more powerful \u2014 it helps users formulate the best search. Good suggestions are proven queries that ensure the best results. The quality of the suggestions is as important as the results they generate.\nToday, we\u2019re releasing Query Suggestions, a new feature that extends the conversational search experience. It is as if the user, with every keystroke, is conversing with the search bar, discussing with it exactly what to send to the search engine to obtain the best results. For this to work, suggestions need to be instantly responsive and intuitively\/accurately relevant to be useful. If either one of these is missing, the conversation is over.\n\nHarnessing analytics\nWhile generating suggestions might appear trivial, it comes with its own set of challenges. Suggestions often derive from popular queries: pick what your users have been searching for most, and suggest it next time so they don\u2019t have to type it.\nThe popularity of each query, extracted from analytics, can also be used as a great way to rank suggestions. If \u201ciphone charger\u201d is the most searched term, it might make sense to make it the first suggestion when your users are typing \u201ciph\u201d \u2014 but not always.\nShowing only relevant suggestions\nNot everything your users are searching for should end up as a suggestion. As is often the case when it comes to data, it needs to be processed and cleaned. For example, it would be a mistake to suggest a query that leads to no results. This is where our Query Suggestions tool comes into play. You are now able to configure a set of rules to create the most relevant suggestions for your business.\nPreventing spam & cheaters\nOne caveat to using analytics as a source of suggestions is","record_index":0},{"post_id":7323,"post_title":"Introducing Query Suggestions: Making Autocomplete Search Experiences Right","post_date":1517561606,"post_date_formatted":"Feb 2nd 2018","author_name":"Lucas Cerdan","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/10b0bc7dd6c81097ef921b3d292c7f0b?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/introducing-query-suggestions-better-autocomplete\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Query-suggestion-release-360x200.png","time_to_read":4,"content":"\u2026 relying on user-generated content: user input should never be trusted blindly.\nCastorama, a French home improvement tools and supplies retailer, made that mistake last year. Typing any kind of a query multiple times was the only requirement for it to start appearing in the autocomplete for everyone. Their system got flooded with inappropriate suggestions and Castorama had to shut down their website for weeks, missing many of sales opportunities.\nFor marketplaces, it is equally important that vendors cannot influence how suggestions are ranked; otherwise, they might be tempted to promote suggestions that favor their own products.\nIntroducing Query Suggestions\nOur Query Suggestions tool has been created to address these issues. Popular queries will be extracted from our Analytics API, and then processed and cleaned. While we have systems in place to limit inappropriate suggestions and prevent fraud, suggestions can also be moderated by the user with an easy-to-use blacklist.\nWe also support external analytics data, which will be particularly useful for new customers who can upload any historical data from previous search solutions, or Google analytics. And for more complex use cases, combining facets is also a possibility: for example a brand \u201cnike\u201d + a category \u201cshoes\u201d can generate \u201cnike shoes\u201d. While this is a simple example, our early users have used the feature extensively to guide their users to very precise content.\nScoping suggestions\nGuiding your user to the right content is paramount, and including a scope in the autocomplete is the perfect moment to do that. Simply indicate which attribute should be taken into account, and our Query Suggestions tool will generate the top related categories for each suggestion.\nExample of a scoped search\nEasy to implement\nAfter extracting and processing the data, the Query Suggestions tool will push all valid suggestions to a new index, and refresh it every 24 hours. Just like any other index, you can query it","record_index":1},{"post_id":7323,"post_title":"Introducing Query Suggestions: Making Autocomplete Search Experiences Right","post_date":1517561606,"post_date_formatted":"Feb 2nd 2018","author_name":"Lucas Cerdan","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/10b0bc7dd6c81097ef921b3d292c7f0b?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/introducing-query-suggestions-better-autocomplete\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Query-suggestion-release-360x200.png","time_to_read":4,"content":"\u2026 with the same API and libraries we provide. Here\u2019s a tutorial with InstantSearch iOS.\nPricing & availability\nQuery Suggestions is available today in two versions.\nFor all users, we have released an open source script that handles all the processing but requires hosting and configuration. Note: the processing will generate operations on an account that will impact monthly usage.\nFor Business and Enterprise tiers, a hosted version is made available directly on the Algolia dashboard. All settings and blacklists can be managed from there. Please reach out to your product specialist to request access, and, of course, give feedback.","record_index":2},{"post_id":7243,"post_title":"New Features, New Docs and a New Foundation for Algolia\u2019s Jekyll Plugin","post_date":1517476555,"post_date_formatted":"Feb 1st 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/new-features-for-jekyll-search-plugin\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/01-2018_Jekyll-plugin-360x200.png","time_to_read":5,"content":"We\u2019re proud to announce the new release of our Jekyll plugin, jekyll-algolia. Jekyll is the best-known static website generator, converting raw markdown content to beautiful HTML pages. This plugin helps developers make their Jekyll sites searchable.\nThe main strength of static websites like those built with Jekyll is that you don\u2019t need any database or backend server to run them. Just deploy a bunch of HTML, JS, CSS and images to a hosting provider and you\u2019re good to go.\nThe downside was that the only way you had to find content was to struggle with  or head back to Google and refine your query. With , that\u2019s a thing of the past. You can now search all your content directly from the static website itself, using the Algolia API.\nIntegration into the Jekyll ecosystem\nWe released the first version of the plugin in 2015, but a lot has happened since then. Jekyll got a major version update. Static websites are blooming everywhere. On our side we also indexed a large number of blogs and documentation websites. Everything we learned with these projects has now been distilled into this new version.\nThe plugin has been renamed from  to . This is one of many small changes to better follow the Jekyll conventions. We wanted  to be a good citizen of the ecosystem, and that comes by speaking the Jekyll language.\nBut plugins do not live in a vacuum. You will surely use this plugin with other plugins, and we had to make sure they were all playing along nicely with each other. For example, we tested it alongside the popular  plugin because they both manipulate the same objects (tags), and made sure our plugin was not interfering with the other and vice-versa. We also ensured it was compatible with the de-facto static hosting platforms: Netlify and GitHub pages.\nUsing the plugin\nTo use the plugin, you need to first add it to your , under the  group (like every other Jekyll plugin). You then need to define your Algolia credentials in the  file, and run the  command.\nView the","record_index":0},{"post_id":7243,"post_title":"New Features, New Docs and a New Foundation for Algolia\u2019s Jekyll Plugin","post_date":1517476555,"post_date_formatted":"Feb 1st 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/new-features-for-jekyll-search-plugin\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/01-2018_Jekyll-plugin-360x200.png","time_to_read":5,"content":"\u2026 code on Gist.\nView the code on Gist.\n\nThis will read all your content (pages, posts and custom collections) and push them to your Algolia index. Each record pushed that way will contain metadata about the original page (, , , ,  and any custom data you defined in your front-matter). All this data will then be available to your front-end, allowing rendering of rich search results.\nCheck out the live demo and the code repository.\nThe documentation explains in great detail how to integrate search into , the default Jekyll theme. Applying the same principles to your own theme is straightforward using our InstantSearch.js front-end library and its UI widgets.\nSmarter indexing\nThe previous version had a naive approach to indexing. It used to delete all records in the Algolia index, then push everything again. It meant that even the smallest typo fix could delete and recreate thousands of records.\nIn the new version, the implementation was replaced with a smarter diff algorithm, reducing the number of operations used by orders of magnitude. Everyone, including people on our forever-free Community plan, should be able to use the plugin without worrying about hitting their monthly quota limit.\nBetter documentation\nOne of our mottos at Algolia is that documentation is part of the code. If it's not documented, it's not usable. We worked hard on making the documentation clear and exhaustive, not only explaining the various options but also why, when and how they should be used.\nWith the Getting Started guide, you\u2019ll be up and running in a matter of minutes. Configuration options will let you customize which files should be indexed or excluded, as well as add your own custom Algolia settings. For those of you who have more advanced implementations, a hook system will let you write your own Ruby methods to alter the records in any way you\u2019d want before pushing them. If you\u2019re coming from the previous plugin version, we also have a migration guide to help you transition.\nBut","record_index":1},{"post_id":7243,"post_title":"New Features, New Docs and a New Foundation for Algolia\u2019s Jekyll Plugin","post_date":1517476555,"post_date_formatted":"Feb 1st 2018","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/new-features-for-jekyll-search-plugin\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/02\/01-2018_Jekyll-plugin-360x200.png","time_to_read":5,"content":"\u2026 documentation also comes in the form of error messages. We all know that even with the best documentation and code, errors will still happen. Maybe it will come from a misconfiguration on your side or a bug in the plugin. Either way, this should not stop you. This is why we made sure the error messages explain clearly what failed, why, and more importantly, how to solve it.\nSome errors are pretty common (badly copy-pasted credentials or malformed HTML) and can be easily identified and avoided before they ever happen. The plugin does both pre-emptive checks and smart error handling to catch those common errors and display a message explaining what happened and the steps required to fix the issue.\n\nHopefully you\u2019ll never have to see those error messages. But if you do, I think you\u2019ll be glad they explain what went wrong, and how to fix it.\nStart your search journey\nEasy to use, yet powerful and extensible \u2014 that\u2019s our new jekyll-algolia plugin. Whether you have a large (or a small) blog, a documentation website or a custom display of your own collections, give it a try. You\u2019ll have a fast and relevant search across all your content and your users will thank you for that.\nQuestions? Feedback? We'd love to hear it: @pixelastic, @algolia.","record_index":2},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"Most of world\u2019s languages feature a wide range of unique quirks that make search engines stumble when processing queries. In this article, we\u2019ll present features we implemented to solve these issues, and go over their rationale and technical implementations.\nOur engine is language agnostic, which means it\u2019s designed to work with every written language. This is no easy feat, considering all the little and big differences languages have, from being based on symbols from different alphabets like Japanese or using many variants of the same stem like in German.\nThere is no silver-bullet or magical way to handle each of these specificities. Some of the most common approaches rely on statistical models, which are by default close to impossible to debug and iterate upon. Our goal was to design a system that is not only simple to iterate upon, but runs fast and works transparently.\nWe believe that we\u2019ve found ways to achieve a great relevance in many of the world\u2019s languages. Today, we\u2019re explaining our approach, and how to best use it to configure the relevance of your Algolia index for every language.\nNormalizing characters\nA well known low-level step while doing NLP is normalization. The goal of this step is to depend more on the letter than on the way it was typed, to reduce friction between what the user wants and what has been indexed in the first place.\nThis step is actually composed of two steps:\n\nUsing the Unicode standard and Unicode supplements to find the canonical form of the letter: \u00c0 \u2192 A or \u00df -> ss, for example. For additional detail on how we handle Unicode, you can refer to this post.\nUsing the Unicode standard to find the lowercase form of the canonical form: \u00c0 \u2192 A \u2192 a.\n\nThose two steps are applied on the text while being indexed, and on the query, when a search is performed.\nThe normalization into lower case characters is easier to grasp: requiring an exact capitalization of the query is too strict to allow for a pleasant search","record_index":0},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 experience.\nLately, however, we have been questioning our strategy of indiscriminately removing the diacritics (\u00b4, \u00b8, \u00a8, etc.).\nThe strategy was designed for the languages we started with \u2014 including the French \u2014 \u00a0and it brought a couple of benefits:\n\nBe resilient to missing\/extra diacritics when users are typing on mobile.\nAllow users that can't or don't know how to type some characters to still be able to search for them.\n\nHowever, our bias led to some languages being harmed by this mandatory step of our search engine: mostly Nordic languages (Swedish, Norwegian, etc.) and Turkish. In French, e and \u00e9 are considered by native speaker as more or less the same letter; for a Norwegian speaker, o and \u00f8 are not interchangeable at all.\nInterchanging two such letters can create an issue by making the search unable to discern between two completely different words, like in this example in Turkish: cam \u2192 glass, \u00e7am \u2192 pine (pretty annoying if you are focused on selling furniture).\nAs we already mentioned, this step is today mandatory in Algolia\u2019s search engine and can't be disabled. We are thinking on improving this to allow a more granular handling of languages for which diacritics play an important role.\nTypo tolerance\nTypo tolerance is an important part of a search engine, and one of the earliest features Algolia brought to the table. Its role is really simple: be resilient to input mistakes, or plain misspelling.\nThe rules for the typo tolerance are as follows:\n\nif a letter is missing in a word, we consider it a typo: \u201chllo\u201d \u2192 \u201chello\u201d\nif there is a extraneous letter, we consider it a typo: \u201cheello\u201d \u2192 \u201chello\u201d\nif two consecutive letters are inverted, we consider it a typo: \u201chlelo\u201d \u2192 \u201chello\u201d\n\nBy default, Algolia accepts 1 typo for words that are at least 4 letters long, and 2 typos for words that are at least 8 letters long. Those thresholds are configurable by the user.\nOn top of those rules, we added some subtleties, such as","record_index":1},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 the fact that a typo on the first letter of a word is \u201cpenalized\u201d double, because it is much less common to mistype the first letter of a word than any other letter.\nThose rules are easily applied to languages using Latin alphabets: while some languages tend to have shorter or longer words on average (which could be translated to increasing or decreasing the minimum number of letters before accepting typos), they are all similar in the way typo tolerance can be configured.\nLanguages using logograms, such as Chinese or Japanese are, however, a whole different story. Because of the way those languages are input, the typos as we know them in Latin alphabet languages don\u2019t really occur.\nHowever, those languages are subject to the presence of alternative logograms for the same word. Those alternatives can exist for two reasons:\n\nthe presence of a traditional and a simplified drawing for the same character\nthe presence of two logograms that can be used interchangeably\n\nRemoving stop words\nIn the last couple of decades, while browsing the internet, most of us adopted the habit of transforming natural language questions such as: \"What is the best search engine?\" to full-text queries we knew would be better interpreted by search engines, such as: \"best search engine\". While those requests used to yield better results thanks to the decrease of noise in the query, this is less and less true because of two major factors:\n\nSearch engines adapted to allow more casual users to find their results without having to pick up this habit\nThe advent of voice search is favoring the use of fully formed sentences\n\nBy default, Algolia is requiring all the words in the query to be present in a record for it to be considered a match. While it is possible to remove words when no results are found, all the words will still be considered\u00a0equal, and \"the\" won't be considered as less important than \"search\".\nTo handle this case, we introduced pretty early a removeStopWords setting that","record_index":2},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 removes all the \u201cunnecessary\u201d words from the query string, and eventually keeps only \"best search engine\".\nTo accomplish this, we are using a really simple dictionary-based approach. We parsed several sources (Wiktionary and ranks.nl) in order to constitute a list of words commonly used as stop words, not only in English, but in ~50 languages available.\nThe first implementation of the setting was suffering from a major flaw: it was a single boolean value, and one could only activate this feature for all languages simultaneously or not at all. A lot of corner cases were observed: like \"th\u00e9\" (the French word for \"tea\") being removed because of \"the\" being an English stop word, thus transforming a really specialised \"magasin de th\u00e9\" (tea shop) into a regular \"Magasin\" (shop).\nSince then, we changed the setting; the feature can now be set to a list of languages, allowing one to remove stop words from one language at a time.\nWhile this feature doesn't work well in itself with languages not using spaces (such as Japanese), once coupled with other pre-processing steps (such as word segmentation) it behaves just the way you would expect it to.\nWhile the list of stop words for a given language is not likely to change, we still have a few ideas in mind in order to improve this feature:\n\nAdd support for more languages\nIntroduce an option to make those stop words optional; meaning that they won\u2019t prevent results from showing but will influence the ranking if they are inside a record. As of today, they are simply removed from the query.\n\nHandling alternative forms\nIn English, words have a rather limited number of alternative forms:\n\nnouns have a plural\nverbs have a few conjugated forms\n\nHowever, even if \"car\" and \"cars\" are not so different from each other, there are use cases where you still want to consider them as one and the same (\u201cfeet massage\u201d and \u201cfoot massage\u201d).\nTo tackle this issue we created a parameter called ignorePlurals which has over the years","record_index":3},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 undergone several transformations.\nThe problem can be solved in two different fashions:\n1. When indexing words, strip them to their singular form: \"feet\" \u2192 \"foot\", \u201cmassages\u201d \u00a0\u2192 \"massage\u201d. Whenever a query is sent, do the same to each words of the query: \"feet massages\" \u2192 \"foot massage\", then search for those radical forms.\n2. Index the words the way they are given (\"feet\" \u2192 \"feet\"), but generate all possible alternatives for all the words in the query: \"feet massages\" \u2192 \"(foot|feet) (massage|massages)\u201c and search for any of these forms.\nAt Algolia we chose to go for the second method. The reason is simple: typo tolerance. If the word is not indexed exactly as it was given, then you can't accurately compute the number of typos afterwards: that's why it is important for us to not alter the words that are indexed.\nThere are two Natural Language Processing (NLP) techniques to reduce a word to its singular form or find its alternative forms: Stemming and Lemmatization.\n\nStemming is relying on rules such as: if your word is ending with \"ing\", strip it. The most popular implementation of stemming is called Snowball.\n\nWe have mixed feeling towards stemming, as a single rule could cause a lot of harm if you consider the following points:\n\nDifferent languages could be present in a single index, meaning that English stemming rules could be applied to French text, which is likely to create a ton of non-valid words, and yield to noisy, confusing results.\nSeveral of our use cases make heavy use of brand names, proper nouns, street addresses, etc., and for those, using arbitrary rules designed for common nouns degrades a lot of the relevance\nLemmatization, on the contrary, doesn't try to create general rules, and relies on a dictionary giving for each word its radical : \"eating\" \u2192 \"eat\", \"ate\" \u2192 \"eat\", ...\n\nWe make heavy use of lemmatization. The only thing that lemmatization efficiency depends on is the accuracy and completeness of the dictionary used. We","record_index":4},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 are continually working on improving our dictionary, primarily based on Wiktionary.\nWiktionary relies heavily on templates such as {en-noun|s} allowing Wiktionary contributors to declare alternative forms when writing a page on a word. For example, {en-noun|s}, would show up like this on the \u201ccar\u201d page of Wiktionary:\nNoun[edit]\ncar (plural cars)\nBy using those templates found inside the Wiktionary data, we are able to build our dictionary of alternative forms. Almost every language has its own template syntax, and many languages have multiple templates to do so. Because implementing everything at once is extremely tedious, we chose to improve our dictionary gradually, mostly upon user feedback about missing words\/languages.\nWe spent a fair amount of time on this, and improved a lot of the features over time:\n\nAs mentioned above, for removeStopWords, the feature started being a simple boolean switch, allowing only an \"all-languages-or-nothing\" behavior. It is now possible to specify which languages the lemmatization should be used with.\nLanguages like German also have alternative forms depending on the grammatical role of the word in the sentence. This is not a plural per se, but we have had a customer reaching out to us in order to support this. And so we did! That makes today's name a bit misleading as it's not about plural anymore, but also declensions, diminutives, etc.\n\nignorePlurals evolves a lot: words are constantly added, new types of alternatives are supported (e.g., most recently, Dutch diminutives). The hardest part is actually not adding words, but finding the right balance between improving relevancy and harming it with noise.\nSegmenting CJK text\nCJK stands for Chinese, Japanese, Korean. The acronym is pretty popular and we use it a tad inaccurately by using it to refer to any language that doesn't use spaces to separate their words (Thai and Lao, for example).\nFinding the words in a query is a capital step of a search engine: when you are searching","record_index":5},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 for something, you want documents containing the words of your query, but not necessarily in the same order. You want the engine to be less strict about whether words are consecutive. For example, if you are searching for \"chocolate cookies\", you don't want only results with \"chocolate\u201d and \u201ccookies\" back to back \u2014 you also want \"chocolate chip cookies\" and \"cookies with chocolate chips\" to show up in results.\nWhen a language delimits its words with spaces, things are relatively simple; when it doesn't, you roughly have 3 choices:\n1. Take each character separately. At first, from the point of view of someone not familiar with CJK languages, this makes sense, especially with Chinese or Japanese. Those languages heavily use logograms, and we have the false impression that 1 character = 1 word. This, however is not the case, as many notions are expressed as logogram compounds: a concatenation of several logograms that make up for a new word. Besides, as mentioned, Chinese and Japanese are not the only ones not using spaces. That leaves this approach out of question.\n2. Require the exact query to be inside the matches, verbatim. This greatly constraints the search, but it at least keeps the meaning of the query. While far from being perfect, it's an acceptable fallback if the method I will introduce next fails. Example: \u5bff\u53f8\u3068\u30e9\u30fc\u30e1\u30f3 (sushi and ramen). While you would want to be able to also find \u30e9\u30fc\u30e1\u30f3\u3068\u5bff\u53f8 (ramen and sushi), with this method we would constraint the results to documents containing only \u5bff\u53f8\u3068\u30e9\u30fc\u30e1\u30f3 (sushi and ramen).\n3. Find by yourself the words inside the query. The segmentation of a sentence into words is a problem CJK experts have always faced, and several satisfactory techniques have been elaborated to solve the issue. None of them guarantee a 100% accuracy, and like everything else in engineering, it's only a matter of compromise between speed\/cost and accuracy\/quality. This is the behavior we implemented inside","record_index":6},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 the engine.\nThere are many ways to implement the segmentation, ranging from statistical to dictionary-based methods.\nWe strive for an as-you-type experience, which gives us roughly 100ms (threshold below which our brain doesn't really register difference in speed of response) to answer a query, network latency included. Because of this requirement, we settled for the fastest method in order to find words inside a query: dictionary-based, as implemented in the ICU library, using the MECAB dictionary enriched with data from Wiktionary.\nThe idea is simple: starting from the left of the query, we will try to decompose the query with words that are present in our dictionary. In case of ambiguity we are prioritizing the solution that maximizes the length of the words and minimizes the number of characters not belonging to a known word.\nAs of today, we already have a couple of ideas about what we could do to improve the segmentation of CJK languages. First and foremost, the segmentation is based on a dictionary, which means that any improvement made to the dictionary is sure to improve the overall segmentation:\n\nImproving the dictionary means removing words that are extremely rare, as much as adding common ones.\nWe also heard the feedback from customers about the segmentation underperforming when named entities (such as brand name, people, etc.) are present inside the query. Those entities are not always found inside our dictionary, and the segmentation step is often not able to accurately separate it from the other words. Introducing a custom dictionary that the user herself can improve with custom names is something we are thinking about.\n\nIndexing agglutinated words\nThere is another issue that is in fact really close to the segmentation of CJK characters. In some languages (German, Dutch, Finnish, etc.) several nouns can be concatenated without space to form new words. For example, Baumhaus is a German word composed of Baum (tree), and Haus (house) which designates a","record_index":7},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 \u201ctree house\u201d.\nA person searching for \u201cBaumhaus\u201d (tree house) may very well be interested in results containing something like \u201cHaus in einem Baum\u201d (house in a tree), but the default Algolia behavior is to only search for Baumhaus. This issue is what triggered us to spend some time working on handling agglutinated words, or compounds, better.\nThe very first step that we implemented was a naive splitting of the words in the query. The idea was simple: in the case of \u201cBaumhaus\u201d, if the index was to contain \u201cBaum Haus\u201d consecutively, a match was possible. This feature was more akin to a typo tolerance on spaces than a real split of compound words, as the words still had to be adjacent to one another, which is a limitation we got rid of in the new version.\nSince then, we settled for full fledged processing steps that look individually at each word. If the word happens to be a compound word (we are here again relying on static dictionaries to avoid false positives), its various parts are indexed separately.\nAs of today, this setting can be activated using the decompoundedAttributes setting. The languages we wish to attempt a decompounding for must be specified; three languages are implemented: Finnish, German and Dutch.\nWe designed this feature to interact with all the other goodies coming with Algolia: synonyms, ignorePlurals, etc.\nThis setting is still rather new, and we expect to allocate time improving the dictionaries used by the decomposition process in order to increase the accuracy of the feature. While today we don\u2019t have a strong demand for languages other than those implemented, additional languages may show up in the future.\nAutomatic concatenation\nFor languages which use spaces to delimit words, there are still subtleties around separating words, for example with acronyms or words separated by hyphens or single quotes. We will review two cases to better illustrate how Algolia handles this.\nAcronyms: D.N.A.. There are two ways of","record_index":8},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 approaching this:\n1. considering that we have three words: D and N and A\n2. considering that we have a single word: DNA\nIn this case, the second solution makes more sense:\n\nPeople may search both with D.N.A. or DNA. Considering it always as DNA makes it searchable whatever its form inside the query or the index.\nAlgolia uses prefix search if configured to do so, which means that \u201cD N A\u201d as three separate words can match every record that has 1 word starting with D, 1 word starting with N and 1 word starting with A. This is likely to match a lot of results that have nothing to do with DNA.\n\nIn this case, we consider D.N.A. the same as DNA.\nCompounds: off-campus or a.to_json. If we were to apply the rules previously established for acronyms, we would index and search for offcampus and classmethod. In those cases, this is not the ideal behaviour. In the case of compounds, words can be searched individually, or even written without the punctuation in between: both \u201coff campus\u201d and \u201coff-campus\u201d are valid spellings. Transforming \u201coff-campus\u201d into \u201coffcampus\u201d would prevent a search for \u201coff campus\u201d to retrieve results containing \u201coff-campus\u201d.\nIn that case, \u201coff-campus\u201d will be considered as \u201coff\u201d \u201ccampus\u201d and \u00a0\u201coffcampus\u201d.\nIn the case of \u201ca.to_json\u201d, we will consider \u201cato_json\u201d and \u201cto_json\u201d.\nThis behavior follows two simple rules:\n\nIf letters are separated by separators, consider the concatenation of those letters without the separators in the middle (D.N.A. \u2192 \u00a0DNA)\nIf each separated component is three or more letters, also consider it as a standalone word (off-campus \u2192 off + campus + offcampus).\n\nThose rules allow for enough flexibility on the presence or absence of punctuation, and avoiding unnecessary small words like \u201ca\u201d for \u201ca.to_json\u201d that could hurt any prefix search.\nSummary\nBelow is a table roughly summarizing the different pre-processing steps we are applying on some of the most popular","record_index":9},{"post_id":7267,"post_title":"Handling Natural Languages in Search","post_date":1517311009,"post_date_formatted":"Jan 30th 2018","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/natural-languages-in-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Inside-Engine-language-01-2-360x200.png","time_to_read":20,"content":"\u2026 languages found across the Internet:\n\nWhile we are always striving to address languages specificities, we know that we are only at the very beginning of the journey, and we already have a long list of language-related issues we want to address further.\nThose issues relate to various parts of the processing of a query:\n\nat the character level: transliteration between hiragana to katakana (two of the three Japanese alphabets) and vice-versa, transliteration between the digits of different languages, better handling of diacritics for languages that suffer from our current method, etc.\nat the grammatical level: improving our dictionaries to even better handle stop words, plurals, declensions, and maybe conjugations\nat the typo tolerance level: a way to handle the Arabic habit of omitting vowels, a deeper look into languages such as Thai that are different from both Indo-European and CJK languages, etc.\nlast but not least, sustaining the effort of improving our numerous dictionaries, in order to improve the efficiency of our current processing steps.\n\nIf you have ideas or feedback, we'd love to hear it: comment below or tweet to us @algolia.","record_index":10},{"post_id":7258,"post_title":"The Meltdown and Spectre impact on Algolia infrastructure","post_date":1515486652,"post_date_formatted":"Jan 9th 2018","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/meltdown-spectre-impact-algolia-infrastructure\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Spectre-Meltdown-01-360x200.png","time_to_read":3,"content":"On January 3rd, several vulnerabilities against modern CPU microarchitectures made news headlines. Those vulnerabilities expose a risk of an information leak. A software could potentially exploit the vulnerabilities to get access to the data of another software stored in memory. This is a major security incident.\nIn total, two attack vectors have been disclosed to the public:\n\nSpectre: two vulnerabilities available in nearly all processors on the market. Those two vulnerabilities are known as \"bounds check bypass\" (CVE-2017-5753) \u00a0and \u201cbranch target injection\u201d (CVE-2017-5715)\nMeltdown: one vulnerability affecting mainly Intel CPUs, known as \u201crogue data cache load\u201d (CVE-2017-5754)\n\nImpact for Algolia\nOur infrastructure is a mix of bare-metal and cloud infrastructure. We have three parts in our infrastructure that are impacted by these security vulnerabilities.\nOur API servers\nThose servers are hosting our users\u2019 data and power the indexing\/search API. They are distributed worldwide in more than 50 data centers with a similar hardware configuration using Intel CPUs (mainly Intel E5-1650v4). The servers are configured and tuned for performance. We have no virtualization layer and only run our own software while applying security best practices.\nThe CPUs we are using are vulnerable, but the impact is mitigated because we do not expose any way to run custom code on our machines. The only way to exploit those vulnerabilities would be to get access to the machine that already gives access to privileged information. Our security efforts remain oriented to making this impossible, and we are working on integrating the KPTI kernel patch and reducing\/testing the performance impact it introduces.\nOur website and dashboard\nWe are using AWS to run our website and dashboard, and this is the place where we have our database listing users. We, of course, consider it a critical part of our infrastructure.\nWe followed closely the AWS actions to protect all instances and they","record_index":0},{"post_id":7258,"post_title":"The Meltdown and Spectre impact on Algolia infrastructure","post_date":1515486652,"post_date_formatted":"Jan 9th 2018","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/meltdown-spectre-impact-algolia-infrastructure\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2018\/01\/01-2018_Spectre-Meltdown-01-360x200.png","time_to_read":3,"content":"\u2026 completed their patch deployment to protect them.\nHowever, we decided to move all our website and dashboard virtual machines to dedicated instances to make sure we do not share our hardware with any other AWS customers. This action was not required to be protected but our general security posture is one of extreme caution.\nAnalytics\nOur analytics stack is computing statistics on your search usage, analyzing query trends.\nWe are in the process of migrating our analytics stack to Google Compute Platform and we already have several customers running on this stack (our current stack is on bare-metal machines, so the status is similar to our API servers).\nLike AWS, Google was working on the fix for a long time and their infrastructure is already protected against those vulnerabilities. Our stack also relies on several systems, including Pub\/Sub and DataFlow which are protected against the vulnerabilities.\nSecurity at Algolia\nOur security team is constantly monitoring services running on our own machines, as well as those hosted on cloud platforms to ensure that we're protected against the latest security vulnerabilities. If you have any questions about our process or want to share any information feel free to reach out to the team directly at security@algolia.com.","record_index":1},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"This year I gave a talk about how to make open source projects successful by ensuring everything is in place to attract all kinds of contributions: issues, documentation or code updates. After the talk, the feedback I got was \"It\u2019s nice, you showed how to make projects successful, but how do I even start doing open source?\". This blog post is an answer to that question; it explains how and where to start contributing to projects and then how to create your own projects.\nThe knowledge shared here is based on our experience: at Algolia, we have released and maintained multiple open source projects that proved to be successful over time, and I have spent a good amount of time practicing and creating open source projects too.\nGetting your feet wet\n\nA key moment for my career was six years ago at Fasterize (a website performance accelerator). We faced an important memory leak on our Node.js workers. After searching everywhere except inside the actual Node.js codebase, we found nothing that could cause it. Our workaround was to restart the workers every day (this reset the memory usage to zero) and just live with it, but we knew this was not a very elegant solution and so I wanted to understand the problem\u00a0as a whole.\nWhen my co-founder St\u00e9phane suggested I have a look at the Node.js codebase, I almost laughed. I thought to myself: \"If there's a bug, it's most probably our code, not the code from the developers who created a revolutionary server-side framework. But, OK, I'll have a look\". Two days later my two character fix to the http layer of Node.js was merged, and solved our own memory leak.\nDoing this was a major confidence boost for me. Amongst the thirty other people who had contributed to the http.js file were folks I admired, like isaacs (npm creator)\u2014 making me realize that code is just... code, regardless of who wrote it.\nAre you experiencing a bug with an open source project? Dig in and don't stop at your local workaround. Your solution can benefit","record_index":0},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 others and lead you to more open source contributions. Read other people's code. You might not fix your issue right away, it might take some time to understand the code base, but you will learn new modules, new syntax and different ways to code that will make you grow as a developer.\nOpportunistic contributions\nFirst contributions labels on the the Node.js repository\n\"I don't have an idea\" is a common complaint by developers who want to contribute to open source but think they don't have any good ideas or good projects to share. Well, to that I say: that\u2019s OK. There are opportunistic ways to contribute to open source. Many projects have started to list good contributions for first-timers via labels or tags.\nYou can find contribution ideas by going through these websites: Open Source Friday, First Timers Only, Your First PR, CodeTriage, 24 Pull Requests, Up For Grabs,\u00a0Contributor-ninja\u00a0and First Contributions.\nBuild some tooling\nTooling is a nice way to publish something useful to others without having to think too much about complex problems or API design. You could publish a boilerplate for your favorite framework or platform that would gather the knowledge of many blog posts and tools into a nicely explained project, ready with live reload and publishing features. create-react-app is one good example of such tooling.\nThere are 58K boilerplate repositories on GitHub, it's easy and rewarding to publish one\nToday you can also build pure JavaScript plugins for Atom and Visual Studio Code like we did with our Atom autocomplete module import plugin. Is there a very good plugin for Atom or Sublime Text that does not yet exist in your favourite editor? Go build it.\nFinally, you could also create plugins for webpack or babel that are solving a particular use case of your JavaScript stack.\nThe good thing is that most platforms will explain how to create and publish plugins so you won\u2019t have to think too much about how to do it.\nBe the new maintainer\nWhen browsing","record_index":1},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 through projects on GitHub, you might sometimes find and use projects that are abandoned by their creator. They are still valuable, but many issues and pull requests are sitting in the repository without any answer from the maintainer. What are your options?\n\nPublish a fork under a new name\nBe the new maintainer\n\nI recommend you do both at the same time. The former will help you move forward with your project while the latter will benefit you and the community.\nHow to become the new maintainer, you ask? Drop an email or a tweet to the maintainer and say \"Hey, I want to maintain this project, what do you think?\". This usually works well and is a great way to start your open source career with a project that is already known and useful to others.\n\nExample tweet sent to revive an abandoned project\nCreating your own projects\nThe best way to find your own project is to look at problems that today have no good solutions. If you find yourself browsing the web for a particular library solving one of your problems and you don't find it, then that's the right time to create an open source library.\nHere's another key moment for my own career. At Fasterize we needed a fast and lightweight image lazy loader for our website performance accelerator \u2014not a jQuery plugin but a standalone project that would be injected and must work on any website, on every browser. I spent hours searching the whole web for the perfect already-existing library and I failed at it. So I said: \"We're doomed. I can\u2019t find a good project, we can't do our startup\".\nTo this, St\u00e9phane replied: \"Well, just create it\". Hmm.. ok then! I started by copy pasting a StackOverflow answer in a JavaScript file and ultimately built an image lazy loader that ended up being used on websites like Flipkart.com (~200M visits per month, #9 website in India). After this success, my mind was wired to open source. I suddenly understood that open source could be just another part of my developer career, instead of a field","record_index":2},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 that only legends and mythical 10x programmers fit into.\n\n \nA problem without any good solution: solve it in a reusable way!\nTiming is important. If you decide not to build a reusable library but rather inline some workaround code in your own application, then that's a missed opportunity. At some point, someone will create the project you might have created. Instead, extract and publish reusable modules from your application as soon as possible.\nPublish it, market it and share it\nTo be sure anyone willing to find your module will indeed find it, you must:\n\nCreate a good README with badges and vanity metrics\nCreate a dedicated website with a nice design and online playground. Want some inspiration? Have a look at Prettier.\nPost your project as answers to StackOverflow and GitHub issues related to the problem you are solving\nPost your project on HackerNews, reddit, ProductHunt, Hashnode and any other community-specific aggregation website\nPropose your new project to the newsletters about your platform\nGo to meetups or give talks about your project\n\n\nShow your new project to the world\nDon't fear posting to many websites; as long as you truly believe what you have made will be valuable, there is no such thing as too much information. In general, communities are really happy to have something to share!\nBe patient and iterate\nIn term of \u201cvanity metrics\u201d (number of stars or downloads), some projects will skyrocket on day one but then have their growth stopped very early. Others will wait one year before being ready for HN frontpage. Trust that your project will be at some point noticed by other users, and if it never does, then you have learned something: it's probably no use to anyone but you \u00a0\u2014\u00a0and that is one more learning for your next project.\nI have many projects that have 0 stars (like mocha-browse), but I am never disappointed because I don\u2019t have high expectations. That's how I always think at the beginning of a project: I found a good problem, I","record_index":3},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 solved it the best way I could, maybe some people will use it, maybe not. Not a big deal.\nTwo projects for a single solution\nThis is my favourite part of doing open source. At Algolia in 2015 we were looking at solutions to unit test and freeze the html output of our JSX written React components for InstantSearch.js, our React UI library.\nSince JSX is translated to function calls, our solution at that time was to write expect(<Component \/>).toDeepEqual(<div><span\/><\/div). That's just comparing two function calls output.But the output of those calls are complex object trees: when run, it would show \"Expected {-type: 'span', \u2026}\". The input and output comparison was impossible and developers were getting mad when writing tests.\nTo solve this problem, we created algolia\/expect-jsx that allowed us to have JSX string diffs in our unit tests output instead of unreadable object trees. Input and output of the test would be using the same semantics. We did not stop there. Instead of publishing one library, we extracted another one out of it and published two libraries:\n\nalgolia\/react-element-to-jsx-string transforms JSX function calls back to JSX strings\nalgolia\/expect-jsx does the linking between react-element-to-jsx-string and mjackson\/expect, the expectation library\n\nBy publishing two modules that are tackling one problem together, you can make the community benefit from your low-level solutions that can be reused on a lot of different projects, even in ways you never thought your module would be used.\nFor example, react-element-to-jsx-string is used in a lot of other test expectations frameworks along with being used on documentation plugins like storybooks\/addon-jsx.Today, to test the output of your React components, use Jest and snapshots testing, there's no more the need for expect-jsx in those situations.\nFeedback and contributions\n\nThat's a lot of issues. Also, it's faked just to have a nice picture \ud83d\ude42\nOnce you start getting feedback and","record_index":4},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 contributions, be prepared to be open minded and optimistic. You will get enthusiastic feedback, but also negative comments. Remember that any interaction with a user is a contribution, even when it seems like just complaining.\nFor one thing, it is never easy to convey intentions\/tone in written conversations. You could be interpreting \"This is strange...\" as: it's awesome\/it's really bad\/I don't understand\/I am happy\/I am sad. \u00a0Ask for more details and try to rephrase the issue to better understand where it\u2019s coming from.\nA few tips to avoid genuine complaints:\n\nTo better guide users giving feedback, provide them with an ISSUE_TEMPLATE that is displayed when they create a new issue.\nTry to reduce the friction for new contributors to a minimum.Keep in mind that they may not yet be into testing and would gladly learn from you. Don't hold Pull Requests for new contributors because there's a missing semicolon;, help them feel safe. You can gently ask them to add them, and if that doesn\u2019t work, you can also merge as-is and then write the tests and documentation yourself.\nProvide a good developer experience environment in terms of automated tests, linting and formatting code or livereload examples.\n\nThat's it\nThanks for reading, I hope you liked this article to the point where you want to help or build projects. Contributing to open source is a great way to expand your skillset, it's not a mandatory experience for every developer, but a good opportunity to get out of your comfort zone.\nI am now looking forward to your first or next open source project, tweet it to me @vvoyer and I'll be happy to give you advice.\nIf you love open source and would like to practice it in a company instead than doing it on your free time, Algolia has open positions for open source JavaScript developers.\nOther resources you might like:\n\nopensource.guide, Learn how to launch and grow your project.\nOctobox, your GitHub notifications as an email. Awesome way to avoid the \"too many issues\"","record_index":5},{"post_id":7200,"post_title":"Start Your Open Source Career","post_date":1513845020,"post_date_formatted":"Dec 21st 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/start-your-open-source-career\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_Open-Source-projects-360x200.png","time_to_read":13,"content":"\u2026 effect by focusing on the ones that matter\nProbot, GitHub Apps to automate and improve your workflow like closing very old issues\nRefined GitHub provides an awesome maintainer experience for GitHub UI at many levels\nOctoLinker makes browsing other people's code on GitHub a great experience\n\nThanks to Ivana, Tiphaine, Adrien, Josh, Peter and Raymond for their help, review and contributions on this blog post.\nJanuary 2018 update:\u00a0\nCheck out discussions about this post on Hacker News and Reddit.","record_index":6},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"In today\u2019s complex world, ops engineers and SREs get to worry not just about quality of infrastructure, but about cost reduction. Here, I\u2019ll share a few tips on reducing cost of servers with popular cloud providers, as well as a general approach for those to whom this type of work is new.\nWhen you get hired as an Ops engineer or an SRE, you probably know what you are getting into and what you are supposed to do: things like maintaining servers, developing build and deployment pipelines, provisioning and initiating cloud servers, monitoring services and the likes come to mind. One thing that does not usually come to mind when discussing the role of the operations engineer are finance-related activities such as cost analysis and cost reduction.\nThis, as it turns out, can sometimes be quite a big part of our jobs in companies.\nWho would have thought, for example, that I\u2019d become quite familiar with a term like COGS, or \u201ccost of goods sold\u201d, defined by Investopedia as \u201cthe direct costs attributable to the production of the goods sold by a company\u201d. In plain English: \u201chow much does it cost to create a product before selling it\u201d, or, if you prefer plain engineering English: \u201chow much do we spend to have our services running in production\u201d. Basically, this is about cost reduction: you start off with creating a budget in order to understand how much you are spending, how much you are going to spend, and whether your costs are going up or down. Let\u2019s look step by step on how to do this with cloud services.\nSteps to success with (cloud) cost reduction\nHandling cost in today\u2019s cloud is very similar to other analysis tasks such as monitoring. You have to:\n\n1. Collect billing data\n2. Analyze and visualize the data\n3. Alert on issues\n4. Act\n\nCollect\nAmazon Web Services (AWS) has a very nice billing dashboard, They basically collect the data for you, so you can just skip to the analyze part \\o\/. Google cloud platform (GCP) does not currently provide a","record_index":0},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"\u2026 very convenient way to see your billing data, but they do provide a way to export your billing data to BigQuery.\nAnalyze\nThere are all kinds of ways to analyze your data, but a simple way you can start with is looking at 2 types of analyses:\n\n1. Monthly Analysis\n\nMonth-by-month analysis over a period of a year (or even less than a year) can give you interesting insights, such as: is the cost of the infrastructure increasing at the same rate as the growth of the business?\n2. Daily Analysis\nTrack the changes you have made this week, and see if they have the effect on the monthly budget that you expected.\nAWS has a great tool for cost analysis \u2014 the Cost Explorer:\n\nIt is a part of the billing dashboard and it has very strong analytical abilities: you can analyze cost by date (yearly\/monthly\/daily), and it has hundreds of other dimensions that can be configured \u2014\u00a0such as what machines and databases you are spending on, when things start going up or down, etc. A good practice here is to tag assets; for example, you can tag an instance by its principal user so you can quickly track back to the person over-utilizing or under-utilizing a machine. Another good practice is to tag by product which will make it easy to know at a certain point in time how much a project costs.\n\nIn GCP \u2014\u00a0once your data is in BigQuery \u2014\u00a0you can use Google\u2019s Data Studio, or Redash, an open source tool to do quite sophisticated analyses. Check out this elaborate blog post from Google. \nAlert\nBoth AWS and GCP let you set up billing alerts, like this one:\n\nIn AWS, you can set up billing alerts (such as notifications of costs exceeding the free tier) in a granular way.\nAct\n\nCleaning. The first and easiest thing to do is start cleaning stuff up \u2014 by that, I mean removing machines that are not used. We all start test machines that we plan deleting at the end of the day, but they often stay behind and incur cost for...a while later. I suggest a continuous cleaning plan for machines,","record_index":1},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"\u2026 instances, IP addresses, files from S3 (here, you can use the lifecycle management, which automatically moves files from hot storage to cold storage, and then deletes them per the schedule you set), EBS snapshots (if you backup your data)...and \u2014 very important if you are using AWS \u2014 you should not forget to check all regions.\nData transfer. In cloud services, data transfer is quite complicated. Here is an example:\n\n\nYou can see that cost of traffic between different regions and AWS services differs greatly, so it can be very important to understand where you are transferring data to and from.\nOne tip: communication between regions is relatively expensive, but can be cheaper between regions that are close, or if one of them is new. If you want to be multiregional, one way to save is by finding cheaper connections between regions.\nAnother tip: you can reduce the pricing of load balancers by simply asking for a discount: cloudfront and CDN pricing are quite negotiable at high volumes.\n\nCompute is more simple and there are quite a few ways to save quite a bit of money. Here are a few:\n\n- AWS: Switch to a new generation of instances. For example, the M3 instance had an inferior CPU, less memory, and cost more than the new M4. The recently released C5 instance family has faster CPU than C3 and C4 and once again costs less.\n- AWS: Reserved instances allows you to get significant discounts on EC2 compute hours in return for a commitment to paying for instance hours of a specific instance type in a specific AWS region and availability zone for a pre-established time frame (1 or 3 years). Further discounts can be realized through \u201cpartial\u201d or \u201call upfront\u201d payment options.\n- AWS: EC2 Spot instances are a way to get EC2 resources at a significant fluctuating discount \u2014 often many times cheaper than standard on-demand prices \u2014 if you\u2019re willing to accept the possibility that they be terminated with little to no warning if you underbid. I highly recommend a","record_index":2},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"\u2026 company by the name of Spotinst that can manage the hard work and uncertainty for you.\n- GCP: if your workload is stable and predictable, you can purchase a specific amount of vCPUs and memory for up to a 57% discount off of normal prices in return for committing to a usage term of 1 year or 3 years.\n- GCP preemptible VM is an instance that you can create and run at a much lower fixed price than normal instances on GCP. However, Compute Engine might terminate (preempt) these instances if it requires access to those resources for other tasks. They also last a maximum of 24 hours.\n\nTagging for cost visibility. As mentioned above, as the infrastructure grows, a key part of managing costs is understanding where they lie. It\u2019s strongly advisable to tag resources, and as complexity grows, group them effectively.\nHuman \"resources\". Don\u2019t be shy about asking your account manager for guidance in reducing your bill. It\u2019s their job to keep you happily using their service.\nServerless. I am not going to advise moving all your servers to a serverless architecture. It\u2019s not great for everything, but let\u2019s say you have a cron machine, and it\u2019s running something every hour \u2013 perhaps moving it to serverless makes more sense. You pay per function run, and save additional money by saving on operation costs: looking at logs, maintaining the server, etc.\nUsing multi-cloud. This is complicated and not for the faint of heart. You need to deeply think about your architecture before you start, and the real downside is that sometimes you can\u2019t do all the cool stuff that each cloud provider gives you (e.g., cloud formation which works on Amazon but not Google). There are some tools that can help you be multi-cloud, like Ansible or Terraform, or Dockerizing everything to run everywhere.However, you can get credits from Google, then credits from Amazon (then you ask for a few more); you have account managers in both companies and can negotiate with the leverage of actually using","record_index":3},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"\u2026 the competing service (this of course does require having some volume in production).\nGo bare metal. Many past disadvantages of bare metal are going away; bare metal companies like LeaseWeb and OVH now have APIs \u2014 you can create a server or replace a broken hard drive on your machine without talking to anyone. The prices are significantly lower.\nWe heavily use bare-metal servers and you can read more about it here.\n\nBonus tips\n\nEc2instances.info This is a great tool that helps you see all your instances in AWS, including prices for different regions; you can compare selected services. It helps you see things clearly in one place. Note: while it updates regularly, I would always double check the information.\nKeep learning: the Open Guides for AWS is updated daily and a fantastic place to learn; including their billing and cost section that has many of the things I talked about and more.\n\nCompute price comparison\nLet\u2019s take the example of a pretty strong machine: AWS EC2 M4.10XLARGE - [40vCPU, 160GiB RAM, 10Gig Network], and compare its prices with other possibilities and different optimizations (the table is sorted by provider).\n\n\n\n\nProvider Machine Type\nPrice \/ Month\nComment\n\n\nAWS - OnDemand\nm4.10xlarge\n$1,460.00\n\n\n\nAWS - Reserved 1 Year\nm4.10xlarge\n$904.47\n\n\n\nAWS - Reserved 3 Years\nm4.10xlarge\n$630.72\n\n\n\nAWS - Spotinstnce\n\n~$447.696\nPrices change very frequently\n\n\n\n\n\n\n\n\nGCP - Sustained Price\nCustom Instance\n$1,054.00\n\n\n\nGCP - Upfront\nCustom Instance\n$873.77\n\n\n\nGCP - Preemptible\nCustom Instance\n$314.40\n\n\n\nLeaseWeb\nR730XD\n$374.99\n2x 10 cores\n256GB DDR3 RAM\n2x480GB SSD\n10 TB traffic\n\n\nOVH\nMG-256\n$365.99\n20 cores\n256GB\nDisks 2x2TB\n\n\n\nThe first thing you can easily spot is that bare metal providers such as leaseweb and ovh will provide the best value for the buck, and they also include storage and traffic in the same package. You can also see that those bare metal providers are much less flexible in terms of machine types.\nAnother thing we need to consider is that","record_index":4},{"post_id":7203,"post_title":"Tips for Reducing the Cost of Your Infrastructure","post_date":1513759287,"post_date_formatted":"Dec 20th 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-infrastructure-cost\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_cost-of-infrastructure-360x200.png","time_to_read":10,"content":"\u2026 in a cloud environment we can pay for only what we use, so if we need a machine for an hour a day, we actually don\u2019t have to pay a monthly fee, and this might reduce costs dramatically, especially if we use spot instances or preemptible instances.\nHere at Algolia we actually chose a mix of providers. Using bare metal for the Algolia engine and API was the best decision for us, but we also use Google Cloud Platform for our log processing and analytics, and AWS for many different production and internal services.\nThe bottom line is that, as always with building and maintaining a robust infrastructure, you need to choose what\u2019s best for your company and your use case. Hopefully, tips above will help you make the right choices at the right price. Have other tips? We\u2019d love to hear them: @eranchetz, @algolia.","record_index":5},{"post_id":7166,"post_title":"Introducing TalkSearch \u2014 making videos searchable, one conference at a time","post_date":1513273974,"post_date_formatted":"Dec 14th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/talksearch-conference-video-search\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_TalkSearch-01-360x200.png","time_to_read":5,"content":"It\u2019s time for one of our favorite annual traditions: revealing the holiday gift we built for the developer community. This year, we\u2019ve made a tool that helps users discover key moments at conferences by searching into video transcripts. It\u2019s called TalkSearch, and we\u2019d like to share a few details about why we built it and how it works.\nWhat is TalkSearch?\nTalkSearch is an interactive search and video experience that any conference or meetup organizer can offer to their community. All the event needs is a YouTube channel or playlist to get started. TalkSearch indexes video titles, descriptions and transcripts and serves up an instant search experience that plays the right video at the right time based on the search results.\nConference and meetup organizers are invited to fill out the TalkSearch request form and get the process started. Once the indexing is complete, a standalone video search page will be built just for that conference. The search experience can then be embedded on the conference\u2019s website (coming soon).\n\nPast community holiday gifts\nEvery year since 2014, Algolia has built something to address a pain point or opportunity in the dev community. Here are a few past examples:\n\n2014: GitHub Awesome Autocomplete, with recently improved UX\n2015: \u00a0DocSearch, now used by 450+ open source documentation sites and API portals, including Stripe, webpack and React\n2016: Yarn package search, now at 700,000 user searches per month!\n\nThe inspiration for TalkSearch\nWe wanted to give back to an especially important community for Algolia\u00a0\u2014 the organizers of events. We at Algolia go to over a hundred events every year, whether as sponsors, speakers, or participants. We benefit greatly from the communities that conference organizers bring together and the opportunities they afford us.\nWe\u2019re also scratching our own itch here. When we return from a conference, we often want to watch talks again or see the ones that we missed. Or we want to leap right to the","record_index":0},{"post_id":7166,"post_title":"Introducing TalkSearch \u2014 making videos searchable, one conference at a time","post_date":1513273974,"post_date_formatted":"Dec 14th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/talksearch-conference-video-search\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_TalkSearch-01-360x200.png","time_to_read":5,"content":"\u2026 moments that were the most relevant to us. We might also want to share with our co-workers a particular moment in a talk or a particular slide. TalkSearch was conceived and built to make this process easier, and to help unlock more value from all of the time and energy that goes into writing talks and producing videos.\nHow does it work?\nAs with any Algolia implementation, TalkSearch can be broken down into two main parts: the indexing process (crawling or scraping the data) and the search interface that users will interact with.\nIndexing - Youtube API\nTalkSearch uses the YouTube API to loop through a conference\u2019s channel or playlist of videos. It extracts the essential information from each video \u2014 title, author, transcript (YouTube \u201ccaptions\u201d), tags \u2014 and pushes that data into the index.\nEach conference\u2019s TalkSearch data is structured as follows:\n\nThere is one record per \u201cphrase\u201d of the talk, all stored in one index.\nEach phrase contains between 5-10 words on average.\nEach phrase has a start time and duration.\nThe title, author, and description of the video are added to each record.\n\nHere\u2019s what an example record looks like:\nView the code on Gist.\nSearch - standalone or embedded; single or multiple videos\nThe TalkSearch user interface is hosted automatically on a standalone page by Algolia when a new conference is indexed. We do recommend, however, that conferences use an embeddable version to put it directly inside of their site, which will create a more seamless experience for their users.\n\nWhen the user first lands on the standalone page or embedded widget, they can search through all videos at once. Up to 3 locations in each video will be shown that match the search query. When the user clicks a search result, a full-size panel will appear containing just the video selected, which will start playing at the right start time according to the transcript data. Within this view, the user can search into just the current video to find other moments","record_index":1},{"post_id":7166,"post_title":"Introducing TalkSearch \u2014 making videos searchable, one conference at a time","post_date":1513273974,"post_date_formatted":"Dec 14th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/talksearch-conference-video-search\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/12-2017_TalkSearch-01-360x200.png","time_to_read":5,"content":"\u2026 of interest.\nOpen source on GitHub\nThe search is built with the React InstantSearch library. React InstantSearch and the family of InstantSearch libraries provide a set of components and building blocks that make it easy to build dynamic, full-page search experiences like TalkSearch.\nAll of the TalkSearch code is open source and you can also run it on your own. See algolia\/talksearch-scraper and algolia\/talksearch on GitHub. Whether you\u2019re a new developer or an experienced pro, we welcome your feedback and contribution in the form of issues and PRs.\nEnjoy!\nWe sincerely hope you\u2019ll find TalkSearch useful as a conference-goer or organizer. If you\u2019d like to create a TalkSearch experience for your event, please start by filling out the request form.\nWe would love to get your feedback to make TalkSearch the best tool that it can be! Say hello on the TalkSearch page on ProductHunt\u00a0or tweet us at @algolia.\nHappy holidays!","record_index":2},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"Geo-spatial search or geo-search is no longer a buzz word or a nice to have in your service or app. If you take a look on the AppStore \/ PlayStore, more than half of the apps will ask your permission for location.\nAnd here's why.\nLocation matters\nAccording to Google's 2011 \"The Mobile Movement Study\", 77% of smartphone users use their smartphone for search.\nWhen it comes to location, things are getting even more interesting. 95%\u00a0of\u00a0smartphone users have looked for local information. After finding the information:\n\n77% contacted the business after \u2014 61% called and 59% visited the location\n44% made a purchase, 26% being online and 36% in the store\n\nWithin a day, 88% would take an action (visit the place, buy, call, etc.)\nLocation matters! And not only for buying food or dining but for the way we interact with friends and even strangers.\u00a0Apps for dating, transportation, social networks and media all leverage location for better content.\nDo all those apps really need our location for their services? Maybe not, and I am not suggesting we all start draining the battery and fetching the location without purpose, especially with GDPR in Europe. Rather, I am suggesting to devs developing apps to think twice about whether they could improve their search by taking location into consideration.\nHopefully, by this point I convinced you that location matters. Let's see in the next parts how to build geo-search interfaces leveraging user's location for better mobile experiences.\nMy use case\nBefore joining Algolia, I tried my luck and built several startups. One of those startup\u2019s goals was to imagine a new way of interacting with people: putting more emotions into the photos or videos we are taking, and, like Hansel And Gretel, leaving a trace of our moments all over the world.\nThe use case was simple: take a picture or video. Instead of sending it to your friends, leave it in the same physical place you took it. The message could be public and hence seen by anyone around","record_index":0},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"\u2026 the place, or private, visible to selected friends.\nAs an example, let\u2019s say you are in a bar and took some very cool selfies (or at least that's what you thought after those five beers). You open the app and post them, so now, everyone around that bar can see your awesome selfies. When your friend, George, who skipped this Friday\u2019s beers comes around next time, he can see how happy you were.\n\nChoosing the dev stack\nAs the tech person in the startup, I was responsible for building an MVP (Minimum Viable Product), which was an iOS app.\u00a0My use case was simple: being able to search for messages around a geo-location; given latitude \/ longitude and a radius, show the messages around the place in question.\nSo, I started a Google search for the following terms: geo-search database, geolocation databases, geospatial database. There weren\u2019t many options popping up, but I did find a couple of solutions: MySQL would work, and PostGIS was a more powerful solution.\nSince I had some experience with Firebase, I learned that I could use GeoFire on top of it, which allows to store and query a set of keys based on their geographic location.\u00a0In case you know nothing about Firebase, it is basically a backend as a service. They have a lot of cool stuff integrated, like authentication, realtime database, storage and so on.\u00a0In a nutshell, for every message sent, I would save an ID and its location with GeoFire.\nWith this in mind, and the fact that I was building an MVP, using Firebase + GeoFire was much faster than building an entire back end with PostGIS.\nFirebase and GeoFire\nAt its heart, GeoFire stores locations with string keys. Its main benefit, however, is the possibility of querying keys within a given geographic area. GeoFire stores data in its own format and its own location within your Firebase database.\nSo, in my case, for every message I would store in Firebase, GeoFire would store another object containing the ID of the message and its coordinates.\nFor our app,","record_index":1},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"\u2026 every time the user was opening the app or moving around, we would fetch his location and start a search.\u00a0Given the latitude, longitude and a radius, we would listen to each of the messages that was in that area.\u00a0Because Firebase is implemented as an Observable, each message would come one by one, every time matching the query. So, in the end, we would have a list of messages IDs with their specific location.\n\nFirebase + Geofire limitations\nEven though Firebase is super fast and reliable, we encountered some limitations with GeoFire.\nDisplaying rich messages on a map or list\nGeoFire will only store the ID of a message and its coordinates; you cannot store any other metadata. When displaying your data on a map you will then have to:\n\nGet all the messages and coordinates in a single request\nMake other requests to Firebase to get the other metadata, like title, image url...\n\nThis approach might be OK when you have 5-10 messages around you, but when you have hundreds, it will kill your network.\nFiltering\nOne of the biggest pain points was filtering. On the map, you can see public messages and messages from your friends. Since GeoFire does not support any kind of filtering, we had to filter the results on the client side. If a user only wants to see messages from her friends, we would have to download all the messages around and filter out the ones that are public. Since the ratio was 1 to 100, we were downloading 99 messages for nothing.\nOnly latitude and longitude queries\nOne of our use cases was to show only messages around a certain area. Let's say you only want to see the messages at your school and nothing more \u2014queries in a polygon were out of question.\nTo sum it up, Firebase + GeoFire can be a very useful and quick solution to your geo-spatial search problem. You might have some limitations, but overall, it works.\nAlgolia and geo-search\nLess than three months ago I started working at Algolia, on the mobile team. My ramp-up project was developing an","record_index":2},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"\u2026 Airbnb-like search experience using Algolia\u2019s geo features (here's the code) which would display available rooms around you, on a map:\n\nI start reading the documentation and dove into the features \u2014\u00a0to be completely taken by surprise. Where was this when I needed it eight months ago?\nHere's what I found out, compared to the limitations I encountered with GeoFire.\nDisplaying rich messages on a map or list\nAfter indexing your object and in order to search based on a location, all you need is to add a field _geoloc with given latitude and longitude. That's it!\nSo, when querying for objects around a location, you will get the entire object. This means that, when displaying it on the map, you can leverage all the attributes of the object. For my use case, I could have, for example, displayed the image and name of the person who left a message.\nOf course, keep in mind your customer. Downloading large datasets will impact the speed and network consumption. To offer the best experience, you could:\n\nlimit the amount of results you are displaying at each iteration\nuse attributesToRetrieve to get only the attributes you are interested in. This is very helpful when your object contains attributes that are not useful in your search context\n\nIn my case, I would limit the attributes to only the location, the user's image URL and his name. Anything else, like the content of the message, comments, and so on, would be lazy-loaded if needed.\nFiltering\nFiltering was my biggest pain point with GeoFire. Not being able to search for specific type of messages was almost a deal-breaker. With Algolia's system, the location is only one of the ways you can filter. You can add additional filters like whether the message is public or private, if it is a video or image, and so on.\nSo, instead of downloading 100 messages for only one relevant message, I was able to only fetch what was truly relevant.\nLimit the result set\nThis feature applies to Algolia's search in general: you can set up the","record_index":3},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"\u2026 limit and download batches of that limit: check out infinite scrolling.\nNot only latitude and longitude queries\nIn some cases you need more than just results around a position. Maybe you want to display results in a certain area. Instead of using the latitude and longitude you can define a rectangle or a polygon. This way your results are bound to that region.\n\nOther features worth mentioning\nMultiple locations for the same object\nYou can add a list of locations for the same object. If it makes sense for your use case, instead of replicating the same object, add the locations where the object is available.\nAutomatic radius\nAutomatic radius is very useful when displaying results in areas that have too many results or too few.\nBy default, Algolia will expand the search radius around the specified latitude and longitude until approximately 1000 hits are found. If the area is dense, the radius will be smaller. If it\u2019s not dense, the radius will be bigger. The benefit of this feature is that it increases the chance of finding more results in low density areas. If a fixed radius is set, there could be fewer\/no results returned.\nResults around user's IP address\nSometimes the GPS is not an option. Maybe the user blocked the access or it is just not working. You can always fall back to displaying results based on their IP address. Algolia will associate a location based on the user\u2019s IP address and search around that location. Here's how.\nConclusion\nWhether you are working on the next big thing or want to improve your product, keep an open mind about the user\u2019s location.\nThe goal of this article was to:\n\nmake you aware of the importance of geo-search\nsince Firebase and GeoFire was the top suggestion I got when looking for a solution, I wanted to give you an overview of features and tradeoffs\npresent Algolia\u2019s approach when tackling geo-search\n\nBy the way, in case you didn\u2019t know, Algolia and Firebase play nicely together. In case you have geo-aware data in","record_index":4},{"post_id":7176,"post_title":"Geo-Spatial Search on Mobile: Quick but Not Dirty","post_date":1513074623,"post_date_formatted":"Dec 12th 2017","author_name":"Robert Mogos","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/bf377ce26c20a343f918f016dcad85eb?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/geo-spatial-search-on-mobile\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/11-2017_geospatial-search-mobile-360x200.png","time_to_read":9,"content":"\u2026 Firebase, you can sync it to Algolia to get these more advanced geo-search features.\nHave you played with geo search? Have cool tips or feedback on this article? We want to hear it: @robertmogos,\u00a0@algolia.","record_index":5},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"Git `rebase` is one of those commands that you may have heard of as a substitute for `merge`. Truth is, it\u2019s more than that \u2014 `rebase` is a completely different set of tools that intersect with the goals achieved by `merge`. Confused? Don\u2019t worry! This blog post is about different uses of `rebase`. First, as a way to integrate and exchange your work between two branches (like `merge` does), but also as a history rewriting tool.\nFrom merge to rebase\nMerging branch is the most common way to integrate changes between two Git branches. A Git workflow common to services such as GitHub or Gitlab is as follows:\n\nCreate a new \u201cfeature\u201d branch called `my-new-feature` from a base branch, such as `master` or `develop`\nDo some work and commit the changes to the feature branch\nPush the feature branch to the centralized shared repo\nOpen a new Pull Request for `my-new-feature`\nWait for your tests to pass and to gather feedback from your peers\n\nFrom there, everything is great. You end up with a nice, clean branch, such as:\n\nHowever, in an imperfect world, here\u2019s what might come next:\n\nCode reviewers have found a few bugs and typos in your first commits and your tests are not passing\nYou make some changes and commit the fixes locally\nYou push your updated feature branch to the centralized shared repo \u00a0(C6 and C7 on the following schema)\nMeanwhile, other commits are merged to the base branch (C8 and C9)\nYour pull request finally gets accepted and is merged into the base branch (C10)\n\nAnd from there, your history becomes a bit more complex:\n\nThere\u2019s nothing wrong with this workflow; in particular, you don\u2019t have to bother about what your coworkers are doing so you can focus on your own work. Since the critical part where Git combines (or merges) your changes with the ones from the base branch only happens once, you will only need to deal with eventual conflicts once \u2014 at the merge step.\nHowever, a few things are also a bit off here. First of all, if you\u2019re working","record_index":0},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 on your branch long enough, you may end up out-of-sync with the base branch for days or weeks. This may not be an issue, but sometimes you would have really appreciated including that specific fix that your team merged, or getting rid of that huge dependency that slows you down every time you compile. Secondly, the history may become too complex to understand once all your coworkers have merged \u00a0their own branches to the base branch. Lastly \u2014 and this one may be a bit more subjective \u2014 you probably kept a logical breakdown between the commits of your branch. Having a single merge commit containing all the changes to all your files probably isn\u2019t what you want to expose in the end.\nLet\u2019s see how rebasing may help you address all those issues.\nRebasing on the base branch\nIn September 2016, GitHub introduced a new way to merge pull requests: the \u201cRebase and merge\u201d button. Also available for other repository managers such as GitLab, it\u2019s the \u201crebase front door\u201d. It lets you perform a single rebase operation of your Pull Request commits on top of your base branch and then perform a merge. It is very important to observe that those two operations are performed in order, and that the rebase is not a substitution of the merge. Hence rebase is not used to replace the merge, but it completes it.\nConsider the previous example. Before the final merge, we were in this situation:\n\nYou can simulate what happens when you click on the \u201cRebase and merge\u201d (when there\u2019s no conflict) by performing the following commands:\nView the code on Gist.\nBy doing so, you finally end up with a \u201clinear history\u201d:\n\nAs you see, rebasing is not a substitution for the merging step. As explained before, the two operations are not performed on the same branch: `rebase` is used on the feature branch whereas `merge` is performed of the base branch. For now, this operation just prevents having a single merge commit with all the changes in it, and it\u2019s still a single operation","record_index":1},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 that happens at the last step of your contribution (i.e., when you want to share your work).\nUntil now, we have only interacted with `master` as our base branch. To stay in sync with the changes of the base branch, it\u2019s just a matter of performing the rebase step with the up-to-date base branch. The longer you wait to do this, the more out of sync you\u2019ll be.\nThe up-to-date version of your base branch is hidden in plain sight. It\u2019s a read-only version of the base branch, prefixed with the name of the remote to which you\u2019re connected, or more simply put: it\u2019s a read-only copy of the branch from your remote instance (such as GitHub or GitLab). The default prefix when you are cloning the repository for the first time is `origin`. More concretely, your `master` branch is the local version of master, whereas `origin\/master` is the remote version of this branch, copied on your computer the last time you performed a `git fetch` operation.\nWe've stepped through a lot of theoretical material, but as it turns out, the end result is relatively straightforward; here's how to sync with changes happening on the remote:\nView the code on Gist.\nThe first step retrieves the latest changes from the distant copy of `master` into your local `origin\/master` branch. The second one checks out your feature branch. The last one \u00a0performs the `rebase` so that all your commits are now added on top of the latest changes that happened parallel to your own work. By applying those commands on our very first example, here is what would have happened:\nAs you can see, the feature branch now includes all the latest changes, so you\u2019re able to work in sync with the rest of your team. By working with the above workflow, you\u2019ll be able to deal with potential conflicts earlier and progressively instead of at the very last moment (when you want to merge your work within the base branch). People often disregard the \u201cRebase and merge\u201d button because they expect too many conflicts at the very","record_index":2},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 last step of the process (so they prefer to perform a regular merge commit instead). Ultimately, it takes a small active effort to stay in sync with the latest changes.\nRebasing your own work\nUntil now, we\u2019ve only used `rebase` to apply commits from one branch onto another. This is pretty much the basic use-case of `rebase`: just default options, actions, and results. Furthermore, we are only using `rebase` to integrate changes from a different branch into our own branch. But \u2014\u00a0it can be used to add\/change\/remove your commits directly from your own branch too! The \u201cbase\u201d on which you rebase can be virtually any commit \u2014 even a direct ancestor.\nIn fact, if you wanted to see what was happening during the rebase we did, you could have used the \u201cinteractive mode\u201d of rebase by adding the `-i` or `--interactive` argument. By doing so, Git will open your editor of choice (the one defined in your `EDITOR` environment variable) and list all of the commits that will be affected by the rebase operation, and what should be done with every single one of them. This is where the true power of `rebase` lies.\nFrom your editor, Git lets you reorder, rename, or remove commits, but you can also split single commits into multiples, merge two or more commits together, or change their commit messages at the same time! Pretty much everything you\u2019d like to do with your history is possible with `rebase`. And the awesome thing is that it\u2019s relatively straightforward to tell Git what to do. Every commit is presented on its own line, in a sequential order, prefixed by the command that will get applied to it. Reordering commits is as easy as reordering lines, with the most recent commits at the bottom of the list. Removing commits is just a matter of removing the corresponding line, or specifying the `d` or `drop` command as a prefix. One of your messages contained a typo? Just use the `r` or `reword` command to keep the commit, but change the associated commit message.\nTo sum","record_index":3},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 up, `rebase` is just a Git command that lets you:\n\nSelect one or multiple sequential commits\nBase them on any commit of your repository\nApply changes to this commit sequence as they are added on top of the new base commit\n\nTo better illustrate this, consider the following series of commits:\nView the code on Gist.\nAs you see here, we have a first \u201croot commit\u201d \u2014 which will serve as our base commit \u2014 followed by 4 commits adding a total of 5 files to the repository. For the sake of the exercise, let\u2019s say this series of commits is your Pull Request, and you aren\u2019t satisfied with it for the following reasons:\n\nThe first commit message is wrong: it should be \u201cadd A file\u201d, not \u201cadd A\u201d\nFile B and C were added in the wrong order\nFile D should have been added at the same time as file C, not with file E\nFinally, file E should be added in its own separate commit\n\nAll of those changes can be performed with a single rebase. The final history would look like this:\nView the code on Gist.\nNote that, except for our base commit, all commit hashes have changed. This is due to the way Git generates those commit hashes, which are not only based on the changes themselves, but also on the parent commit hash and other metadata.\nAnyway, let\u2019s rebase!\nLet\u2019s start with a `git rebase -i HEAD~4`. This tells Git to interactively rebase the last 4 commits from HEAD inclusive. `HEAD~4` points to the \u201croot commit\u201d which is the commit upon which we will rebase. After hitting ENTER, your editor of choice will open (or `vi` by default on Unix-style systems). Here, Git is simply asking you what you want to do with the commit you performed.\nView the code on Gist.\nAs explained earlier, every line represents a single commit, prefixed by the corresponding rebase command that will get applied. All the commented lines are ignored during the rebase and are here to remind you of what to do now. In our case, we will go with the following commands:\nView the code on Gist.\nHere, we","record_index":4},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 told Git to perform three tasks for us during the rebase:\n\nStop at the first commit to let us change the commit message\nReorder the second and third commit to have them in the correct order\nStop at the last commit to let us do some manual amending\n\nUpon saving the file and quitting your editor, you\u2019ll be presented with your editor again, and the first commit message will be in front of you. The rebase is happening and you are being prompted to change the first commit message. Let\u2019s change it from \u201cadd A\u201d to \u201cadd A file\u201d, then save and quit.\nThe reordering of the second and third commits is done transparently by Git. This leaves us with the last amend we asked to perform. Here, we\u2019re stopped after the \u201cadd D and E files\u201d commit. As we wanted to create a single commit with C and D files and a new one only for E, we need to perform the following steps as if we were amending additional commits on the top of our branch:\nView the code on Gist.\nThese commands (except the last one) make the \u201cadd C file\u201d and \u201cadd D and E files\u201d commits become the \u201cadd C and D files\u201d and \u201cadd E file\u201d commit we wanted to have. The last command, though, is just to inform Git that we\u2019re done with the `edit` step. After that, Git will happily tell you that the rebase finished successfully. Great!\nWe\u2019ve covered pretty much everything you might like to do with your commit history. A few more commands are available and may help you better depending on your use cases.\nHandling conflicts\nWhen it comes to using `rebase`, people are often confused about the way to fix conflicts that may happen when rebasing a branch on top of another one. It\u2019s actually convenient to fix conflicts when they do arise with Git, for multiple reasons.\nFirst, when a conflict arises, Git doesn\u2019t try to be smarter than you \u2014 it stops the current `rebase` and asks you to fix the conflict. The conflicting files will be marked \u00a0as \u201cboth modified\u201d and the conflicting sections will","record_index":5},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 have some markup to help you find what differs. When you\u2019re done with the modifications, you then `git add` the modified files and run `git rebase --continue` to let the rebase continue.\nSecond, there are two tools that are very powerful when you are not confident with an on-going rebase, or with a rebase that went wrong. Consider `git rebase --abort` which \u00a0rewinds history to just before the current Git rebase operation.\nWith these techniques, changes made using `rebase` can be undone, so the risk of impact of making mistakes is minimal.\nFinally, you might find yourself dealing with a long and boring conflict to resolve and even then, it\u2019s possible that the same conflict happens again at a different time. For example, it is common, unfortunately, for the base branch to change while you were working on your own branch. Another scenario is that you aborted a rebase and are now attempting to redo that rebase. To avoid resolving the same conflict again, Git provides a solution which is disabled by default. This feature is named \u201creuse recorded resolution\u201d or \u201crerere\u201d and can be enabled with: `git config --global rerere.enabled true`. This way, Git will keep track of all the conflict resolutions you perform. When the exact same conflict happens again, you should see from Git outputs that a recorded resolution was used.\nGoing further\nI hope this article helped you to see what\u2019s possible with the `rebase` command. Of course, the best way to learn Git is to use it, but the second best way is to read about it. If you\u2019d like to read more, I highly recommend the Pro Git book - specifically the section about rebase itself. And, because once in a while we all end up in a bad situation, you should probably take a look at Data Recovery in the Maintenance and Data Recovery section. If you\u2019re not into reading the whole documentation, maybe you\u2019d prefer these Git Flight Rules.\nHave additional rebase tips or feedback on this post? We\u2019d love to hear them:","record_index":6},{"post_id":7150,"post_title":"Master the Rebase (and the Other Way Around)","post_date":1512562283,"post_date_formatted":"Dec 6th 2017","author_name":"Anthony Seure","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/1f3796cf19a76dafe544f2cf4541bb49?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/master-git-rebase\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/12\/2017-11_Git-Rebase-360x200.png","time_to_read":12,"content":"\u2026 @aseure or @Algolia. Thanks for reading - and happy Gitting!","record_index":7},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"A challenge that documentation writers face is how to encourage developer readers to stay in the documentation and not reach out to support until absolutely necessary. To meet this challenge, writers of course focus on the text\u00a0\u2014 its clarity, coherence, and exhaustivity. They also focus on form, devising ways to guide developers to the relevant text \u2014 structuring the information, highlighting key words, hyperlinking, creating menus and submenus, thinking up cool titles. Algolia adds search to that effort.\nAlgolia's new integrated search panel\nFrom browse to search\nDevelopers come to documentation seeking answers, hoping to find what they are looking for. They browse, read, and browse a bit more. Often they find what they need, but not always. Some will eventually contact support for more personalized guidance, which may send them back to the documentation, but this time to the exact paragraph or code sample they were looking for. \nAlgolia\u2019s documentation is about search\u00a0\u2014 to wit: how to use our search API. So, we thought: if our users can\u2019t always find what they need using our own search bar \u2014 and worse, to learn later that what they were looking for was actually present in the documentation\u2014 what sort of message were we sending about our API?\nSo we\u2019ve faced this challenge head-on with an example of Algolia\u2019s powerful search engine \u2014 by expanding our current search bar into a fully-integrated search panel.\n\nThe new search panel is designed to be prominent and conversational, so that whenever our developers ask themselves What is or How to or Why, they simply type the remaining part of their question in the search bar, and our new panel lights up with the answers they are looking for.\nAdopting best practices\nOverall, our UI\/UX model was Google. We adopted a Google-like feel, but used our own search engine + the knowledge of our own documentation to drive the whole user journey from question(s) to answer(s).\nWe also believe that search is a","record_index":0},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 bridge between customer support and documentation. That\u2019s why we included support discussions from our public forum in the new search panel. Now, when you search our documentation you\u2019ll also be looking into our support history. That way, you get a side-by-side view of all relevant information about our API \u2014 relevant texts, code snippets, and all support tickets.\nThis time our model was Google + Stack Overflow, that well-known dynamic duo that has saved every developer from the great unknown. Stack Overflow, and more generally community-driven support, have become essential to the developer experience. By integrating our own developer community into our documentation, we will be giving our developers that same standard\u00a0\u2014 and maybe even more, given that we know our own support data and can therefore fine-tune the search results.\n\nFinally, taking this Google\/Stack Overflow model a bit further, we decided to display code samples in the search results. Many developers come to our docs with a very specific question in mind; for them, finding a well-written line of code is often the best, most direct answer. So we added a toggle button to switch between text and code, allowing developers to search only for code.\n\nWith these features in place \u2014 a prominent search panel, integrated support, and code searching \u2014 we hope to extend the trust with our readers, so that they keep coming to our documentation expecting a useful experience. \nWe are also backing up our efforts with analytics: real metrics that will help us follow developers from query to query, page to page, and even from support to documentation. That kind of feedback loop will tell us how we can shorten the reading process and make it more pleasant, and it can also indicate how we can encourage our doc readers to use more advanced features, to push our API to its limits, which benefits everybody. \nAnd we won\u2019t stop at analytics. Because the challenges \u2014 to write clear, coherent, exhaustive, and","record_index":1},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 easy-to-find information \u2014 will never go away, we will need to keep improving by focusing on different kinds of search strategies that work particularly well for natural language documentation.\nSearch in more detail\n...or more specifically \u2014 what strategies did we use to ensure that our readers find what they are looking for?\nIn a nutshell: a successful document-based search relies in large part on how you organize your content. Global sections, individual pages, headers \/ sub-headers, and paragraphs \u2014 these are only some of the textual elements that, when done consistently and ordered logically, matter a lot. In our case, with a well-thought and cohesive collection of texts, Algolia\u2019s speed and relevance work out of the box. \nAnother focus is on the query itself. The search engine can, for example, behave differently depending on the specificity of the query: for simple, general queries (like \u201cindex\u201d or \u201cinstall\u201d), the content can stay high-level. For longer or more precise queries (like method names, or longer sentences), we can switch the results into more low-level API references and concepts.\nSearching structured data\nLet\u2019s look at what Algolia does best \u2014 searching structured data. Here is an example of a T-shirt inventory. If the user is looking for a \u201cslim red T-shirt with nothing on it\u201d, you can help them find it by filtering:\nType: T-shirt\nColor: red\nDesign: blank\nType: slim\nIf the user types in \u201cT-shirt\u201d, they get the whole inventory (1M records). If they add \u201cred\u201d, you divide the 1M t-shirts by 5 (let\u2019s say there are 5 colors). If you add \u201cslim\u201d, you divide by 3 (there are 3 types: slim, wide, and stretch). If you start adding other criteria - like \u201cmidriff\u201d, \u201csleeveless\u201d, \u201cmulti-colored\u201d, and so on, you could conceivably reduce the relevant stock to 25 t-shirts. Not bad, from 1M to 25! And a good UI would make this process as easy as possible for the user.\nAll this works as described when the content in","record_index":2},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 which you are looking contains very clearly defined items. The discrete clarity of commercial products is what lies behind the success of structured data searches. \nBut not everything is so discrete. English language has an unlimited number of ambiguities, so creating categories for natural language is not a scaleable solution.\nUnstructured text \u2014 from search to research?\nLet\u2019s now take a look at two queries which make for a difficult search structuring as described above. \nExample 1\u2014 legal text\nLet\u2019s switch subjects to better illustrate the point. Let\u2019s say a lawyer types \u201cout of order\u201d in a legal database that contains court cases, laws, and legal journals. For this query, there are at least 4 relevant categories of documents, with each category containing 1000s of documents:\n\nA database can be \u201cout of order\u201d, causing system failure (computer law)\nAn elevator is \u201cout of order\u201d, causing monetary loss (commercial law) or personal injury or death (torts law)\nA lawyer is \u201cout of order\u201d in a courtroom (procedural law)\nA factory produces widgets \u201cout of order\u201d, breaking contractual terms (contract law)\n\nThe lawyer clearly needs to signal to the search engine which of these categories is relevant.\nExample 2 \u2014 API documentation\nIt would be the same if a developer were to come to Algolia\u2019s documentation and search for \u201cindexing filters\u201d and find two categories of documents: : \n\nthe back end (creating filters in your data) \nthe front end (filtering at query time)\n\nand four formats:\n\nconcepts, tutorials, code snippets, API reference\n\nThe developer will want to have control over both the subject and format of the documents retrieved. I\u2019ll use the term \"researchers\" for our confused lawyers and developers above.\nAs-you-think\nLet\u2019s go back to the T-shirt example to see if that can help here. That example was about one item: the consumer is searching for one thing, and the quicker they find it the better.\nThe other extreme are","record_index":3},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 researchers: researchers are often not looking for one thing. Their query is to think about a subject, to get a better understanding and to construct and support an argument. They have to be patient. If they are searching a site with 1M documents, they are ready to scan through 1000s of results (in say one or two hours, or days, or longer), and to read 100s of documents. We are clearly not talking about consumers. \nDevelopers fall somewhere between these extremes. Sometimes they know more or less what to look for and so are searching for one thing \u2014 for example, a method or a specific setting. Other times they don\u2019t really know what they are looking for: they might be onboarding, or trying to solve a difficult problem, or looking to enhance their current solution. In this case, they are more researcher than searcher.\nBut even here, we don\u2019t want to waste a researcher\u2019s time with irrelevant results. And we surely don\u2019t want them to fail by not presenting them with important results (this is the difficult balance of precision and recall). \nEssentially, we want researchers to have the same quality results\u00a0\u2014 and the same confidence in Algolia \u2014 that our consumer clients have.\u00a0\nAnd so the challenge is clear. How do we structure our \u201cunstructured\u201d documentation to come up with consumer-grade results?\nSearch strategies for documents\nBehind our new search panel \u2014 what we\u2019ve done\nAlgolia\u2019s power lies in structuring the data before searching. To put this into action, we focused on four key areas:\n\nOrganizing content \u2014 the way that our documentation is organized (within the page as well as the combination of all the pages) is probably the most important step\nIndexing \u2014 structuring the text for search and relevance\nRelevance \u2014 testing out numerous queries and engine configurations, to ensure a consistently good relevance\nUI\/UX \u2014 the developer experience: how to encourage our readers to use and to keep using our integrated search panel.","record_index":4},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 (Although this is of equal importance, we do not describe how to implement the InstantSearch UI)\n\nOur indexing and relevance strategies follow our DocSearch methodology, which has been well documented by our CTO in a previous blog post on The Laravel Example. There he describes:\n\nHow to structure your texts with sections, headers \/ subheaders, and small paragraphs\nHow to order the overall content so that that some information is more important than other\nHow to tweak our tie-breaking algorithm using customer ranking (again, relevance)\nHow to configure Algolia with filters, facets, synonyms, and many other settings. \n\nA recent feature not mentioned in the post is our extensive use of Query Rules to handle special use cases like specific products or coding language queries. \nThere is, of course, the matter of documentation tools. We have written about that in a separate post.\nExploring what\u2019s next\nSearching through thousands of documents is not an exact science. There are many pitfalls, and though we\u2019ve solved many of them, it\u2019s hard not to wonder: what happens when there are 1,000,000+ documents? Here are some interesting features not yet implemented.\nA word cloud filter\nAlgolia offers complete filtering out of the box, but we rely on our users to define the most effective filters for their information.\u00a0One way to do that is to use word clouds. Word clouds, in this context, are a set of filters that act as one global filter. For document-intensive searching, word clouds can be quite powerful. \nFor example, we can help resolve the above lawyer-researcher\u2019s \u201cout of order\u201d ambiguity by using word-cloud filtering: \n\nAs you can see, the four word clouds above match the four distinct areas of law mentioned in the \u201cout of order\u201d example. Normally, a filter is one word: by presenting a set of related keywords within a single frame\/word cloud, we offer the user more information to help choose the best filter. And by making these word clouds clickable (as seen","record_index":5},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 below), the user can think-by-clicking, to test which set of words most closely matches his or her train of thought.\n\nThere are many ways to build word clouds, one of which is to scan each document using specialized dictionaries, to pick out keywords that make the document unique \u2014 and to do this before indexing them. For the example above, you would use different specialized legal dictionaries. For our API docs, we would use our own API reference pages as dictionaries for each new document added to our documentation.\nThematic frequency\nSome documents are so similar in terms of relevance that it is impossible to know which should be presented first. At this point, the engine needs help. With structured data, such as shopping items, this is achieved through custom ranking, using metrics like \u201cmost popular\u201d or \u201cmost sold\u201d. However, using metrics is not always relevant for texts. For example, we can use \u201cmost cited\u201d or \u201cmost read\u201d, but these metrics are often irrelevant to a researcher. \nSo, why not create front end tools that help researchers \u2014 documentation readers themselves \u2014 choose between different ways to break ties? \nBelow is one such tool, which implements thematic frequency \u2014 a shortcut term to refer to the classification of documents by theme. Each document can be placed in one or more themes based on how close (or far) its content is from the theme. The themes are represented by word clouds. Documents can be scored using the thematic word clouds by matching the document\u2019s content with the keywords contained in the word clouds. Later, filtering can use that scoring to both find and order the documents. \nFor example, here\u2019s a subset of results for the theme \u201cserver-side indexing and filtering\u201d, in the order returned by Algolia:\n\nThe UI can offer the ability to switch between rankings:\n\nKeep the ranking returned by Algolia\u2019s tie-breaking algorithm. \nAdjust the ranking with Algolia\u2019s custom ranking feature, ordering the list","record_index":6},{"post_id":7111,"post_title":"API Documentation 2.0: From Browsing to Search","post_date":1512041184,"post_date_formatted":"Nov 30th 2017","author_name":"Peter Villani","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3b5a9996811a15b2e34b8270c3f26e75?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-from-browsing-to-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Doc-Search-Panel-360x200.png","time_to_read":15,"content":"\u2026 by \u201cmost read\u201d or \u201cmost cited\u201d.\nAdd thematic frequency on top of Algolia\u2019s ranking algorithm, reordering the results according to the strength of the document\u2019s relationship to the active theme. \n\nBy choosing the last option - thematic frequency - the researcher could reorder the results from (1, 5, 20) to (20, 1, 5), because record 20 contains the largest number of thematic keywords. In other words, document 20 goes to the top because it is more consistent with the theme of \u201cserver-side indexing and filtering\u201d than documents 1 and 5.\nThese bonus strategies, as well as many others, will keep us - and hopefully our readers - confidently within the conversational search powered by Algolia. \nWe look forward to your feedback on the effort we\u2019ve put in so far, and on future ideas: @algolia, @codeharmonics.","record_index":7},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"Black Friday & Cyber Monday are single-handedly the two most important days of the year for retail & e-commerce players alike. On Friday, American shoppers spent $5 billion dollars online \u00a0- that\u2019s 17% more than in 2016 - on a selection of deals that were too good to refuse. Whether in store or online, shoppers looking for flash deals are going to rely heavily on search & discovery in order to proactively find the best deal on headphones, for example, as well as to discover an item, brand or category that they might not have thought of initially.\nToday I\u2019m taking a look at search experience of five of the most popular retailers and e-commerce pure players. Starting with the Algolia Search Grader, I\u2019ll dive into the fundamentals of each site\u2019s search speed, relevance & design. Afterwards, I'll take a look at some of the interesting aspects of the search bar placement, the initial search experience, the results page & the search refinement experience.\nFor this review, I\u2019ve chosen in advance three keywords that I\u2019ll use (with and without typos): my queries can be categorized as Vague 'headphones', Long 'iphone case blue with red stripe', and Branded 'bose headphones'.\nBest Buy\nBest Buy knocked the Search Grader out of the park with a 100% score (here), meaning they have all the fundamental features we would expect from a great search experience - there are a few places where it was a close call, though. Best Buy is typo-tolerant, but it\u2019s not immediately visible in the drop-down menu. Instead, when you hit enter to see the results page for \u2018haedphones,\u2019 the page automatically displays results for \u2018headphones.\u2019 This may seem small, but if a user starts typing and see the dropdown menu showing nothing, they may get the impression that what they're looking for isn't available here (even if they've misspelled their query).Best Buy\u2019s search bar is a bit small, and the suggested queries before you start typing give you the","record_index":0},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 impression that you\u2019re in for a text-only drop-down menu, but once you start typing you immediately get both suggested queries and results - this is a great use of the drop-down menu providing both discovery (queries you might not have thought of) and finding what you\u2019re looking for (you can skip the search results page altogether).\nBrand queries like \u201cBose\u201d take you straight to the dedicated brand page, while Vague queries like \u201cheadphones\u201d give you a search results page that empowers you both to discover categories like \u201ctrue wireless\u201d that you may not have thought of (or thought existed). In addition, refinement of the query is very easy via facets on the left-hand side, which have counts and are searchable by Brand.\nThat said, Best Buy doesn\u2019t seem to handle my Long Query too well - I didn\u2019t get blue or red iphone cases, let alone a blue iphone case with a red stripe. Color identification is a great way to deliver more relevance to longer queries - no need to go full machine learning to interpret \u2018blue\u2019 and the color blue on a product.\nLike many retailers, Best Buy leverages Click & Collect with it\u2019s \u201cPick up Today\u201d tab, a different way of displaying a facet than in the left-hand bar. Best Buy\u2019s lack of slider for pricing isn\u2019t unique; however, we find that a slider for pricing adjustment, such as you might see on Airbnb, is a much more intuitive way to filter by price.\neBay\neBay came in at a 73% on the Search Grader (results here). Not providing results as-you-type, but simply query suggestions in the drop-down menu, makes a big difference in the UX when compared to Best Buy. The results are not instant, and displaying results for anything other than products, like Brands, is not nearly as visible as it could be.\nThe white search bar on a grey header is notably less visible than on most ecommerce sites, and eBay\u2019s dropdown menu only provides two suggestions: query suggestions and categories to search my given query inside","record_index":1},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 of. With so much blank space, eBay could display top results like BestBuy does and help shoppers who know exactly what they\u2019re looking for skip the search results page altogether. I liked the fact that empty search queries with a specified category in the right-hand drop-down menu took me directly to the category page; however, if eBay\u2019s search was a bit quicker, they could take me to the category page without me having to click-to-search in order to get there.\neBay struggled with my Long Query - I began looking for the New York Rangers case I found on other sites which perfectly matched my query and didn\u2019t find anything; however, with a few query tweaks I did find an appropriate FC Barcelona case and a Coach case that I would\u2019ve expected to show up give my query. eBay also struggles with query variance. If you\u2019re looking to buy a PS4, for example, the two accepted normal queries \u2018PS4\u2019 and \u2018Playstation 4\u2019 will give you structured results that encourage you to go to eBay\u2019s dedicated PS4 page; however, \u2018play station 4\u2019 and even \u2018playstation four\u2019 return ad hoc results that won\u2019t likely convert Cyber Monday shoppers looking for a discounted PS4.\neBay\u2019s left-hand facets could use a bit of work as well. Facet counts on the categories, search for facet values & replacing the pop-in modal when I click \u2018More Refinements\u2019 would improve the user experience here a bit.\nWhile eBay performs better on paper than some of our other sites, the quality of the experience is still rough around the edges.\nAmazon\nThe Search Grader gave Amazon a score of 70% (results here) - Amazon most notably gets docked hard for being slow to load results and requiring users to click to display results, instead of displaying instant search results as the user types. Instead, Amazon displays suggested queries in the dropdown menu bar, along with some options for category filters. Still, this is pretty common practice among top e-commerce sites today, and Amazon is","record_index":2},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 making a conscious decision here to make their desktop and mobile drop-down search look the same.\n\nAmazon\u2019s home page makes search pretty prominent, displaying a white search bar on a dark header, meaning the search bar doesn\u2019t have to take up a lot of real estate. Amazon\u2019s home page is very geared towards discovery, displaying Black Friday deals by category, recommended deals, hot deals. It\u2019s deal-city!\nAmazon displays a category drop-down menu directly, but selecting a category doesn\u2019t show you results and hitting enter without a query doesn\u2019t take you to the well-designed category page, but instead displays top results in that category for an empty query. \u00a0Amazon\u2019s drop-down menu displays suggestions, but takes up a lot of real estate which could easily be used to display results directly in the drop-down. Our Solutions Engineer Olivier Lance pointed out that this may be because many ecommerce sites imitate what works on mobile for their desktop version, opting for experience parity instead of optimizing for each interface.\nAmazon\u2019s search results page performs well across a variety of queries, including the typo-ed \u2018iphone case blaue with red stirpe,\u2019 which turns up a nifty New York Rangers iPhone case that meets all my typo-filled criteria.\n\nMy Vague query pushed a \u2018Recommended\u2019 result as well as a \u2018Best-Seller\u2019 result, which is a great way to drive the user to discover products. For refining my search, I am disappointed that the facets lack counts, and I can't search for facet values (like refining my search to find Bose headphones) inside the long list of brands, and are all-around a bit too cluttered for me to really make use.\nAs an accepted authority on shopping, Amazon makes its recommendations known, appearing regularly in the first few results. It could be easier to get access to the long-tail of Amazon\u2019s catalog - I had to open up \u2018Show all departments\u2019 and select electronics to see other cases, even though case was in","record_index":3},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 my query. By not displaying results instantly with each keystroke, it also takes longer to check Bose headphones and then change your query to Audiotechnica headphones.\nTarget\nTarget scored an 83% on the Search Grader (here) - like most sites we\u2019ve looked at today, Target doesn\u2019t display results in the drop-down menu, so users have to click-to-search. Query suggestions aren\u2019t typo-tolerant on the drop-down menu, meaning my Vague Query only gets suggested popular brand queries if I don\u2019t make an error. The left-hand \u2018categories\u2019 section is also a bit of a rabbit hole - sub-sub-categories is a lot of diving before seeing results.\nWhile they may not be typo-tolerant, I did like that query suggestions appear after you click the search bar before even typing - this is a great way to encourage discovery in a minimalist way that doesn\u2019t feel promotional. After typing, the lack of instant results and typo-tolerant query suggestions means that Target is hoping you\u2019ll click-to-search and that their results page will seduce you.\nTarget\u2019s search results page stands out for a number of reasons. They have a lot of the features we love to see in a search results page - search for value values in the brand facet, facet counts for almost every facet - they even have a few features that eCommerce stores could get inspired by. For a retailer, I think Target\u2019s execution on the click-and-collect feature is among the most natural I\u2019ve seen - the simple checkbox \u2018get it today\u2019 above the results is very inviting, and the UX is very slick. I also very much enjoy the hover-triggered second image that Target displays on results - it\u2019s definitely eye-catching, and if the load speed doesn\u2019t suffer, it\u2019s a great trick for image-centric search results pages.\n\nWhen digging in with a long, typo-ridden query like \u2018iPhane blaue case\u2019 (I actually typed that by accident - don\u2019t judge), I found that, while Target understood \u2018iPhone,\u2019\u2019 it didn\u2019t understand","record_index":4},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 \u2018blue,\u2019 and I really started to feel the limits of Targets relevance here. I tried a Vague query \u2018Phone\u2019 and was a bit surprised to find two flip phones & \u00a0a landline among the top results - this may be the most relevant results for their shoppers, but I would\u2019ve expected newer products, as someone looking for a flip phone will likely drill down by price or category to get there.\nWalmart\nPicking up a 90% on the Search Grader (here), Walmart has all of the same characteristics & features I've come to expect in these five Cyber Monday deals sites. It's typo-tolerant, it's injecting business metrics, and results load pretty quickly. There's not much feature-wise that can be said about Walmart that hasn't already been said about our predecessors, so let's jump in to the experience.\n\nWalmart's relevance really blew me away. It crushed it on my Long Query, and when I refined by device (not in image), I got various options on striped & colorful phone cases. I would've preferred to see facet counts and the facets are a bit too cluttered for me, but Walmart really came through here.\nMy vague query turned up headphones for under $10 - a truly Walmart experience - and when I refined my query to 'Bose Headphones,' it picked up on the brand and gave me a full-banner ad letting me know that the high-end brand was indeed available at Walmart.\nThe experience itself was a bit cluttered - not just the facets, but the drop-down menu and the second menu that runs underneath the search bar - it took me far too long to find out how to to click-and-collect, for example.\nBest Practices & Opportunities\nAcross these five sites, it was clear that there are some common practices and some places for improvement.\nIn terms of placement, it is pretty common practice to make the search bar as visible as possible - you can\u2019t go wrong with a white search bar on a dark header. Dropdown query suggestions are the mode du jour, but I think Best Buy takes the cake for","record_index":5},{"post_id":7089,"post_title":"Cyber Monday Search Review: Walmart, Target, Best Buy, Amazon & eBay","post_date":1511766149,"post_date_formatted":"Nov 27th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/cyber-monday-search-review-walmart-target-best-buy-amazon-ebay\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/search-review-illustration-1-360x200.png","time_to_read":11,"content":"\u2026 leveraging the width of the desktop drop-down bar to display instant search results. It\u2019s one less load time, one less second of attention before shoppers go somewhere else.\nIf content is king, then relevance is queen here. The sites that stood out to me were the ones that dug through the color interpretation, through the typos, and through the vagueness to provide me not only with relevance results, but with context-enriched suggestions for categories, brands & products that I might be interested in.\nBalancing discovery & finding isn't just a relevance problem. Shoppers are increasingly willing to refine an initial search if they don't find what they were looking for - they're having a conversation with your search bar - but many of the biggest e-commerce sites could make that conversation more fluid by providing more value to their facets and by providing instant search results for queries that are refined within the search results page.","record_index":6},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"Here I\u2019ll share a few things I\u2019ve learned managing libraries at Algolia. They are by no means a perfect set of instructions or recommendations, but tips for making your (developer) life easier. \nHere are some good signs that you may want to re-work your release workflow:\n\nYou tend to batch features before releasing them\nYou wait as long as possible before implementing something although you know well it will eventually need to be done\nYou are stressed when releasing\nYou\u2019d rather leave bugs in your code than have to release\nYou only release when forced to, and in general it puts you in a bad mood \n\nDon\u2019t rely on a single person to release\nWhen I was just about six months into my job at Algolia, a bug inside one of our JavaScript libraries impacted customer search implementations in production. We realized that one of the latest changes committed to the repository was having unexpected side effects, which led to always displaying empty search results. Unfortunately, the one person who knew how to release the library had just left the office and was commuting home.\u00a0Thankfully, they still managed to point us at a document detailing the release process of the library. We were relieved. \nThe team\u2019s action plan was to:\n\nRevert the latest code changes introduced during the day\nQuickly release a patched version\nCalmly investigate the issue we had pushed while clients enjoy a working search\n\nThe first step was easy to do given we had used Git as the source control system for the project. We just had to revert some of the latest commits. For the second point though, the procedure included many steps.\nEven though having a lot of steps is not a problem in its own right, what was a problem is that a lot of questions came up:\u00a0\n\nWill I be able to authenticate to the npm registry?\nAm I even part of the organization owning the library on npm?\nHow do I propagate the change to the CDNs so that clients can have their bug fixed as fast as possible?\nI thought that the CDN","record_index":0},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 update was now automated, why do we need to do that part? \n\nEven if I was asking myself only one of the above questions, I would have started doubting myself and feeling uncomfortable about the entire process. Asking myself all four was...too much. \nI learned two things that day:\n\n1. Any developer in the company should be able to release any given project. Put differently: relying on a single person to deploy\/release a project can be dangerous.\n\n2. To be able to have anyone deploy a project, they must be comfortable doing so. Comfortable means that no questions are left unanswered in the release process, and that the release process itself is simple enough to be actionable.\n\nTo make sure a fair amount of engineers in your company can deploy a project, the best solution I can think of is a single bash script which would guide them through the publishing steps.\nSome companies like Etsy take this quite seriously and have their new employees release something to production from day one in the company\nNext time you publish a project, here are a couple of questions you could ask yourself to evaluate the quality of your release workflow:\n\nCould this be released by someone else than myself?\nIf yes, will they have to reach out to me to fix the build?\n\nAutomation is key\nI recommend automating the entire release process as much as possible, which includes taking care of those tiny little things you think are not worth automating.\nFor example, you should not have to replace the version number in any file manually, the reason being that one day you will forget about replacing one version number, or one day it won\u2019t be you releasing. That day, you will probably lose a lot of time and possibly negatively impact production environments. Furthermore, chances are that the time you will lose that day is more than the time it takes to automate your release process today. \nA good practice is to aim for a single command line you can execute to get your software released. Ideally the","record_index":1},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 command should be interactive and guide you in the process by asking some questions like::\n\nDo you want to release a beta or production version?\nHere are the current changes, which one would you like to push as a new version?\nHere's the previous changelog and version, given we use SemVer and the selected changes, what should be the next version?\nDo you also want to release the documentation website?\n\nHere's an example including some of those questions:\n\nMake sure your can release fast\nIn almost every project I work on, there is some kind of a continuous integration setup. \nOn every new commit pushed to the repository under source control, all tests are run in a single environment or in multiple ones. This has the advantage of making sure future releases are working correctly on the targeted platforms, but has the drawback of slowing down the time needed for the release to get out.\nBecause implementing pragmatic releasing is an iterative process, some projects I\u2019m working on are still taking up to 45 minutes to have all tests pass. This mainly happens when the project has many integrations and end-to-end tests including relying on calls to an external API, Algolia in my case.\nHaving long running tests like these can be a real bottleneck for productivity. I would personally tend to avoid having to add features to those projects, because I know it is going to be time consuming.\nJust to give you an idea, here is what the process of adding a single feature to such a repository looks like:\n\nPush the changes\nDo something else during more or less 45 minutes\nGo back to the project, and eventually realize that a test failed on a given platform\nPush a fix to support the platform and wait again for 45\nEventually remember that you had pushed a given feature to the repository and ask for reviewers to approve the code\nRedo steps 2 & 3 if any feedback has been given by any reviewer\nMerge the changes\nRelease the project with newly integrated changes\n\nIn a best-case scenario,","record_index":2},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 it takes about an hour to release even the simplest possible feature.\u00a0If you are unlucky, though, you could spend a day working on getting a feature out. Now imagine this feature is an actual bug fix impacting production environments. It would be totally unacceptable to have to wait one full day to get the patch out.\nIdeally, the time to release a new version should be equivalent to the time to implement the feature and get it reviewed. A few ways to help accomplish this: \n\nReducing as much as possible any kind of a long running process directly impacting the speed of releasing new features or bug fixes.\nIf there are end-to-end tests relying on external APIs in place, caching API responses, and making tests run on mocked calls.\nIf you often have to reject PRs because the format of the commit is incorrect, have your CI platform validate the format for you. If your CI platform is slow, it is worth investigating that issue as well.\n\nWhatever the issue might be, taking some time to speed up to release cycle is key to fast iterations.\nBe confident in your code\nIn the previous paragraph, I shared how much I think speed of releasing is important for a project. One thing that is equally important is the quality of the builds you are shipping.\nThe moment you get something released, you have to be confident it achieves what it was designed for. In other words, you should test your code to ensure business expectations are met. There should be no way to release a project that has failing tests. \nHowever, I think that it is also perfectly fine to ship a \u201cwork in progress\u201d feature as long as it doesn\u2019t impact other features. If you have a chance to break down a big feature into many smaller ones, you\u2019ll be able to iterate faster because the review will be easier.\nA couple of ideas to challenge your existing release workflow\nHave someone else do your next release\nNext time you are about to release your library, ask a colleague who knows nothing about your project to","record_index":3},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 deploy it. Give them the URL of your repository as the only instruction. If they succeed in releasing, it probably means the quality of your release cycle is not bad.\nIterate faster, deploy more\nNext time you have an incoming task that is easy to address, get it done right away and force yourself to release the change. If you feel like you would have preferred to open an issue and deal with it later instead, it probably means you can still optimize your release workflow.\nThe benefits of releasing often\nImprove your mental well-being\nWhen you reach the stage where you can release on demand, you address issues differently. Instead of polluting the repository with issues that would distress and distract you, you get things done instead. Plus, given other teammates are able to release the project without your help, you can live without the fear of having to remotely guide a stressed out coworker on your day off.\u00a0\nImprove your productivity\nWhen you have a robust and simple release script, you can release when you see fit. The mental gap between release intention and the actual release should be negligible. \nBe more reactive\nIn the \u201cDon\u2019t rely on a single person to release\u201d section, I shared a real story about a project that impacted real production environments. Between the initial report and the actual fixed release we lost about an hour of time. \u00a0By implementing a release process that follows the principles shared here, the time to release the bug fix is now equal to the time necessary to implement the bug fix. In most cases, this would be a couple of minutes if you are using a source control system like Git: just the time to revert the changes and release again.\nTooling to create better release workflows\nHere\u2019s how I like to design the release workflow as of today:\nUse conventional commits\nConventional commits dictates a format which every single commit of your repository should follow. By doing so, you\u2019ll be able to:\n\nAutomate the generation of your","record_index":4},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 CHANGELOG file at every new release. Conventional Changelog is a good tool for this.\nMake it easy to see what has been done since last release by dumping out unreleased work.\nMake sure your CHANGELOG never misses a single entry and that the formatting stays consistent.\n\nI would recommend you use conventional commits to avoid the burden of manually having to update the CHANGELOG file.\nI would also suggest you have your CI platform test all commits to ensure a badly formatted commit cannot be committed to your master branch. You could use something like commitlint to ensure the format is correct.\nCreate a release-pr script\nCreate a script (in bash for example) that will do the following:\n\n\nCheck that you are currently on master\nCheck that the working tree is clean\nInstall the dependencies \nRun the tests\nAsk for the new version of the project after having dumped the unreleased changes\nUpdate all the version numbers in all the files where it appears\nPush a new branch `chore(release): ${VERSION}`\nWait for a teammate\u2019s approval\nMerge the branch\n\nHere is an example of such a bash releasing script:\nView the code on Gist.\nCreate a publishing script (manual)\nCreate a script (in bash for example) that will do the following:\n\nCheck that you are currently on master\nCheck that the working tree is clean\nInstall the dependencies \nRun the tests\nPush the new version to a public listing if required (e.g., npm for the JavaScript example below)\nTag the current commit with Git with the current version, and push the tag to the remote repository\n\nHere is an example of such a bash publishing script:\nView the code on Gist.\nCreate a publishing script (Continuous Delivery)\nPrevious step assumes you manually check out the changes after you have merged the release branch, and then run the publishing script. You could also let your CI platform handle this publishing for you every time something gets merged into your master branch. Personally, I like to keep this publishing task manual, because","record_index":5},{"post_id":7062,"post_title":"Pragmatic Releasing: Less Worry, More Shipping","post_date":1510906425,"post_date_formatted":"Nov 17th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pragmatic-releasing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/11-2017_Release-Workflow-360x200.png","time_to_read":13,"content":"\u2026 if something goes wrong, you are able to fix it easily.\nFinal word\nReleasing software should be something you enjoy doing: each time you release, you either fix a bug or introduce awesome new features. Spending some time to optimize your release workflow helps delivering better quality faster.\nI would love to hear what you do to make your release flow less stressful and more fun: @rayrutjes.","record_index":6},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"According to its 2017 State of the Octoverse report, Github now hosts more than 25 million active public repositories. Many of these projects are tools made by developers for developers, and being able to search through the masses and find exactly the right tool for the right problem has become a valuable skill of its own.\nCIOs, engineering directors and project managers don't only rely on developers for tool choice because of their technical expertise. They also understand that developer happiness equates to developer productivity\u2014which with engineering salaries and talent shortages at a historic high\u2014translates directly into higher quality products and lower total-cost-of-ownership for internal applications.\nDeveloper experience, it turns out, is good for business.\nTools with great developer experience are marked by intuitive design, up-to-date documentation, clear error messages, rock-solid stability and access to friendly support. Underpinning all of that, the tool must solve the problem the developer needs it to.\nDevelopers love fast test suites and code that\u2019s easy to follow\nDeveloper Experience in search\nBoth Algolia and Elasticsearch have risen in popularity because they provide a better developer experience than the search platforms of the past.\nThough a critical feature of many applications, search remains a challenging problem for developers to solve. Moreover, building search as fast and relevant as Google and Amazon\u2014companies that set the consumer\u2019s expectation of what search should be\u2014is a great deal harder than building the press-enter-and-wait search of the bygone Web 1.0 era.\nTo compete, engineering teams need to go faster than ever before, identifying and adopting tools that shorten the developer journey and enhance the developer experience at each stop along the way.\nIf developer journey is the \u201cwhere\u201d, developer experience is the \u201chow smoothly\u201d\n \nIn Part 1 of the Comparing Algolia and Elasticsearch for Consumer-Grade","record_index":0},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 Search series\u2014End-to-end Latency\u2014we saw that search performance requirements are more demanding than for other types of applications. In Part 2\u2014Relevance Isn\u2019t Luck\u2014we dove deep and looked at how search engines accomplish the task of serving results that accurately match user intent, even when intent is scarce, flawed or unexpected.\nHere in Part 3, we\u2019ll look at the high-level journey that developers take to build search, using an API like Algolia or an open source tool like Elasticsearch. We\u2019ll also focus in one of the parts of the journey that\u2019s important to consider early on\u2014the crafting of the user experience.\nFive aspects of a \u201cSUPIR\u201d search\nNo matter what search technology a developer is using, there are five aspects that the developer must consider on the way from design to production. These are:\n\nScalability\nUser Experience\nPerformance\nIndexing\nRelevance\n\nRemember them easily as a typo-tolerant acronym - SUPIR.\nOrdering the steps this way makes a convenient acronym, but it is generally not the order that the developer proceeds in. Nor is the order linear. For example, indexing data is required before testing out any relevance or user experience, and is therefore a key step during an early prototyping phase. But indexing will also be revisited later, when additional signals become available for relevance or when scaling puts stricter demands on record size.\nWhat makes search DX different\nJust as the world that your users live in changes frequently, so must their search. Like performance tuning, relevance tuning is never \u201cdone\u201d, it can only be \u201cdone for now\u201d.\nThis is the first differentiator between building search and building other types of applications. Developers don\u2019t visit each SUPIR aspect just once but on a progressive and recurring basis, roughing out and sharpening each one during design and build, and later maintaining and improving them as an ensemble in production.\nThe second differentiator between search and other","record_index":1},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 relational or NoSQL database-backed applications is the tighter level of coupling required between each layer of the stack. In traditional CRUD-based apps, the implementation details of the database\u2019s storage engine have little bearing on the types of user experiences that can be built on top of it. This is not true in search\u2014the feasibility of a particular user experience can come down to specific implementation details like whether the search engine uses Tries or not.\nThe iterative nature and tight level of coupling in search can be the source of numerous headaches for developers. Improving one SUPIR aspect can break several others, a frustrating event that feels more like retracing steps than moving forward. Some examples:\n\nA relevance improvement for one query makes all queries slower\nA relevance improvement for one query worsens relevance for others\nAt scale, heavy indexing loads begin to degrade search performance\nDegraded search performance requires implementing UX workarounds\n\nAs we head to the next sections and start to address the individual aspects of SUPIR, we\u2019ll take care to point out what developers using Algolia or Elasticsearch need to do to avoid these pitfalls. First up, we\u2019ll look at the \u201cU\u201d in SUPIR\u2014user experience.\nCryptic error messages cause developers distress\n \nUser Experience\nIn search, it\u2019s important to begin with the end in mind. What should happen when the user starts typing in that shiny new search box? The user interface (UI) and user experience (UX) will heavily affect the design of the other four SUPIR aspects, so it\u2019s important to sketch it out early.\nDesign and best practices\nThere are many different types of search UX, ranging from a simple keyword-based autosuggest up to a full-page search-as-you-type experience with several ways to facet and filter. Today\u2019s search experiences also often include powerful browsing and navigation capabilities.\nTo be on par with what we expect from Google or Amazon, the","record_index":2},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 search should feel like it\u2019s a step ahead of us and practically guessing what we want. It should also provide instant feedback so we can correct our mistakes or refine our search, and it should work wherever we happen to be computing from. A satisfying search UX will:\n\nUpdate results immediately as the user types, putting them in control of the experience\nHandle spelling mistakes, omissions and concatenations\nMake an obvious connection between search terms and results\nWork on all web and mobile platforms\n\nIn many cases, these additional search UX patterns will be used to enhance the experience:\n\nHierarchical faceting and filtering\nBrowsing with or without keywords, including pagination\nSearching multiple different data types simultaneously\n\n \nA powerful product search with instant results\n \nOnce the initial requirements and design for the UI\/UX have been decided on, the next step for the developer is to start building it. Ideally, they won\u2019t have to start from scratch.\nFrontend API clients\nBoth Algolia and Elasticsearch have a REST API that makes it possible to interact with them over HTTP\u2014therefore directly via web browsers and mobile devices. API clients contain the logic to send requests over HTTP, parse responses, and handle network and security concerns. They\u2019re an essential building block for user interfaces and user interface libraries built on top of them. They\u2019re also one of the first places where the developer looks for guidance before they start coding.\nAlgolia and Elasticsearch both have a JavaScript API client that works in the browser and node.js. Algolia also has official native API clients for iOS and Android.\n\nAPI clients, security and performance\nAPI clients must also implement security, including authentication and authorization logic. In search, this goes beyond just passing auth credentials. Just because a user can search an index doesn\u2019t necessarily mean they\u2019re allowed to retrieve all records, or see all of the fields","record_index":3},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 of each record.\nWhen using Elasticsearch, this level of fine-grained security is often implemented by an application server, rather than exposing Elasticsearch to frontend clients directly. You can think of the parallel to your primary application database like Postgresql or MongoDB\u2014these databases are accessed directly by your servers, not your users\u2019 clients. The advantage of this is flexibility\u2014you can implement any authorization scheme you like\u2014but the disadvantage comes in having to build and maintain it, thereby adding an additional step to the developer journey. A second disadvantage comes in terms of performance, as every search query will need to be processed by an application server that sits in front of your Elasticsearch instance, and it is likely that the application server will perform one or more calls to your relational database in order to authorize the HTTP request. Many precious milliseconds are consumed therein.\nWhen Elasticsearch is accessed this way, the developer does not use a frontend Elasticsearch API client at all, but connects to an existing backend which proxies the request to an Elasticsearch instance.\nAlgolia is designed to be called directly from the client, with authentication and authorization being handled by passing API keys. Algolia secured API keys can encode a lot more information than just an authorizing token, including index and search parameter restrictions, ipv4 source restrictions, rate limits and expiry information. This gives developers flexibility when it comes to security, without slowing them down.\nThe direct connection between frontend API client and Algolia host is also fast. The entire search request is fulfilled by an Algolia module running inside of NGINX\u2014no expensive application servers or database calls required. Lastly, the connection is reliable. API clients are aware of all 3 Algolia hosts in a cluster, as well as any global DSN replicas, and can work around any downed hosts. Additionally, Algolia","record_index":4},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 API clients can fallback to a second DNS provider in order to work around DDoS attacks, like the 30-50+ Gbps attack on NS1 in May 2016.\nSo while API clients may seem simple at first glance, the details can have a big impact on both performance and reliability \u2014 and therefore the quality of the user experience. If developers don\u2019t have to reinvent the wheel just to pass data back and forth between clients and servers, their experience (and productivity) will be vastly improved.\nReusable UI building blocks\nThough the API client is fundamental, it\u2019s still only a small piece of the overall frontend story. User-initiated keystrokes and clicks must be handled, results must be displayed and made selectable. That could be via an autocomplete menu or a more advanced page with controls for faceting, filtering and pagination. In either case, this can add up to a lot of custom HTML, JavaScript and CSS.\nIn the Elasticsearch ecosystem, community developers have created projects like SearchKit and ElasticUI to help create these interfaces.\nIn the Algolia ecosystem, an officially-supported family of libraries called InstantSearch was created to encapsulate the various consumer-grade UI\/UX search best practices and make them easy to implement. Each InstantSearch library is composed of a set of UI widgets that can be added one by one to a search page. Here are examples of common widgets:\n\nSearch box\nHits (for displaying results)\nSort by selector\nRange slider\nRefinement list (for faceting)\n\nThe framework-agnostic JavaScript InstantSearch library, simply called InstantSearch.js, contains a total of 18 ready-made widgets. It also exposes factories for creating others. Each widget is connected to any change in the state of the search and re-renders automatically. Other members of the InstantSearch family work similarly. Here\u2019s a quick look at each one.\n\nFull-featured API clients and component libraries provide a good DX for building the UX\u2014 the \u201cU\u201d in our SUPIR search. They","record_index":5},{"post_id":7020,"post_title":"Comparing Algolia and Elasticsearch for Consumer-Grade Search Part 3: Developer Experience and UX","post_date":1510738771,"post_date_formatted":"Nov 15th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vs-elasticsearch-developer-experience-ux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/Screen-Shot-2017-11-09-at-10.38.15-360x200.png","time_to_read":11,"content":"\u2026 also reduce maintenance and total cost of ownership by cutting down on custom code. Component libraries in particular mean that the engineer building the UI doesn\u2019t need to be a JavaScript expert; in most cases a basic working knowledge of JavaScript will do.\nConclusion\nIn a consumer-grade search implementation, the user experience is critical. Consumers are picky and, if unsatisfied by a site or mobile app\u2019s search, are likely to abandon it in search of an alternative.\nBecause we rely on developers to build the amazing search user experience that we desire, we have to account for their experience as well. A tired, frustrated development team can\u2019t compete with a team that is happy and productive. A developer mired in the details of applying security patches or debugging deeply-nested JavaScript promises may not have the time\u2014or the energy\u2014to make their product\u2019s ambitious design spec a reality.\nWith an open source and more open-ended tool like Elasticsearch, the degree to which a developer gets mired or inspired is largely a function of their search-building experience and expertise. With Algolia, many of the most complex moving parts of the search-building journey\u2014including infrastructure, performance and scalability\u2014are handled automatically underneath the API, freeing up the developer to focus on building a great user experience and ultimately a great product.\nRead other parts of the Comparing Algolia and Elasticsearch for Consumer-Grade Search series:\nPart 1 - End-to-end Latency\nPart 2 - Relevance Isn't Luck\nPart 3 - Developer Experience and UX","record_index":6},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"Since December 2016, as part of our yearly community gift effort, Algolia has powered the JavaScript packages search of the Yarn website. This blog post explains how we collaborated with the Yarn team, what the challenges were building such a search interface, and how much this search is used today.\nYarn is a JavaScript package manager similar to npm. In the beginning, there was no way to search for JavaScript packages on the Yarn website. Since we were heavy users of Yarn, in December 2016 we built a proof-of-concept search UI, and it was released on their website one month later (Try it here!). As of today, every month there are about 700,000 searches (that's 2.3M API calls) being done on the Yarn JavaScript packages index on Algolia.\nnumber of user searches per month on the Yarn website\nFrom a Slack discussion to PR and Merge. All in one month.\nSearch on the Yarn website started with the documentation. We wanted people to easily find information on how to use Yarn.\u00a0As with 300 other programming community websites, we went for Algolia's DocSearch and this was merged in yarnpkg\/website#105. Then another Yarn contributor (@thejameskyle) asked in yarnpkg\/website#194 if there should be package searching abilities, much like npm had.\nThis is where Algolia came into play. We are a search engine, Yarn wants search and we are heavy users of Yarn, so we figured: let's do this!\nThis is how it started on December 5th in our #2016-community-gift Slack channel:\n\n\"Hey what could we build for the JavaScript community that would help them in their daily workflow?\"\n\"It's not that easy to find relevant JavaScript packages when you need one\"\n\"I like Yarn a lot...\"\n\"Let's build Yarn search!\"\n\nWe wanted the community to feel empowered by a great new way to search for JavaScript packages. This was also an opportunity for us to work on something cool while benefiting the company. Three weeks later, on December 22th 2016, via yarnpkg\/website#322, we proposed our first package search","record_index":0},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"\u2026 solution. Ten days later it got merged, and instant-search for JavaScript packages was available on the Yarn website.\nIn early 2017, we met with the Yarn team for a one day in-person brainstorming in London. The goal was to think about evolutions of the search experience along with defining a package details page. Algolia proposed design views of what search could be and from that we drafted a master plan.\nFeatures behind a great package search\nIt shows results instantly \u26a1\ufe0f\n^ This is not sped up. It is THAT fast (try it!). Yes, it still wows even us every time.\nInstead of showing a dropdown of results, we chose to replace the page completely with the search results. This requires more data to be available immediately, but gives more context on the decisions you make while searching for a fitting package. Having the search page be the main entry point will make sure that you don't need to know exactly what you want before \"committing\" to a search.\nIt displays a lot of metadata\nAfter using npm search many times, we knew what was missing and what was superfluous from the search results and package detail pages. We brainstormed a bit and iteratively added a lot of useful metadata.\nHere's a comparison between the two search results pages (npm on the left, Yarn on the right):\n\nnpm search results on the left, Yarn search results on the right (click to enlarge)\nIn the search results of Yarn we decided to directly display, for example, the number of downloads for every packages, the license, direct links to GitHub, and the owning organization.\nThis metadata helps the user to not have to open many package detail pages before getting the information they want.\nIt has completely rethought package detail pages\nFor the package detail page, we took a similar approach. We started with the same base metadata as npm shows, but also took the opportunity to add a lot more. We decided to show changelogs (when available), GitHub stargazers, commit activity, deprecation messages,","record_index":1},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"\u2026 dependencies and file browsing.\nHere's what it looks like:\n\nnpm detail page on the left, Yarn detail page on the right (click to enlarge)\nWe believe (and we had a lot of feedback about it) that all those additions are providing an enhanced experience that helps users when finding and comparing JavaScript packages.\n\nTIL yarn has a responsive package details pagehttps:\/\/t.co\/w2QkQoDP9P pic.twitter.com\/wBnQ9biD85\n— John-David Dalton (@jdalton) March 30, 2017\n\nThis is an iterative process, and suggestions and feedback are always welcome.\nTechnical implementation and challenges\nThe first step to providing a search for JavaScript packages is to replicate and monitor changes from the npm registry into an Algolia index.\nThe code for this replication is all open source and available at algolia\/npm-search. The most important API being used here is the npm replication API.\nThe npm registry is exposed as a CouchDB database, which has a replication protocol that can be used to either set up your own npm registry, or in our case a service (the Algolia index) that has the same data as the npm registry.\nReplicating the npm registry\nReplication in CouchDB is a very simple but powerful system that assigns an \"update sequence\" (a number) to any changes made on a database. Then, to replicate a database and stay in sync, you only need to go from the update sequence 0 (zero) to the last update sequence, while also saving the last update sequence you replicated on your end. For example, right now, the last update sequence known on the npm registry is 5647452 (more than five million changes).\nEarly on we saw that going from 0 to 5647452 was very slow (multiple hours) and we wanted it to be faster. So, we made a replication system consisting of three phases:\nThe bootstrap. Instead of going over all update sequences, we save the current last sequence, then we list all JavaScript packages and replicate them by bulk to Algolia\n\nThe catch-up. Starting from our last known update sequence,","record_index":2},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"\u2026 we catch up to the new last update sequence of npm (maybe 5000 changes since bootstrap start, which is fast)\nThe watch. Once we are \"up to date\" then we just watch the npm repository for any new changes and we replicate them\n\nFor all of those phases, we use the PouchDB module which we recommend because it has an awesome API to interact with CouchDB databases.\nGetting the necessary metadata\nAll the phases go through \u200bthe same steps to get the required metadata for displaying. Some of the metadata is also retrieved on the frontend directly, like the GitHub ones (stargazers, commit activity).\nHere are all our sources:\n\nnpm registry, example for express: http:\/\/registry.npmjs.org\/express\nnpm download counts: the npm downloads endpoint\nPackages' dependents: the dependents endpoint of npm (there's no official documentation on that)\nChangelogs: a clever first resolved, first served list of calls to various ChAnGeloG files, like History.md's express changelog\nGitHub Stargazers\u2b50\ufe0f, commit activity: we get them on the frontend directly from GitHub using the browser of the person doing a search. This way we benefit from a larger rate limit on GitHub\u00a0shared amongst all users. This is also what npm search does for opened issues on their detail pages.\nBrowse view: we get this from the unpkg API, which gives us the files, folders and sizes for all published packages\n\nQuery Rules to the rescue\nThere are a lot of Algolia features baked in our JavaScript packages search index; you can see the whole index settings in our repo.\nOne of them that really helped us is Query Rules. When you are searching for a package, there are two questions to answer: the package that you exactly typed, and the package that you probably typed. We found that other searches often don\u2019t have what you typed exactly early in the results, even though it exists.\nWhat we have as a solution is a query rule that applies when the user types the name of a package exactly or without special characters (to","record_index":3},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"\u2026 allow people affordance in how they type dashes).\nExample query rule to boost exact matches\nThis allows a query like `reactnative` to have as first result `react-native` which is very popular, and as second result `reactnative`, which is deprecated and not supposed to be used, but still exactly what the user typed\u00a0and may be looking for.\nFor a package search, we can't make any assumptions like \"Maybe the user was looking for this package instead of what they typed\". Instead we want to always present them both the exact package match if any and then the popular ones.\nThe future of Yarn search\nA big part of our success was made possible because we opened the JavaScript package search to multiple websites and applications (which is another milestone for us!), namely:\n\nYarn (65% of searches)\njsDelivr, the free Open Source CDN (10% of searches) serving one billion downloads per month of our libraries\nAtom autocomplete module import\n\nWe will soon open the JavaScript package search API to more websites and make it an official community project. The plan is to create a single page documentation for anyone to reuse this JavaScript search API in their applications. From editor plugins to community websites like CodeSandbox, we know the demand for an easy-to-use and instant package search is high.\nBuilding on that we want to add new features like:\n\nBundle size information like siddharthkp\/bundlesize\nAdvanced filtering with tags\nAnything YOU would like to see, let us know\n\nWe did not stop at the search feature. I am proud to be a frequent contributor to the Yarn website, helping on adding translations, reviewing or merging PRs and updating the Yarn cli documentation.\nThanks\nThis project wouldn't have been feasible without the help of everyone from the Yarn and Algolia teams. Since our first contact with the Yarn team, communication has always been great and allowed everyone to feel confident about shipping new features to the Yarn website.\nWe also want to thank very much the","record_index":4},{"post_id":6982,"post_title":"Yarn: From Zero to 700,000 User Searches per Month","post_date":1510219443,"post_date_formatted":"Nov 9th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/yarn-search-javascript-packages\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-11_Yarn-Algolia-360x200.png","time_to_read":11,"content":"\u2026 npm team for being responsive and advising us while we were building our replication mechanism. \nWe hope you enjoyed this article, see you soon for this year\u2019s community gift \ud83d\ude80\nThanks to Vincent and Ivana for their help while writing this article.","record_index":5},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"What would a search interface without a search box look like? After a few weeks of exploration, I created an app that gives music advice through a conversation, and called it \u201cMusicologist\u201d. This is the story of how I built it, hoping to give you an idea of what kinds of new search interfaces are coming to the fore and how you could start building them yourself.\nIntroduction: a dying search box\nIf you were a search box, you would worry about the recent press. From The Verge reporting that it will disappear in the next decade, to eBay saying it will at best \u201cbecome redundant,\u201d it\u2019s a rough time for our good old search box. But is the end of the search box the end of search interfaces?\nIf you go beyond the headline, it\u2019s a somewhat different story. The Verge\u2019s article is referring to research by Susan Dumais, who said that \u201c[search box] will be replaced by search functionality that is more ubiquitous, embedded and contextually sensitive\u201d.\nSo the search box might disappear, but only to be replaced by a new generation of search interfaces: accessible everywhere, integrated in your activities, and aware of the context in which you are.\nConversational search interfaces are a first step in this direction. They are ubiquitous, as you can deploy them on any platform where your users reside. You can design them to be integrated in your user\u2019s journey, providing value while interrupting them as little as possible. Finally, they let you leverage your user\u2019s context by remembering previous conversations to adapt to them.\nBut what would a search interface without a search box look like? I spent a few weeks to explore what has been done in this field, and to propose an example of what kind of UX a new type of interface could provide: the \u201cMusicologist\u201d. This application gives you music advice through a conversation, and its code is now released open-source on Github. Here is a demo to show you the Musicologist in action:\n\nIn this demonstration, you see a","record_index":0},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 search interface built on three components: a mobile frontend, a conversational agent, and a backend powered by Algolia. Let\u2019s see what each component does and how you can build it.\nThe agent: understanding your users\nWe built the conversational agent with Dialogflow (formerly api.ai), a conversational API that helps you build conversational experiences in natural language.\nAt its core, the agent is a middleware capable of turning an input sentence into an intent that can contain several parameters. It can then forward those to a web service, and turn its answer into a reply sentence. Intents describe what the user can say: each one is a different intention that the user can express in many ways.\nFor the Musicologist there are two main intents: search for songs and choose a result. Each one is defined by the sentences that a user could use to express it; for example, the search for songs intent:\n\u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 Two expressions that would show an intent to search for songs\nAs you can see in the screenshot, some words are highlighted. These are parameters of the intent, corresponding to different entities. Each entity describes a category of objects you could recognize. You can use the many system entities provided (phone numbers, city names, months of the year, etc), or you can define your own if none of the former fits your use case.\nTypo-tolerance and other limitations\nThere are, however, some limitations in the way entities are recognized. For example, there is almost no typo-tolerance in the entity recognition system. If you use the system in its default configuration, you can only recognize exact entities. Let\u2019s define a project entity called \u201cDocSearch\u201d, and use it in an Intent. The agent recognizes it:\n\nBut if you do a typo, the agent is confused and does not recognize the entity anymore:\n\nThe agent didn\u2019t recognize the DocSearch project\nThere is an option called Allow automated expansion, but unless you give a lot of","record_index":1},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 examples to your agent, it has good chances of inventing non-existent entities:\n\nWith automated expansion, the agent believes \u201cDocsarch\u201d is a valid project\nThis demonstrates the limitation of an entity recognition system that you cannot customize further. Like with handling typos, there are several techniques you would want to use when detecting entities, like removing superfluous terms or normalizing them.\nThankfully, you don\u2019t have to implement these on your own. You can augment your agent by leveraging an API to give it more powers:\n\nHandling typos: you might think that with conversational interfaces, users won\u2019t do typing mistakes as they are speaking to their devices. But being ubiquitous, your conversational interface can be deployed on various platforms, through which your users can talk (Amazon Alexa, Google Assistant, \u2026) as well as write (Slack, Messenger, \u2026). As your users will expect the same quality of results in both cases, customizable typo-tolerance is key to a great user experience across channels.\nIgnoring variants: be it via voice or text, your users might not speak exactly how you expect it. From plurals to declinations or missing diacritics, it is important to understand your user across all variants.\nRemoving optional words: especially via voice, your users might tell you more than what you expect, and phrase a query that\u2019s too precise. But when you don\u2019t find anything relevant to it, you will provide a better experience to your users by giving them results for a part of the query: being able to configure optional words lets you fine tune this experience.\nDefining advanced synonyms: you can define two-way synonyms in your agent, but this only helps if the two terms are strictly equivalent. This is not always the case: for example, you might want to show songs of the genre Rockabilly when queried about Rock, since the former is a subgenre of the latter. In this case, you don\u2019t want a query with \u201cRockabilly Song\u201d to return","record_index":2},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 any Rock song: you can solve this with one-way synonyms.\n\nThis is only a glimpse of what you can do: there are several\u00a0relevance features you can use to improve how your conversational interface understands its users.\nThe backend: coming up with meaningful answers\nThe agent now gets what the user means (let\u2019s say searching for songs) and the eventual entities in the intent (for example Led Zeppelin). But how can it provide a relevant answer to its users?\nFor the Musicologist, the backend is the agent\u2019s intelligence engine: it will use Algolia to connect the intent with the relevant content taken from an index of songs. This gives some memory to the agent, giving it access to a great amount of data \u2014\u00a0as well as relevance, as the agent can now recognize an artist even if it is mistyped:\nThe agent now recognizes the correct entity, and can provide relevant results.\nThe backend\u2019s code, built with Node.js, is actually quite simple: it takes the request from Dialogflow, extracts the search parameters and queries Algolia with these, then transforms the search results into the format expected by the agent.\nOnce your backend is deployed, you can start talking to your search interface! You can deploy it on various platforms in a few clicks:\n\nDeploying your agent to Slack, Messenger, or Alexa is only a few clicks away\nFor example, here\u2019s what your agent could look like in Slack:\n\nOf course, you can customize each integration further. For example, you can use Messenger\u2019s Quick Replies pattern to propose predefined answers to your agent\u2019s responses. Still, this is quite powerful: your agent can be deployed on pretty much any platform without more work.\nYou now have an conversational search interface that can be accessed anywhere, and can already satisfy your users through voice-first interfaces or text-based conversational channels like Slack. This approach is optimal for low-friction interactions where you want to help your users find what they are looking for","record_index":3},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 without interrupting their current flow.\nAt other times however, your users will use your search interface to discover the content you can provide. It\u2019s thus important to adapt your search interface accordingly, by helping them navigate and understand your results. In this case, a purely voice interface is not optimal: being able to ask your questions in natural language is great, but getting many search results as a spoken sentence make them hard to understand. If this is the only way to delve into your data, your users will likely miss the rich search results UI they have gotten used to with consumer-grade search on Google, Amazon, etc. It also makes it harder to interact further with the results: imagine filtering results on several criteria by talking! It\u2019s a lot more difficult to think about such a sentence than to tap on a few filters.\nIt seems that for exploring your content, conversational interfaces and graphical search interfaces both have some advantages over the other. But why couldn\u2019t your users have both?\nThe frontend to navigate your results\nTo provide our music aficionados with an interface more suited for discovery, we built an Android frontend to the Musicologist. This is the actual search interface where users will interact with the agent.\nThere are several advantages to having a mobile frontend for our Musicologist. First of all, it helps us leverage existing speech-to-text and text-to-speech systems. The application will use any speech recognition service installed on the device to turn the user\u2019s voice into a written sentence, forward it to the agent, and then voice the response using the installed text to speech engine.\nBut having a mobile frontend brings more than voice-related components. It lets you adapt the user experience for content discovery by providing new ways to grasp the search results and interact with them.\nThe first area where this is helpful is understanding your search results. A voice interface puts a much higher","record_index":4},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 cognitive load on the user than a graphical interface, because they will need to pay attention to the voice interface as long as it speaks to avoid losing information. When you display the results, the user will read the information at the time they sees fit, rather than being forced to listen.\nMoreover, you can apply all the usual techniques that help the user get the most out of your results. You can display minor attributes that could be useful, highlight the results, snippet the relevant text, etc.\nThis can sound like a lot of work on top of the agent and backend, but it is very straightforward to build with the InstantSearch library - Algolia\u2019s open source search patterns library providing widgets and helpers for building an instant-search experience on Android. It is built on top of Algolia\u2019s Android API Client to provide you a high-level solution that abstracts the hard work of building such an interface.\n\nSearch results, visually highlighted and snippeted when relevant\nAs you can see, with a graphical interface it will be easier for your users to get the information they want. But that\u2019s not all: you will also make it easier to interact with this information. A lot of the patterns your users learned to expect on great search interfaces still make sense in a voice-enabled application.\nTake infinite scrolling as an example: sure, you can ask the agent to show more songs, but now that you have a list of results, it feels quite natural to simply scroll to see more results.\nLikewise, you can tell the agent to play a given song, but sometimes a tap on the search result will be more adequate to play that song in your favorite music app. With hands full, it will be natural to voice a query, then ask the agent to select a result, while in another context you\u2019ll find it quicker to tap on a search result.\nThe power of hybrid search interfaces\nSuch voice and graphical interfaces bring the best of both worlds: accessibility and ease of use via voice input\/output,","record_index":5},{"post_id":6956,"post_title":"Building an App That Gives Music Advice Through a Conversation","post_date":1510132608,"post_date_formatted":"Nov 8th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/conversational-ui-music-advice-app\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/11\/2017-10_Search-as-a-Conversation-360x200.png","time_to_read":11,"content":"\u2026 but also rich information display and interaction possibilities.\nOn the one hand, you have the possibility to use the search interface hands-free, and you get the results you are looking for without having to touch your phone. You can even get your search results without looking at it!\nOn the other hand, you can interact with the results without using your voice again, for example if you already started listening to some music and just want to hear the next song. All the familiar interactions with search results can be leveraged in such a hybrid search interface.\nThe Musicologist is a proof of concept of such a hybrid interface, but many search interfaces could benefit from this approach. There are many use cases for which a hybrid interface would bring value to your users: cooking, driving, troubleshooting your car\u2019s engine\u2026 whenever they already use their hands but would enjoy a screen. Those are just a few examples of situations where users would benefit both from hands-free control and a rich graphical user interface.\nWith the Musicologist, I wanted to give you an idea of what kinds of new search interfaces you could build. We hope this article will help you get started with building great voice search interfaces. If you build something with Algolia, please do let us know!","record_index":6},{"post_id":6926,"post_title":"Good API Documentation Is Not About Choosing the Right Tool","post_date":1509444873,"post_date_formatted":"Oct 31st 2017","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-choosing-right-tool\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/Illustration-doc-API-1-360x200.png","time_to_read":8,"content":"As a member of Algolia\u2019s documentation team, I am often asked: \u201cWhat tool are you using to build your documentation?\u201d Of course, I answer the question, but I am often tempted to say that it\u2019s probably the least valuable piece of information I can provide.\nIn this post, I am going to give you some of that information: what things you should care about when building your docs, and how those things will make the choice of tool the least important thing.\nDocumentation is usually composed of two big parts: the content and the software rendering it. You might have guessed where I am going with this: a quality README.md stored on GitHub can be far more efficient than over-engineered documentation that is well displayed but has issues with content.\nThe perfect tool is not out there\nThere are plenty of ways to build API documentation: static website generators (Jekyll, Hugo, Middleman), web frameworks (Ruby on Rails, Laravel, Django), dedicated doc tools (Docusaurus, Read the Docs, Sphinx), SaaS products (HelpDocs, Corilla), and that\u2019s just the tip of the iceberg \u2014 there are so many more.\nDepending on the one you choose you\u2019ll have more or less flexibility, and more or less work to build and maintain. All tools will let you decide to a certain extent what you can do, and constrain you on the other end. I don\u2019t believe there is a tool that can fit 100% of the needs in the long term. Documentation is something that needs to evolve, and you may have to change your tools several times as you outgrow certain constraints and have new needs.\nTwo years ago, we moved away from an internal tool that was aggregating content from GitHub ReadMes, a database and an external software managing our FAQ. This change took us full two months, and this is not counting the months of preparation prior to making the change.\nBy far the most time consuming task was to undo the formatting that our original tools required us to make. We had no consistency \u2014 some were Markdown, some","record_index":0},{"post_id":6926,"post_title":"Good API Documentation Is Not About Choosing the Right Tool","post_date":1509444873,"post_date_formatted":"Oct 31st 2017","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-choosing-right-tool\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/Illustration-doc-API-1-360x200.png","time_to_read":8,"content":"\u2026 were Markdown with custom extra features, some were plain HTML \u2014 and so while moving away from our previous tools, we had to edit thousands of files manually in order to unify everything.\nPut the focus on making the tool fit your content (not the other way around) \nInstead of focusing on which tool to use, a better option is to focus on whether you are doing everything possible to be as little software-dependent as possible. If you can respond to: \u201cCan I switch easily to a new tool?\u201d with a \u201cYes!\u201d, then you are on the right track.\nBuild all components to be software-independent\nWhile developing custom components, it's good to keep in mind that they should be as little dependent on the software as possible.\nNow, to answer that question from the beginning of the article\u2026It\u2019s been two years since we\u2019ve been using Middleman for our main documentation. It\u2019s doing the job, but it has some downsides and we\u2019ve had to customize it quite a bit to our needs.\nHere are some of the things that we added\/modified:\n\nCustom sitemap \nCustom data files system\nCustom snippet generation\nInjections of custom metrics from Google Analytics for ordering purposes (for example the FAQ entries listed in https:\/\/www.algolia.com\/doc\/faq\/ are the most viewed ones)\nAbility to have a snippet file that can be auto-injected into any content file\nAn Algolia indexer\n\nThese customizations are done in a way that we can reuse them in any project. The modifications represent ~800 lines of custom code, which is rather small for documentation like ours, but it enables us to be able to move data files to any other software in a matter of days rather than months; this is us adapting the tool to our content.\nKeep the content properly structured\nWhat is even more important is how you organize your content so that you can re-organize it programmatically when needed, or transform it so it fits another tool.\nThe more structured the information is, the easier it is to:\n- reuse it across different","record_index":1},{"post_id":6926,"post_title":"Good API Documentation Is Not About Choosing the Right Tool","post_date":1509444873,"post_date_formatted":"Oct 31st 2017","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-choosing-right-tool\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/Illustration-doc-API-1-360x200.png","time_to_read":8,"content":"\u2026 parts of the documentation\n- change the organization system itself when needed\nTwo tips on that front:\n\n1. Keep the content centralized\n\nAs mentioned earlier, our documentation comes from a system that was split in many different parts. Today, we have documentation in a single repo that you can run independently from the main website. This removes dependencies and allows us to focus on content and doc-specific modifications. It also give us the ability to iterate more quickly both on content and code parts.\n\n2. Choose the right file format\n\nAlso very important is where you write your content. When using a static website generator, it is \u201cthe norm\u201d to put all content inside the Markdown files. This can work for small docs, but when your documentation starts growing above hundreds of pages, with different types of content (guides, tutorials, reference, FAQ), and you are seeking consistency, using structured data files is a better approach.\nThat\u2019s why at Algolia we documented all methods and parameters in yml files and not Markdown files. While we ended up with Markdown inside yml, which can seem a bit counter-intuitive, it is quite useful. It also allows you to reuse the content in different ways across the website.\nThe two pages above are generated from the same yml file. When editing, this makes it very easy to keep consistency between different parts of the website.\n\nSo by focusing in this way on content first - its needs, structure, maintainability - and then finding and customizing the tool, you can come up with a documentation that is easy and quick to evolve.\nOnce this is in place, a good next step is to have your team and customers contributing content. Which leads me to ...\nBonus tip: get more contributions from the team and your customers\n\u2026.and make those contributions as frictionless as possible.\nThere are a few actions we took to achieve this. When logged in as an admin, next to every section or code snippet we have an edit button that links to the","record_index":2},{"post_id":6926,"post_title":"Good API Documentation Is Not About Choosing the Right Tool","post_date":1509444873,"post_date_formatted":"Oct 31st 2017","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-choosing-right-tool\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/Illustration-doc-API-1-360x200.png","time_to_read":8,"content":"\u2026 correct file in GitHub, enabling admins to modify the content he or she is currently viewing.\nLet\u2019s take an example of a new developer joining the team. One of the first things they are going to do is learn about the product by working with it. The easiest route for that is using the documentation. While they are using it, they will notice typos, unclear bits, undocumented features\u2026if they have to think about where to provide feedback, or how to edit the file, there is a high chance that they will do nothing and switch to another task.\nAnd if that\u2019s true for your team, it\u2019s even more true for customers. It is very unlikely that a user who noticed a typo will look around the website for the support email to tell you that they found something wrong.\nThis is one of the reasons we have the following big form at the bottom of every documentation page and also accessible from every section of the content:\n\nThe customer cannot miss it and that\u2019s the point: if they see it once they know it\u2019s there and the friction to contribute is low.\nThis also has the benefit of giving a great developer experience to our customers. When someone reports an issue on the doc, it goes straight to our regular support channel, where we have a good response time. We also fix and deploy a majority of the issues within the same day. A company that takes immediate actions on feedback gives an impression of care (one of our core values), and that\u2019s exactly what we are aiming for.\nWhen someone in the team creates or updates a PR, the change will be rebuilt and deployed to a new domain. To achieve that, we use the continuous deployment feature of Netlify, which brings several benefits. The main one is ability to preview, not only for the person doing the PR, but also for the person reviewing it, because they don\u2019t have to deal with running the doc locally for small changes.\nThis is just one example of how to reach out to your readers. It creates a virtuous cycle where everyone (team +","record_index":3},{"post_id":6926,"post_title":"Good API Documentation Is Not About Choosing the Right Tool","post_date":1509444873,"post_date_formatted":"Oct 31st 2017","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/api-documentation-choosing-right-tool\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/Illustration-doc-API-1-360x200.png","time_to_read":8,"content":"\u2026 customers) contributes more and more, and enjoys doing it.\nIn short...\nThere are so many things to consider before worrying about which tool to use. Naturally, you do have to start somewhere and choose a tool, so I advise you to choose the one you and your team are most comfortable with. Just keep in mind to focus on the content, and adapt the tool to the content, not vice versa.\nWe\u2019d love to hear your feedback and other experiences with this topic: @maxiloc, @algolia.","record_index":4},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"Today we\u2019re announcing the release of Multi Cluster Management, a feature dedicated to improving our fellow SaaS customers\u2019 lives when it comes to search.\nWhile our search engine works for every industry, and our features \u2014 from fast indexing to typo-tolerance \u2014 work out of the box for every vertical, there are some needs that are particular to certain verticals.\nBeing a SaaS company ourselves, we share some of the pain points of our SaaS customers, and developed the Multi Cluster Management feature to help address some of these challenges:\n\nRapidly scaling to large data sizes \u2014 several TBs and growing for our current biggest SaaS users\nManaging infrastructure costs and gross margin contributions to ensure business success of SaaS applications.\nManaging and tracking multiple TBs of data across multiple Algolia clusters without additional work required for load balancing and splitting at the user level\nMatching and migrating individual users to the right cluster based on capacity needs\nEnhanced features and service offerings for users depending on their plan or tier level\n\nHow the problem of scale is addressed today\nRegardless of your search provider, when the total size of your indices reaches a certain size \u2014 typically a few hundred GBs \u2014 you\u2019ll need to scale your infrastructure beyond the first cluster or server with which you initially started. For the sake of argument, let\u2019s assume that you reach this limit when you reach 10k users.\nSo while the first 10k users were handled smoothly, scaling beyond that will add a great deal of pain and complexity into your search infrastructure. At this point, you\u2019ll need to redesign your search solution to spread your users into more servers and manage all the additional complexity that comes along with it.\nOn Algolia (without Multi Cluster Management)\nAt Algolia, a single search instance consists of a triple-redundant cluster of servers that are operating in sync to provide the fastest and most reliable","record_index":0},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 search experience possible for your users. Without Multi Cluster Management, if your 10k customers no longer fit on a single cluster, you will need to create a new Algolia application on a second cluster, and manually split your users to balance the load: 5k users on the first cluster, 5k on the other one.\nThis \u201csolution\u201d doubles the complexity of your initial search application as you will now need to manage two Algolia applications \u2014 with two APP_IDs, two sets of API keys \u2014 in addition to tracking which user is on which application.\nThis is where the pain merely begins. When you reach 20k users, both clusters will be full. You\u2019ll need to set up a third cluster, and re-partition your users manually, and again at 30k users, and at 40k users...a painful process indeed!\nOn other engines (like ElasticSearch)\nOn other engines, you would rely on the cross-server sharding mechanism. Your users would be automatically balanced across the different servers, but this balancing comes at a cost.\nFirst, each search query requires a full search across each server (or shard), which increases search latency and lowers users\u2019 perception of your search engine\u2019s performance.\nFurthermore, since the users are automatically sharded across servers, you don\u2019t have the ability to control which user is assigned to which servers, so you can\u2019t make sure that a given user will be allocated the right amount of resources.\nAdd in the fact that as you scale (either up or down) and the number of shards changes, you will need to reindex all of your documents, potentially taking search offline until the indexing is completed or doubling your infrastructure and associated costs in order to avoid this downtime. So once again, a painful process!\nSolving for scale with Multi Cluster Management\nMulti Cluster Management is designed to solve the pain of scaling for SaaS customers on Algolia by making it as easy to manage two, three, four or 100 clusters as it is to manage one cluster. For","record_index":1},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 starters, this means that all clusters will share one application; with one APP_ID, and set of API keys.\nLet\u2019s return to our example above, but this time with 100k users and cluster sizes of 10k users per cluster. To manage this setup with Multi Cluster Management, we would always use the same application ID to access our set of 10 clusters. Assigning users is as simple as making an API call to the application and telling it which user belongs on which cluster. No longer is it necessary to track each user\u2019s location or specific application ID.\nOnce users are assigned, the cluster management becomes abstracted: if you add, update, or delete records for a specific user, there is no need to remember which cluster or application that user is on as the request will be automatically routed to the appropriate cluster. For users, every query entered is always routed to the right cluster with no increase to latency, preserving the instantaneous search experience that they have come to expect.\nAnd finally, let\u2019s say you need to migrate a user to a new cluster for any number of reasons\u2014rapid growth, upgrading to a new tier of service or moving data to a different geographic region. Doing this is now a straightforward, two-step process, with no user facing downtime introduced by this migration:\n\n1. Order a new cluster\n2. Make an API call to assign this user to the new cluster\n\nThat\u2019s it! The user will be automatically migrated with no downtime, no manual operation, and most importantly, no complex migration process.\nThe inverse is also true: adding a cluster or removing a cluster is as straightforward as sending the necessary API calls to migrate users.\nHow this helps SaaS companies\nWith Multi Cluster Management, we have started addressing many key considerations that our SaaS customers experience as their businesses scale:\n\n1. Search performance\n\nYou know how obsessed we are with performance. For that reason, we wanted to avoid having to search across multiple","record_index":2},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 servers when performing a search. With this design, each user is hosted on a single cluster, so a single server can answer queries for a given user. Not having to merge results from different servers allows us to keep the same performance Algolia is known for.\n\n2. Adding and removing clusters\n\nAdding or removing a cluster doesn\u2019t require a complex migration strategy. A few API calls and your new setup is live, with no downtime in the process.\n\n3. Flexibility\n\nYou have a complete control of mapping users to specific clusters, so you can assign users as your needs dictate; for example, you may want to host a specific user on a dedicated cluster, on a shared cluster in a specific region, or even on a dedicated cluster in a specific region.\n\n4. Resource control\n\nBy controlling to which clusters users are allocated, you are able to manage your infrastructure to achieve the most effective and efficient outcome based on your needs.\nTechnical choices we made\nIn this section, we\u2019ll dive deeper into the feature by investigating some of the technical choices we made when creating Multi Cluster Management (MCM). In particular, we want to look at four different topics:\n\n1. Why search on only one server at a time\n2. Why we made user-to-cluster mapping configurable\n3. Why not use clusters with more servers?\n4. What use cases are not covered by Multi Cluster Management?\n\nThrough these four topics, we hope to explain our reasoning in developing this feature, give our users an in-depth explanation of the feature\u2019s capabilities, and finally explain how to optimize its use.\nWhy search on only one server at a time\nThis is probably the most defining design choice of MCM: every search happens on a single server.\nWhen facing a situation where the size of the data is bigger than one server, most search engines decide to shard the data across a cluster with multiple servers using a hash-based mapping: this means that, when searching, you need to query each server, retrieve the matching","record_index":3},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 results, merge the results, and finally provide the answer to the user. As you can probably guess, this introduces a performance bottleneck at the network level between the servers themselves. Requiring multiple servers to communicate lists of results to be able to merge them is slow.\nOne of our main objectives at Algolia is to make search fast. To achieve this with Multi Cluster Management, we avoid communication between different servers of a cluster by storing 100% of the required data on a single server. Fortunately, this is easily achievable for SaaS use cases where the data is easily partitioned on user-by-user basis.\nOne good property of data partitioned by user is the fact you can create small, autonomous slices of data, where each slice of data fits within the storage limitations of a single server. This ensures that each user will fit within a single, Algolia cluster. And because an Algolia cluster is defined as three fully-redundant servers, we need to query only one of the servers within the cluster to get the full list of results - eliminating the network bottleneck discussed above.\nWhy we made user-to-cluster mapping configurable\nOur initial intuition was to automate the assignment of users to clusters, so that the first time you send an object related to a new user, this new user would be automatically assigned to a cluster. However, we found a few use cases which required additional flexibility:\nMulti-region indexing\nLet\u2019s say that \u2154 of your users are in the US, and \u2153 in Europe. Ideally, you would like to have your servers located as close as possible to your users. By making user to cluster mapping fully configurable, it is now within your control to assign users to whichever cluster makes sense to ensure optimal performance for their given regions.\nDifferent tiers of service\nIf you are a SaaS service, chances are you provide different levels of plans. In that case, you may want to provide a different quality of search to different users. For","record_index":4},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 example, you may want to provide advanced search features such as Query Rules\u00a0to your premium plans.\nOr you may even want to make sure that your most important prospects don\u2019t face an issue during the trial or testing phase that could jeopardize their perception of your product, by placing them on clusters with greater search or indexing resources.\nFor these reasons, we decided to keep the mapping of users to clusters configurable: the first time you send us a userID, you need to decide in which cluster this userID will be hosted.\nAnd to make this smooth, we provide two handy tools:\n\n1. A migration tool: re-assigning users is as easy as an API call\n2. An overview of each cluster\u2019s status: we expose an API giving you an overview of how full the different clusters are, and how much capacity your largest users are consuming on each cluster.\n\nIn the future we may provide different strategies to further simplify user assignment. An example of this would be to introduce multiple mapping modes:\n\nManual: you need to provide us the userID<->cluster mapping\nRandom: the users are assigned randomly to a cluster in the pool\nEmpty-First: the users are automatically assigned to the cluster with the greatest unused capacity\n\nWhy not use clusters with more servers?\nAlgolia\u2019s infrastructure is built from the ground up to ensure reliability. For that reason, every single search cluster consists of three fully redundant servers, so that if any two servers go down, the end-user\u2019s search availability and performance will remain unaffected.\nWhen designing MCM, one of our options was to add more servers to these clusters. For instance, we discussed having clusters of four, five, or 15 servers, where each of these servers would have hosted only a subset of the overall data.\nInstead, we decided to trust our initial design choice of three servers per cluster, and then link these clusters together for the following reasons:\nReliability\nHaving each cluster composed of three","record_index":5},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 servers allows us to build a highly available infrastructure where the three servers are hosted within different data centers. This allows us to continue offering a 99.99% SLA by default (and 99.999% SLA as a premium option) for our customers, even when multiple clusters are in use.\nEasier to manage\nFor cluster size, complexity grows with scale. The bigger a cluster gets, the more issues it creates. For example, a cluster of 15 servers has 5 times more chances of having any one server go down. Therefore, using five clusters of three servers each and keeping these clusters independent from one another removes an unnecessary risk created by large cluster sizes.\nWhat use cases are not covered by Multi Cluster Management?\nAs you may have understood by now, one of the core design choices for Multi Clusters Management is the idea that a search will happen on only one cluster at a time.\nThis choice means that there are use cases for which MCM is not applicable. As long as the size of the dataset into which you\u2019re searching at a given time fits into one cluster, MCM will work perfectly. But if the size of that single dataset outgrows a single cluster\u2019s capacity, and that dataset is not partitionable, then MCM will not work.\nFor example, let\u2019s say that you want to create an index of all the known objects in the universe (such as stars and planets). This index may end up representing a few TB of index size. If your plan is to enable search across all of the planets and stars at once, then you\u2019ll need to search into your entire dataset. However, this dataset doesn\u2019t fit onto a single cluster, so in order to run this search query you would need to be able to search simultaneously across clusters. This use case, however, is incompatible with this design, as queries are always routed to a single server to be answered.\nAn example of where MCM would be a fit for searching all the known objects in the universe, however, would be if searches are always done on a solar","record_index":6},{"post_id":6899,"post_title":"Announcing Multi Cluster Management:  SaaS Search Built to Scale","post_date":1509012137,"post_date_formatted":"Oct 26th 2017","author_name":"Xavier Grand","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/cd74d9fc2183df7ba0866bee7b787af1?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-multi-cluster-management\/","categories":["Infrastructure","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Multi-Cluster-360x200.png","time_to_read":11,"content":"\u2026 system by solar system basis. Assuming that no single solar system consists of a data set too large for a single cluster, then MCM will route each search request to the cluster on which that data is hosted and return results.\nThis design choice is what makes Multi Cluster Management particularly adept for SaaS use cases. For example, for CRM companies, the user data is easily partitioned by end user who is paying for the service. From our experience in SaaS, the number of use-cases where a single customer doesn\u2019t fit in an entire server are very rare (we haven\u2019t encountered the case yet).\nBuilding on the tradition of Algolia\u2019s infrastructure solutions\nAt Algolia, we always pay special attention to problems of scale. A few years ago, we launched DSN to give customers the ability to provide great search to users anywhere in the world. We then started to spread our clusters to different data centers to be able to rely on multiple provider for servers, electricity, networks... Multi Cluster Management is a unique\u00a0 \u2014 and we hope \u2014 effective solution to a set of problems that scale presents to SaaS companies.\nWe look forward to hearing your feedback: @algolia, or comment below!","record_index":7},{"post_id":6854,"post_title":"Introducing \u201cWe Rate Tweets\u201d \u2014 a Glitch App for Searching Your Twitter Timeline","post_date":1508409215,"post_date_formatted":"Oct 19th 2017","author_name":"Jessica West","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/fa656ecbbe005e8abd5a013dcd5407dd?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/glitch-app-searching-twitter-timeline\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_GlitchAlgolia-360x200.png","time_to_read":5,"content":"Glitch, the latest project by creators of Trello, FogBugz and co-creators of Stack Overflow, is a new platform for building web applications that features online code editing and instant deployment. In this post, you\u2019ll learn how to use Glitch to host and remix an Algolia-powered search for your tweets.\n\nWant to try it out right now? Try searching through\u00a0Algolia's most popular tweets\u00a0or read the documentation\u00a0to learn how to create and customize your own copy.\nWhat is Glitch?\nGlitch was created out of a desire to\u00a0make code more accessible to people across the board and give companies that make developer tools a fast route for users to start building on top of their product.\n\u201cA key goal of Glitch is to make development more accessible. We\u2019re building Glitch as much for new starters, and those with just a little or rusty programming knowledge, as we are for full-time, professional developers.\u201d- Gareth Wilson\nThe thing we dig about Glitch is the power it puts into the user\u2019s hand to get going fast on a variety of projects ranging from a sample API call to a robust web application or blog. By allowing users to raise their hand when they need help, Glitch is making it easier for new coders to get up and running. It is helping knit communities together with building blocks for coding.\nThere is a plethora of getting-started applications that allow you to do something called Remix,\u00a0which will create a copy of the codebase on your own profile to work with. This allows you to take your idea even further after some of the foundation has been laid for you.\n\nJenn Schiffer, community engineer at Glitch, affectionately talks about the platform as\n\"the friendly community where you'll build the app of your dreams\"\nAlgolia is one of the early adopters of Glitch alongside Twilio, Facebook, Trello, and the Slack API. Visit Algolia's home on Glitch -\u00a0glitch.com\/algolia\u00a0- to see more apps that use the API and learn more about search.\nIntroducing We Rate Tweets\nInspired","record_index":0},{"post_id":6854,"post_title":"Introducing \u201cWe Rate Tweets\u201d \u2014 a Glitch App for Searching Your Twitter Timeline","post_date":1508409215,"post_date_formatted":"Oct 19th 2017","author_name":"Jessica West","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/fa656ecbbe005e8abd5a013dcd5407dd?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/glitch-app-searching-twitter-timeline\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_GlitchAlgolia-360x200.png","time_to_read":5,"content":"\u2026 by one of DevRel team\u2019s favorite Twitter accounts, Dog Rates, we created We Rate Tweets to scratch our own itch of wanting a faster tweet search with a way to classify and rate our tweets by their popularity. Ratings are on a scale of 1-10, or more than 10 in special circumstances. \ud83d\ude09\n\nOnce you have connected your Twitter account, you can start searching your own timeline. We will have already indexed your tweets and assigned a rating score with each one. That way, rather than seeing your search results in a chronological order, you see your most retweeted and liked tweets first.\nIf you recognize that you talk a lot about a subject - selfies with your dog, live-tweeting at a conference, hitting up some sweet yoga class or drinking wine while doing any of those \u2014 you may want to search on keywords you use in tweets, like dogs, JavaScript, wine, or yoga. From there you can see your tweets that match those words, and even share those exact search results via the page\u2019s updated URL.\nJust using the app doesn\u2019t require any coding skills on your part. Just login with your Twitter account and you'll be able to search back through your last few hundred tweets.\nRemix to get your own live copy\nWant to index even more\u00a0tweets and index tweets of other users?\u00a0You can do this very easily with the \u201cRemix\u201d feature on Glitch.\u00a0This will create a copy of We Rate Tweets onto your Glitch profile. From there, you can:\n\nCustomize the look and feel, including the emojis used for rating\nAdd more calls to the Twitter API to get a larger history; the world is your oyster! \n\nYou\u2019ll need a Twitter account and an Algolia account first. Instructions for creating those and everything else are in the README\u00a0on Glitch.\nWe love hearing from you ?\nReach out to us on Twitter and tell us what you think, or join our online community on Discourse and share what you've done with your remix. Happy searching and Glitch-ing!","record_index":1},{"post_id":6826,"post_title":"We Built an Atom Plugin to Find and Install Any NPM Module","post_date":1507211162,"post_date_formatted":"Oct 5th 2017","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/atom-plugin-install-npm-module\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Atom-IDE-Plugin-360x200.png","time_to_read":6,"content":"The desire to automate everything is pretty common, especially with developers: why should I waste my time doing the same operation countless times?\nThat desire is at the base of the process that led to the making of our Atom plugin: an autocomplete that allows importing packages from NPM quickly, with a few keystrokes. On top of this, the plugin will prompt the user to save the selected package as a project dependency if a `package.json` is present.\nWe published the extension on the Atom Package Registry and open-sourced the code on GitHub,\u00a0and here is the story about how we built it.\nThe problem we sought to solve\nMany of us use JavaScript and Node daily for a variety of different tasks: scraping data, slackbots, contacting APIs, indexing content... you name it. Luckily, there are plenty of open source libraries on NPM which we can leverage to perform any task in the simplest and fastest way.\nWe really wanted to solve two main productivity issues in the process of building tools:\n\nLooking up packages on NPM or Yarn, either via CLI or on the web\nInstalling the package in our `node_modules` folder (normally via CLI)\n\nLet\u2019s make a contrived example: let\u2019s say we\u2019re writing a small terminal script which needs to add padding to a string, and we don\u2019t want to re-implement this functionality from scratch.\nWe would normally start by researching the library on NPM or Yarn\u2019s Website, querying for \u201cpad\u201d and evaluating each library by looking at the number of downloads and GitHub stars, and checking if it\u2019s maintained or not.\n\nAfter selecting our library, we need to explicitly require it in our project dependencies using the command line:\n\n \n \n \nFinally, we can get back to our text editor to actually require the package:\n\n \n \n \n \n \nThe number of required steps in order to install a simple library is definitely high and requires to hop from your text editor of choice to a browser and to the terminal - all that for just","record_index":0},{"post_id":6826,"post_title":"We Built an Atom Plugin to Find and Install Any NPM Module","post_date":1507211162,"post_date_formatted":"Oct 5th 2017","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/atom-plugin-install-npm-module\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Atom-IDE-Plugin-360x200.png","time_to_read":6,"content":"\u2026 one library.\nContext Matters\nWhile installing a one-shot library is not a problem, doing this operation frequently can lead to a frustrating experience. We didn\u2019t want to waste time researching libraries or running the same instructions again and again. We rather wanted to spend our time focusing on solving our functional problems and creating product features.\nOne of the first pain points we wanted to solve with our extension was to reduce the need to switch to a browser in order to search for a library. On top of that, we wanted to contextually provide useful information for exploring different options.\n\nIn order to provide a selection of the most relevant packages, we\u2019re using an Algolia index with custom ranking which takes into account the number of downloads from NPM in the last 30 days, in conjunction with a Boolean attribute `popular` computed at indexing time, which sorts the most popular results to appear first.\n\nIn short, having contextual help (in this case, importing JS modules) is tremendously helpful compared to using 3-4 separate tools.\nLet me do that for you\nAfter researching and finding the best suited package for our task, we still need to install it locally on our laptops and track in our project that we\u2019re using that dependency.\nNormally, this is automatically handled by NPM or Yarn by using their CLI; that said, the Atom extension will detect if the package is missing and will prompt the user to save it:\nThe different options come down to different types of dependencies handled by NPM and Yarn, `dependencies` and `devDependencies` being the most used ones for developing a project.\nProof of concept in the time it takes to brew a double shot\nThe idea for Algolia Atom autocomplete was born in front of a coffee machine on a rainy Parisian morning. My colleague Raymond (the author of Vue InstantSearch, amongst other things) and I discussed how we could leverage Algolia to build something that may prove useful to other developers.\nWe remembered","record_index":1},{"post_id":6826,"post_title":"We Built an Atom Plugin to Find and Install Any NPM Module","post_date":1507211162,"post_date_formatted":"Oct 5th 2017","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/atom-plugin-install-npm-module\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Atom-IDE-Plugin-360x200.png","time_to_read":6,"content":"\u2026 that Haroen, another fellow Algolian, built a really nice search UI for Yarn:\n\n\ud83c\udf89 https:\/\/t.co\/BdkV4mw8Vg \ud83c\udf89 check out the new search available everywhere pic.twitter.com\/IvorEaCzN1\n— ${HARO_ENV} (@haroenv) February 25, 2017\n\nAs this implementation is using indexed packages from NPM, most of the job was already done: we just needed to display results where it made sense the most. Chatting by the steaming cup of coffee for 10 minutes, we figured out that we could use a package autocomplete inside a text editor to automatically require packages, the lazy way.\nAfter moving the project to our desks, we choose to go with Atom because its simple JavaScript API layer allowed us to go very fast: from the concept to the first working prototype it took less than a couple hours (also thanks to Ray\u2019s superhuman developer skills). The idea of also saving the dependency came the day after, in a pair-programming session on our lovely couch.\nSince it was a fun hack project we didn\u2019t do any promotion of it at the time. Until today, the only proof that we actually released it was this rather embarrassing tweet.\nLessons learned\nBuilding something quick but useful that could also be useful to other developers feels great. I had a fun time building this project, and I also had an insight or two in the process:\n\nValidate your ideas with the people around you as soon as possible. If the people around your are non-technical, try to see if you can explain the problem clearly without them fainting from exhaustion.\nRe-use existing tools and existing knowledge as much as possible. This will help remove the noise and focus on the actual problem to solve.\nAs soon as you have defined the problem and have a rough idea how to solve it, don\u2019t wait any longer, just do it!\n\nSo long and thanks for all the fish\nIf there\u2019s enough interest around our Atom Plugin, we will consider porting it to other text editors (DEVisual Studio Code, Sublime Text and all the rest), and work a little bit","record_index":2},{"post_id":6826,"post_title":"We Built an Atom Plugin to Find and Install Any NPM Module","post_date":1507211162,"post_date_formatted":"Oct 5th 2017","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/atom-plugin-install-npm-module\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/10\/2017-10_Atom-IDE-Plugin-360x200.png","time_to_read":6,"content":"\u2026 more on cross-platform support (Windows, I\u2019m looking at you).\nIf you have any feedback about this project or want to share something you built using Algolia feel free to leave a comment here, join us on Discourse or ping @algolia, @proudlygeek & @rayrutjes on Twitter.","record_index":3},{"post_id":6802,"post_title":"Serving One Billion JavaScript Library Downloads","post_date":1506504677,"post_date_formatted":"Sep 27th 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/serving-one-billion-javascript-library-downloads\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/delivering-billion-packages-360x200.png","time_to_read":6,"content":"This month, with the help of jsDelivr, we reached a milestone 1 billion downloads (that\u2019s 26TB!) across all of our libraries. Since we now know how to deliver that many JavaScript libraries in a fast, robust and stable way, we thought we'd share our experience with the JavaScript community.\nAlgolia is developing multiple open source projects (like InstantSearch.js) to simplify the integration process of our search engine. When a developer wants to include one of our JavaScript libraries, she can use either:\n\nThe npm package published on the npm registry\nA JavaScript file hosted on a CDN, to be used in a `<script>` tag\n\nProviding those two ways allows us to target different use cases:\n\nAdvanced JavaScript users familiar with module bundlers like webpack\nBeginner or intermediate JavaScript users used to copy\/paste `<script>` tags in their html\n\nOur goal with this strategy is to reduce the friction of using our service at every level of the developer experience.\nPublishing on npm is rather easy: you just run `npm publish` in your directory and that's it.\nCDN delivery of JavaScript libraries\nAs for CDN delivery of our JavaScript libraries, an obvious option would be to find a commercial CDN provider and write a deployment script to automatically upload new versions to the CDN.\nThis also means someone would need to maintain this new system, configure the CDN, update the scripts, monitor uptime (if it fails, our users\u2019 websites are down) and everything that comes with maintaining a live production system. Early on we decided that our main business was search and not delivering JavaScript libraries to the world, so we searched for solutions already providing this.\nThat's when we found out about jsDelivr.\nDelivering your libraries with jsDelivr\njsDelivr is a free CDN built exactly for this. It offers production quality of service and natively integrates with npm.\nOur deployment process looks like this:\n\nAs you can see, the only manual action is to run `npm","record_index":0},{"post_id":6802,"post_title":"Serving One Billion JavaScript Library Downloads","post_date":1506504677,"post_date_formatted":"Sep 27th 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/serving-one-billion-javascript-library-downloads\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/delivering-billion-packages-360x200.png","time_to_read":6,"content":"\u2026 publish`. After that, the npm registry and jsDelivr handles everything.\nSince the beginning, Algolia and jsDelivr have been in close relation. jsDelivr offers Algolia a great way to easily deliver JavaScript libraries to the world. And jsDelivr uses Algolia to provide\u00a0JavaScript search on their website.\nDive into jsDelivr\njsDelivr was launched in 2012 and is currently developed and maintained by Prospect One, a company opened by the original founders of the project. Since the start, it was fully open source and focused on developers, including the development of an API for all public CDNs with the help of the community.\njsDelivr has a unique-among-public-CDNs infrastructure: it uses multiple CDN providers and load-balances between them based on performance and uptime, instead of using a single CDN provider.\nAt the moment, jsDelivr uses Cloudflare, Fastly and Stackpath for all global file delivery except for mainland China, which is exclusively served by Quantil, a company that has a large network of servers across all major Chinese cities. This is made possible by the ICP license held by jsDelivr.\nIn total, jsDelivr serves about 19 billion requests every month. That\u2019s almost 500TB of bandwidth of small ~3kb js files \u2014\u00a0all that from 176 locations all around the world.\nTo see more detail on how jsDelivr works under the hood, check the infographic below that visually explains all the failover systems integrated into it.\n\nPro tip: websites do not need auto updates\njsDelivr has some great features to automatically provide updated version of packages to consumers. For example, you can use scripts like:\n\nhttps:\/\/cdn.jsdelivr.net\/npm\/instantsearch.js@latest would be always updated with any latest version\nhttps:\/\/cdn.jsdelivr.net\/npm\/instantsearch.js@2 would always be updated with the last minor version of instantsearch.js v2 (following semver)\n\nThis sounded very cool for early Algolia: we could update our client websites at any moment with auto updating, and we could","record_index":1},{"post_id":6802,"post_title":"Serving One Billion JavaScript Library Downloads","post_date":1506504677,"post_date_formatted":"Sep 27th 2017","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/serving-one-billion-javascript-library-downloads\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/delivering-billion-packages-360x200.png","time_to_read":6,"content":"\u2026 advertise that they would be always up to date. Alas, WRONG!\nIn fact, this turned out to have many disadvantages:\n\nExtra consciousness about any line of code added to our libraries\nNo way (when using \/latest\/) to make any substantial permanent breaking changes. For example, to release a completely revamped, modernized JavaScript client with a clear upgrade path was not feasible if people used \/latest\/: their website would auto update even before they could make changes to support new versions\n\nUltimately, when your website works and has no issues on Monday, there's no need to push an update on Tuesday from our side. When there are new features and fixes you might want, we will communicate with you via our community forum\u00a0or\u00a0change log.\nNowadays, for production systems, we recommend using fixed versions (like https:\/\/cdn.jsdelivr.net\/npm\/instantsearch.js@2.1.0), and being aware of new updates when you need them.\nStill, auto updating packages has their advantages. When creating jsFiddle or CodeSandbox examples for our users to show us bugs to be fixed, we use auto updating so that we never have to manually updates those templates.\nThe future of JavaScript delivery\nJavaScript modules are a great addition to standardize the modularity aspect of JavaScript apps. If you want to learn more about them, read the modules chapter of Exploring ES6 book by Axel Rauschmayer.\nThis feature is now coming to Node.js and browsers, which means there will be adjustments to make to both our build and delivery strategy, so we can support new usages like `<script type=\"module\"><\/script>`.\nWe trust jsDelivr to help us make the right decisions and potentially build new features for JavaScript modules.\nBig thanks to Dmitriy Akulov for his help with this post.\u00a0","record_index":2},{"post_id":6774,"post_title":"From Windsurfing to Venue Search: Great Search Implementations","post_date":1505987188,"post_date_formatted":"Sep 21st 2017","author_name":"jason harris","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a0afd70a440899174387474bc9573a23?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-as-a-service-implementations\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_show-and-tell-360x200.png","time_to_read":4,"content":"Want to show off what you\u2019ve built with Algolia? The\u00a0Show and Tell category on our Community Forum is a great place to gain visibility for your project and get feedback at the same time. Our 1,100 community members actively swap stories of success and overcoming obstacles on their projects.\nWe ? our community and enjoy seeing the many ways developers are building with Algolia, including the great search experiences built within large enterprises, small business, and those midnight-oil burning side projects that include massive amounts of creativity and originality.\nToday, we\u2019ll highlight recent posts to the Show and Tell category so you can see some great ways to implement search.\nDigging into PHP's #internal mailing list\nCommunity member Mattieu Napoli built externals.io as a way to give PHP's #internal mailing list a new level of accessibility. The site uses an Algolia InstantSearch to afford the user threaded views, search and up\/down voting to enhance the UX. Externals.io also features highlighting and inline results in a responsive manner.\nSee Externals.io on ProductHunt and Algolia's Community Forum.\n\nFinding the right windsurfing equipment\nBuilt by Spanish developer Kikobeats, windtoday.co is a marketplace to find deals on windsurfing equipment. Windtoday.co enables visitors to find windsurfing goods based on category, brand, condition, and more, using great features like type-ahead search and facets. Also, for those on mobile, Kiko recently introduced a mobile version of windtoday.co using react-instantsearch.\n\n \nPowering philanthropic giving with Grantmakers.io\nGrantmakers.io is a site intended to provide live search, summary data and rich profiles for 71,042 US-based foundations. Developer Chad Kruse built Grantmakers.io with Algolia, Jekyll and GitHub Pages in this great example of a search implementation. Even better, Chad has open sourced everything on GitHub for others to learn, hack and build upon.\n\nUniting blog content and e-commerce sales","record_index":0},{"post_id":6774,"post_title":"From Windsurfing to Venue Search: Great Search Implementations","post_date":1505987188,"post_date_formatted":"Sep 21st 2017","author_name":"jason harris","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a0afd70a440899174387474bc9573a23?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-as-a-service-implementations\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_show-and-tell-360x200.png","time_to_read":4,"content":"\u2026 through search\nOften times it\u2019s hard for companies to find a way to tie content marketing and ecommerce sales together. With AskFrannie (an essential oils website), developer Rolondo Garcia built a custom WordPress plug-in that uses product search at the bottom of articles, and invites the visitor to click to an online catalog of products. This method is great because it matches reader intent with relevant products in a casual way. That is, if a reader has read an entire article on, say, essential oils, they\u2019re more likely to have a desire to see products presented in a relevant plug-in that a banner ad on the site.\n\nFind event space at hotels, restaurants and activity spaces\nAsk any event manager and they\u2019ll tell you that finding the right event space at the right location can be very time consuming. Bizly.com aims to solve the problem by enabling a curated venue search in major cities across the United States. Powered by optimized SQL queries, Laravel Scout and Algolia, Biz.ly\u2019s search is lightning-fast and very rich.\nLet the community see what you\u2019ve built\nFeeling inspired by these Show and Tell examples? Head on over and share your post. An ideal post describes how the search works, the Algolia libraries you're using, and other parts of the stack including languages, frameworks and APIs. The community also loves screenshots and a link to your project. We look forward to seeing what you've built!","record_index":1},{"post_id":6747,"post_title":"Enhance & Promote Results: Introducing Query Rules","post_date":1505235844,"post_date_formatted":"Sep 12th 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/promote-search-results-query-rules\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_Query-rules-1-360x200.png","time_to_read":6,"content":"Today, we are releasing Query Rules, a new feature which enables you to modify, override, & enhance the behavior of the engine\u2019s configured ranking for a subset of the queries. We wanted to share with you how we approached this key addition to our API, why we decided to build it, and explain the design steps leading to the release today.\nBefore we dive in, let\u2019s look at a few examples of what you can do with Query Rules:\n\nTransform descriptive text into a filter: for example, by automatically transforming a query word into its equivalent filter (\u201ccheap\u201d would become a filter price< 400; \u201cred\u201d a filter on the color\u2026)\nMerchandising: manually select a result for a specific query (e.g., decide that the latest iPhone 8 should be in the first position for the query iPhone)\nEntities detection: detect entities (colors, brands\u2026) in the queries and transform them into filters or boosts\n\nBuilding upon our ranking formula\nWith Algolia, the same relevance algorithm is applied across the entire index. We expose features like Custom Ranking so that customers can customize the ranking strategy for their needs, and achieve a great relevance for the vast majority of queries. However, in the past few years, our customers started to bring to us examples of outlier queries.\nWe began compiling a list of these outlier situations. Here are a few examples:\n\nWhen Apple releases a new iPhone, it should be in the first position of the list for the query \u201ciPhone\u201d (even though at this stage it has no views, no sales\u2026 no \u201clogical\u201d reason to be ranked high)\nYou need to get rid of a big stock of a specific model of a vacuum cleaner, so you\u2019d like it to be promoted in the results (at least until enough units are sold)\nThe query \u201cphone\u201d naturally (according to the configured ranking strategy) retrieves feature phones instead of smartphones, but you\u2019d like to highlight smartphones first\nThe query \u201cred dress\u201d matches with non-red results because the name of","record_index":0},{"post_id":6747,"post_title":"Enhance & Promote Results: Introducing Query Rules","post_date":1505235844,"post_date_formatted":"Sep 12th 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/promote-search-results-query-rules\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_Query-rules-1-360x200.png","time_to_read":6,"content":"\u2026 the brand begins by \u201cred\u201d\nThe query \u201ccheap phone\u201d doesn\u2019t return any results (the records don\u2019t contain the word cheap), and would behave better if \u201ccheap\u201d was transformed into a numeric filter on the price\nQueries originating from users on mobile devices should highlight results that are mobile-friendly\n\nThere are hundreds of examples like this where having an exception to the general rule would make sense, either to improve the relevance, or to override the ranking for business reasons.\nThinking about our options\nThere are two main ways to address the types of exceptions we were seeing. The natural way to handle this would be to analyze the use cases one by one and add a configuration to the engine to handle each of them individually. We could, for example, develop a form of synonyms that would transform a word into a filter. Eventually, these settings would form a merchandising tool, allowing users to tweak and override the ranking logic.\nWe certainly had the experience on the team to execute on this approach. Several team members, including our founders, have used, or even built merchandising platforms prior to founding\/working for Algolia. However, it is exactly because of this experience that we had doubts that this was the right approach:\n\nCreating a new setting inside the engine for each exception quickly leads to a bloated solution: there are just too many varieties of merchandising strategies\nBuilding a quality merchandising tool that would work for all of our users - each of whom have specific needs & exceptions - is virtually impossible\nMultiplying the exceptions inevitably affects performance\n\nMore importantly, we wanted to do more than merchandising and address the needs of other industries. Media sites don\u2019t use merchandising, but they still want to promote, for example, partner content. SaaS systems may want to improve the ranking by adding rules automatically based on the output of a machine learning tool.\nTo do this, we would","record_index":1},{"post_id":6747,"post_title":"Enhance & Promote Results: Introducing Query Rules","post_date":1505235844,"post_date_formatted":"Sep 12th 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/promote-search-results-query-rules\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_Query-rules-1-360x200.png","time_to_read":6,"content":"\u2026 need to be able to impact search results on a subset of queries in two distinct places:\n\nBefore the search query is processed, in order to override the query parameters\nAfter the results are found, in order to modify the results list\n\nThe solution\nWhat we came up with is a rules system for queries \u2014 or Query Rules \u2014 that sits inside the engine via two modules:\n\nA query preprocessing module that will modify the search parameters before the search is processed by the engine\nA results postprocessing module that will modify the results before they are sent back to our users\n\nEach rule has the following pattern: IF the query contains X, THEN modify Y. A condition and a consequence:\n\nIf the query contains \u201ciPhone\u201d, add this result in the first position\nIf the query is \u201ccheap phone\u201d, replace the word \u201ccheap\u201d by a numeric filter price<200\nIf the query contains a color, filter by that color\n\nThis approach makes it simple to modify the behavior of a subset of the queries, without impacting the rest of the queries. We created two types of conditions, and seven types of consequences, which together allow us to handle a variety of exceptions, including:\n\nPromote the latest iPhone on the query \u201ciphone\u201d\nFilter on color white for the query \u201cwhite dress\u201d\nFilter on price<400 for the query \u201ccheap phone\u201d\nDisplay a promotional banner for the query Samsung\nRemove the word \u201chotel\u201d in a hotels search\n\nConclusion\nWith Query Rules, we\u2019re bringing to Algolia the ability to handle queries on an individual basis by making exceptions on the regular ranking strategy.\nWe think the approach we took has a few interesting benefits:\n\nIt is low-level and flexible enough to adapt to a large variety of use-cases without requiring a lot of settings and configuration\nIt is exposed via an API, allowing both our customers and potential partners specialized on merchandising to build upon it\nIt doesn\u2019t compromise performance for the sake of feature sets: you can add up","record_index":2},{"post_id":6747,"post_title":"Enhance & Promote Results: Introducing Query Rules","post_date":1505235844,"post_date_formatted":"Sep 12th 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/promote-search-results-query-rules\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/2017-9_Query-rules-1-360x200.png","time_to_read":6,"content":"\u2026 to a million rules to an index, with little to no impact on search performance\n\nIn fact, we like to think of Query Rules as more than a feature: we like to think of it as an extension of our engine. We have designed it to be open-ended enough to allow our users to solve their own unique problems and push the boundaries of what is possible with search.\nIt\u2019s been exciting to start sharing the feature with a few beta testers \u2014 we\u2019ve been amazed at how easily they grasped its potential, and the combinations it allows.\nThe two conditions and seven consequences we are initially releasing are only the beginning \u2014 we look forward to getting your feedback and learning which ones you\u2019d like us to add next!\n\nMeanwhile, we put together an online workshop, where two of our team members will show you in practice how to create smarter search with Query Rules.\n=>\u00a0You can register here.","record_index":3},{"post_id":6728,"post_title":"6 Tips to Make the Most of a Hack Day","post_date":1504767927,"post_date_formatted":"Sep 7th 2017","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/6-hack-day-tips\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/Hack-day_Illustration-360x200.png","time_to_read":6,"content":"When I joined Algolia, the InstantSearch team was composed of two developers. Ten months later, we doubled the team and will have two more engineers joining us soon.\nWe are now focusing on several big projects at the same time. Even though we are a single team, we are inherently split into different projects. This means that not everyone is interacting with one another on a regular basis.\nIn addition, we don\u2019t always have enough time to step back and try new things. As we were thinking of a way to solve those issues, the hack day retreat was born. The concept is simple: 6 team members, 5 hacks, 1 place outside the Paris office, and then non-coding team building activities to round off the day.\nHere are some tips you might find useful if you want to organize a similar hack day.\n1. Prepare the hacks upfront\nTo prepare for this special day, we were asked to think about several hack ideas we wanted to realize upfront. The idea was to choose the one we wanted to work on the day before. This is a mandatory step because you don't want to spend your time looking for an idea on D-day. You\u2019ll only have something like six hours dedicated to coding \u2014 use them wisely.\nYou should also make sure that everyone spent some time to do pre-research and pre-work around the hack, particularly if they know that they will manipulate things they\u2019re not very familiar with.\nFor example, one of the hacks involved virtual reality throughout React VR plugged with a speech-to-text feature. Prior to hack day, we bootstrapped a blank app and made sure that those blocks were correctly connected to each other. When talking to the computer through the browser, the VR part would receive the corresponding text. Without doing that in advance, we could have spent half of the day configuring it.\nA hack implies a lot of unknowns, so you really don\u2019t want to get stuck with configuration issues.\n2. Have a hack day outside of your office, in a nice place\nAs you're going to spend a full day coding,","record_index":0},{"post_id":6728,"post_title":"6 Tips to Make the Most of a Hack Day","post_date":1504767927,"post_date_formatted":"Sep 7th 2017","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/6-hack-day-tips\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/Hack-day_Illustration-360x200.png","time_to_read":6,"content":"\u2026 it's important to find a nice place where everybody can feel great and enjoy themselves. Being outside of the office really helped switch our brains off the regular daily routine and be totally focused on the hacks.\nWe manage to find a nice loft in Paris\u2019s 11th arrondissement, full of commodities for everyone:\n\n3. Take the opportunity to empower junior developers\nIf you happen to have a junior profile in your team (we recently onboarded one), a hack day is definitely an opportunity to learn a lot, albeit with some slight modifications. Here a few tips to make the most of it:\n* Never assign a junior person a solo project. It\u2019s the best way to make them feel excluded and not integrate with the team.\n* Choose a simpler project that will cover some fundamentals related to the team's day-to-day job (for instance, how to build a simple application with React).\n* Put the emphasis on pair-programming and sharing knowledge (it is the perfect time to learn some workflow or debugging tricks!)\n4. Focus on making an MVP\nFocus on making a Minimal Viable Product rather than the best product possible. It's way more rewarding to present something that works during the demo than a vision of something that may never happen. Try to add things step by step without thinking too far ahead.\nFor instance, in one of the projects, a spreadsheet was used: you don't have to think about how it can scale and spend time researching the best tool to handle infinite scroll. Instead, you can allocate this time trying to go deeper in the project's features. Here as always, setting priorities is key to effective time management!\n5. Demonstrate your hacks\nDon't end the coding time abruptly. Instead, plan a demo time where everybody can showcase what they did. If the team can manage to push the demos on a repository afterwards, that's even better, as you'll be able to share the content with everyone interested.\nHere are the hacks we realized on our hack day:\n\nAMP + Algolia\n\nAccelerated Mobile Page is","record_index":1},{"post_id":6728,"post_title":"6 Tips to Make the Most of a Hack Day","post_date":1504767927,"post_date_formatted":"Sep 7th 2017","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/6-hack-day-tips\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/Hack-day_Illustration-360x200.png","time_to_read":6,"content":"\u2026 a page format proposed by Google. Its goal is to speed up their display on smartphones. The goal of this hack was to find out if you could build such a page using Algolia. See what Haroen discovered on this GitHub thread.\n\ncreate-instantsearch-app\n\nReact has create-react-app to quickly bootstrap a React application, but they were no equivalents to start easily crafting a vanilla js app using instantsearch.js. \nCheck out what Alex built here: create-instantsearch-app.\n\nEditing indices from a spreadsheet\n\nThe goal of this project was to create a simple app allowing you to edit your index from a spreadsheet:\u00a0https:\/\/github.com\/algolia\/algosheet\/\n\nIndex videos transcripts and search inside them\n\nThe goal of this project was to index transcripts of YouTube videos and offer to search through them. Looking for a quote? You\u2019ll be redirected to the right place inside the video.\n\nReact VR + React InstantSearch (featuring Speech-to-text)\n\nIn late 2016, we launched React InstantSearch compatible with React and React Native application. This hack was the occasion to see if it was working fine with React VR and to think of a new search interface using voice.\nCheck out what Marie built here: react-vr-feat-react-instantsearch\n6. Mix fun and work\nWhile coding is fun, you'll also want to do some team building activities involving everyone at the same time. We choose to do an escape game at the end of the day as it's a great way to have fun while strengthening team ties. It's also a way to discover how your teammates are thinking and to collaborate.\nWe had a great experience staying for 60 minutes in the vastness of the ocean trying to restart the submarine engine and make the torpedo ready for any possible threats... and guess what? We all managed to accomplish our mission.\n\nFinally, you can wrap up the day by sharing a drink and a dinner. It gave us the opportunity to share our feelings about our work at Algolia and the place of our team \u2014 and to simply enjoy the moment.\nThe","record_index":2},{"post_id":6728,"post_title":"6 Tips to Make the Most of a Hack Day","post_date":1504767927,"post_date_formatted":"Sep 7th 2017","author_name":"Marie-Laure Thuret","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/c2ba74de85628d77813868a3fd791a5a?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/6-hack-day-tips\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/09\/Hack-day_Illustration-360x200.png","time_to_read":6,"content":"\u2026 hack day was definitely a success for us: it was a great parenthesis from our day-to-day jobs, and not only did we have some time dedicated to exploring other things (be it AMP, virtual reality, or exploring the possibilities of Algolia), but it was also the opportunity to get to know each other in a different context.\nIt was the first time we organized a hack day for our team but it will definitely not be the last. We are thinking about what we could do next time to make it even better. Have ideas? Comments? Lessons learned from your own hack days? Let us know: @mlthuret,\u00a0@algolia","record_index":3},{"post_id":6696,"post_title":"A Simple, Secure Tool for One-time (Self-Destructing) Messages","post_date":1504168964,"post_date_formatted":"Aug 31st 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/secure-tool-for-one-time-self-destructing-messages\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Secret-Messaging-Tool-360x200.png","time_to_read":5,"content":"It is a very common practice and a very bad idea to send sensitive information over Slack or email. No matter how much you trust Slack or Gmail, there are types of company information (for example, SSH keys, certificates, security tokens...) that warrant an extra layer of security.\nThe challenge, then, is to create a more secure platform that is also easy to use in order to invite adoption.\nA complicated way to tackle the issue\nAlgolia\u2019s Foundation Squad, which I am a proud member of, sends secrets and passwords on a daily basis. Existing ways of sending secure messages were quite cumbersome: you had to first ask the receiver to create a private and a public key, have them publish it somewhere like a web site, MIT\u2019s public key service, or KeyBase, or send it directly to our squad. Only then we could start sending secure messages to each other.\nWhile operations engineers are used to (if not happy with) this level of effort, asking other teams\u2014\u00a0dev, sales, marketing, execs \u2014 to indulge in such a procedure is simply not practical: I could already see employees reverting back to email or Slack.\nExperience has shown us that the best way to mandate security measures is to make them as simple and easy to use as possible. Hence, I started racking my brain to find a new solution.\n\u201cThis tape will self-destruct in five seconds. Good luck, Jim\u201d\nA solution is emerging\nWhile working on a different project in our continuous efforts to make Algolia security top notch, I started using Hashicorp Vault. Vault \u201csecures, stores, and tightly controls access to tokens, passwords, certificates, API keys, and other secrets in modern computing. Vault handles leasing, key revocation, key rolling, and auditing through a unified API\u201d. Through Vault, Hashicorp changed the way companies store their secrets.\nDigging a little deeper inside the Vault API and engine, I stumbled upon the concept of Response Wrapping. \u201cResponse Wrapping\u201d is using Vault\u2019s cubbyhole backend.","record_index":0},{"post_id":6696,"post_title":"A Simple, Secure Tool for One-time (Self-Destructing) Messages","post_date":1504168964,"post_date_formatted":"Aug 31st 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/secure-tool-for-one-time-self-destructing-messages\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Secret-Messaging-Tool-360x200.png","time_to_read":5,"content":"\u2026 Cubbyhole is a short-term storage place where you can put a secret that will be consumed by a user or a service who is holding a specific token. No token can access another token's cubbyhole, whether to read, write, list, or for any other operation. When the token expires, its cubbyhole is destroyed.\nSo, a Vault token has all the features we need to create a self-destructing message service:\n\n1. TTL - a time during which the token will be alive\n2. Number of uses - the number of times you can use the token\n3. Cubbyhole - a special place where we can save secrets\n\nLet\u2019s put all of these features into use and create a secret message service workflow:\n\n1. User inputs secret message\n2. Create a token with TTL=24Hours and NumberOfUses=2\n3. Write the secret to the \u201ccubbyhole\u201d\n4. Token NumberOfUses - 1 = 1\n5. Give the token back to the user\n6. User sends token to relevant recipient\n7. Recipient uses token to open the secret message in cubbyhole\n8. Token NumberOfUses - 1 = 0 ; hence the token and secret are deleted\n9. Success!! \\o\/\n\nYet Another Secret Messaging Service?\nNow, before building the final tool, I did some research to make sure I am not reinventing the wheel. I decided to look for a self-destruct messaging software, and found a couple of candidates, but they all had at least one of the following issues:\n\nThey didn\u2019t allow the option of self hosting, which made security an issue, thus defeating the purpose\nNot simple enough to use\nThey required a complex deployment on the user\u2019s part, such as installing Redis, Node and other dependencies\nThe backend storage is typically not that secure\nThey are not open source\n\nSide note: I don't explicitly list the tools because the \"domain\" of secret messaging services is so tiny, that I believe your own research will take only a few minutes to come to the same results as I have. You can see a list of some nice projects here:\u00a0\nhttps:\/\/github.com\/Kickball\/awesome-selfhosted#pastebins\nThe research justified building a","record_index":1},{"post_id":6696,"post_title":"A Simple, Secure Tool for One-time (Self-Destructing) Messages","post_date":1504168964,"post_date_formatted":"Aug 31st 2017","author_name":"Eran Chetzroni","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/981e889b16e26b876ccd24b26a77a75c?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/secure-tool-for-one-time-self-destructing-messages\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Secret-Messaging-Tool-360x200.png","time_to_read":5,"content":"\u2026 new tool with the following requirements:\n\nHosted on our server (aka super secure) easy to deploy\nEase of use\nUsing Hashicorp Vault (nust as an experiment)\n\nAll I had to do now is create a very simple API with 2 public methods:\n\nSetSecret - which puts the secret in Vault and returns a token\nGetSecret - uses the token and gives back the secret\n\nOn top of that I built a very simple web UI:\n\n1. You insert your secret, submit it\n\n\n2. You get a URL with the one time token\n\n\n3. You send the URL to the happy recipient via Slack or email\n\n\n\n\n\n\n4. She opens the secret message\n\n\n\n\n\n\n\n\n\n5. If you try to open the message again\n\n\nAnd that\u2019s all there is to it!\nWhile this was at first just an experiment in using Hashicorp Vault for a secret messaging tool, it has really caught on at Algolia, where I see many coworkers using it for all kinds of secrets sharing.\nIf you like the tool, you can try it yourself on GitHub. It is open source, and we put on ProductHunt so it can be found easily (and of course, we\u2019d love your vote \ud83d\ude42\n\nLet us know what you think!","record_index":2},{"post_id":6684,"post_title":"Bringing the Missing Frontend Search Block to the Laravel Vue.js Community","post_date":1503467643,"post_date_formatted":"Aug 23rd 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/laravel-vue-js-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Vue-Instant-Search-360x200.png","time_to_read":5,"content":"Last year, Taylor Otwell, the maker of Laravel, decided to create an official Laravel package named Scout. The package allows Laravel users to add full-text search on top of their existing database without much effort.\nI was very happy to notice that Taylor identified search as being something complex that developers would need help with. I was even more excited by the fact that he chose Algolia as the default engine.\nVue.js and Laravel === the perfect mix\nIn Laravel 5.3, Taylor Otwell chose to power the frontend with the Vue.js framework. Every Laravel app since 5.3 now ships with a ready-to-use Vue.js skeleton.\nBecause Taylor made that move, Vue.js instantly got my attention. My thought process was that if Taylor chose to ship his backend framework with Vue.js in the frontend, it probably means that Vue.js shares some core values with Laravel. Vue.js is easy to learn, it thrives at empowering developers to add logic to the frontend, and it advances forward quickly.\nI remember this one time I opened a very naive PR to adapt the lifecycle of the components in core of Vue.js. Evan You, maker of Vue.js immediately picked it up:\n\nTook a long flight and got 10 new PRs when I got off the plane \ud83d\ude02\n— Evan You (@youyuxi) March 3, 2017\n\nLaravel and Vue.js are each made and driven by a single person. While some might argue this is a disadvantage, I see it as an opportunity to have opinionated people be more innovative and also be more reactive. This is even more true when those people have a very clear idea of where they are going and manage to stay open to the community.\nBuilding a UI components library for Vue.js\nBuilding search on the frontend has the advantage of offloading all search related operations from your Laravel app. Thanks to the Laravel Scout package, little setup is required to send your data to Algolia. I wanted to help developers also benefit from great search experience \u2014 this time, on the frontend.\nWe wanted to create a library that would embody","record_index":0},{"post_id":6684,"post_title":"Bringing the Missing Frontend Search Block to the Laravel Vue.js Community","post_date":1503467643,"post_date_formatted":"Aug 23rd 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/laravel-vue-js-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Vue-Instant-Search-360x200.png","time_to_read":5,"content":"\u2026 the same principles as Laravel: something users could easily install in existing applications and shape for their own need.\nVue.js offers a very convenient way of creating components in single files. This approach is very readable and makes it easy to reason about the search experience you are building.\nSo, we ended up bootstrapping a bunch of components that can just be dropped into your Vue.js skeleton. Components will then handle all the logic consisting of listening for user changes and asking Algolia for the new results:\nView the code on Gist.\nAll you need in order to use the above syntax in your own Vue.js application is to actually register the plugin:\nView the code on Gist.\nHow I challenged Vue InstantSearch\nAfter sharing my excitement about this new library, other engineers at Algolia wanted to try it out. They were intrigued by both Vue.js and Vue InstantSearch, so we started writing some documentation in order for them to try out my work.\nTo allow users to experiment with the library, I created a step-by-step getting started guide. When you reach the bottom of the guide, you have a working instant search experience.\nWe were really happy when testers told us they managed to get their search up and running in a couple of minutes, even insisting on the fact that they knew nothing about Vue.js previously. At the same time, I have to say that the feedback was largely due to the benefits of Vue.js framework itself, since Vue InstantSearch leverages its available features.\nBecause of the excitement around the library, and because we had already been invited to speak at the annual Laracon US event, we thought giving a preview of the library at the conference would be a good idea. We didn\u2019t expect much other than to share the idea with the community and get some feedback in advance of the full release.\nMaxime Locqueville gave a live coding talk where he explained how to index data with Laravel Scout, and how to leverage Vue InstantSearch. By the end of the 30","record_index":1},{"post_id":6684,"post_title":"Bringing the Missing Frontend Search Block to the Laravel Vue.js Community","post_date":1503467643,"post_date_formatted":"Aug 23rd 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/laravel-vue-js-search\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Vue-Instant-Search-360x200.png","time_to_read":5,"content":"\u2026 minute talk, he had illustrated how to build a fully blown instant-search experience with customized relevance. It was the first time Vue InstantSearch was mentioned in the wild, and then came this:\n\n\ud83d\udc83 Blazing-fast instant search coming to Laracasts tomorrow. pic.twitter.com\/vIZiAoMV0X\n— Jeffrey Way (@jeffrey_way) July 31, 2017\n\nLaracasts is the place where Jeffrey Way explains Laravel A-Z. I was really honored that Laracasts didn't just talk about us, but were the first real Vue InstantSearch users.\nTo be honest, I was a bit scared that Jeffrey would find blocking issues with the state of the library at that time. He raised some small questions in his first recording which we quickly addressed, and I was also very glad he gave us some more feedback via direct message.\nThe Vue InstantSearch journey is just starting\nToday we have over 100 users and about as much GitHub stars, but the project was still considered unreleased until today, because I wanted to make sure we had the right docs and enough feedback from testers.\nThis project has been so far the best pre-launch experience I\u2019ve witnessed, largely thanks to \u00a0the communities it addresses. I\u2019d also like to give a special mention to my colleague Haroen who has been challenging ideas and reviewing a fair amount of everything that is in Vue InstantSearch. The project would not have been in the current stage without him.\nV1 is, of course, just the beginning. Vue.js & Laravel have a strong future and we are excited to be a small part of it. Now, what can we do better? How can we help your use case? We look forward to your feedback: @algolia & @rayrutjes.","record_index":2},{"post_id":6664,"post_title":"Redesigning Our Pricing from the Ground Up","post_date":1502864302,"post_date_formatted":"Aug 16th 2017","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pricing-update\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Screen-Shot-2017-08-16-at-13.28.30-2-360x200.png","time_to_read":3,"content":"Today we\u2019re rolling out new pricing plans. You can see them \u00a0on our new pricing page and get some quick answers in our FAQ.\nCustomer feedback is in our DNA and core to how we make decisions about our product and company. We heard loud and clear from our customers that our pricing structure was boxing them in to using the Algolia platform in a specific way, so today we're doing something about that. As of today, we're rolling out a brand new tiered pricing structure that will give companies of all sizes the flexibility to use the Algolia platform however they choose.\nSpeaking with Algolia customers in a variety of market segments, verticals and stages of growth, we kept coming back to two main points of feedback:\n\nAs users get started with Algolia, they want the pricing to be easy to understand, with costs that are in line with their usage. \nAs companies grow both in size and usage of Algolia, they are looking for enhancements in the customizability and flexibility of our platform. \n\nChanging pricing is never an easy task. Many companies like ours make decisions about whether to optimize pricing for many small customers (startups, SMBs) or a small number of large enterprise customers. We have always taken the bold position that Algolia is for teams and projects of all sizes, and based on the feedback we received, we approached our new pricing tiers with this philosophy in mind.\nThe first major change in our pricing is that pricing is no longer connected to search operations. We want all customers to be able to create the best search for their users without thinking about how it impacts cost. As-you-type search, with API calls made at each keystroke, is a core element of a successful experience, and therefore searches via the Algolia API are now free for all Algolia users. To ensure the reliability of the service, there is a rate limit on the number of Queries Per Second (QPS) that users are able to perform.\nNext, we\u2019re introducing a new Essential Plan which is","record_index":0},{"post_id":6664,"post_title":"Redesigning Our Pricing from the Ground Up","post_date":1502864302,"post_date_formatted":"Aug 16th 2017","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/pricing-update\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Screen-Shot-2017-08-16-at-13.28.30-2-360x200.png","time_to_read":3,"content":"\u2026 designed to give product teams more control over their search costs based on usage. Starting at $35\/month, Essential Plan users will be charged based on the number of records and the number of Indexing Operations \u00a0(edits to indices or to index settings performed). \nFinally, we\u2019re launching the notion of add-ons. Add-ons are specific features and products that customers can purchase in addition to their plans in order to further customize their implementation of Algolia. We have several add-ons that are available starting today, all available on our Pricing Page.\nWe designed our pricing with the goal of making it better for both new and current users; however, if you're a current customer and are happy with your current plan, that's great! You can remain on your existing plan indefinitely. We support the commitment we made when you first signed up for our service, and will let you decide what works best for you at this time and in the future. \nWe love feedback, so should you have any questions or concerns, please do not hesitate to reach out to us at support@algolia.com.","record_index":1},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"Silky smooth interactions are critical for providing a natural-feeling application. The devil is in the details, and ill-performant web animations feel awkward, \u201cjanky\u201d, and, above all,\u00a0slow. Developers often invest quite a bit of time to reduce first page loads by even a few milliseconds, but forget to consider the impact of the interactions that follow.\nLike many at Algolia, I\u2019d consider myself user experience obsessed. Performance is a crucial part of our formula for a stellar user experience: just as a speedy search leads to happy users, so do performant web animations.\nMeasuring Success\nYou can use the frame rate of an animation to measure how responsive an application feels. 60 frames per second is the generally accepted target for achieving a \u201cnatural\u201d feel. What this means for developers is there are roughly only 16.7ms (1000 milliseconds divided by 60) to achieve all the work that has to happen in each frame. Consequently, the overarching goal is to limit the amount of necessary work. \nAs with all things in life, this principle comes with a caveat: more frames means more processing, which means frames might be dropped. If all the updates necessary won\u2019t fit into the 16.7ms time allotted, it might be better to deliver lower frame rates (30fps) more consistently.\nBrowser 101: How a Pixel Is Born\nBefore diving too far into the nitty-gritty, it\u2019s important to take a step back and ask: how is the browser actually generating the pixels a user sees from the code you write?\nOn initial load, the browser downloads and parses the HTML, and then converts elements into a \u201ccontent tree\u201d of DOM nodes. Additionally, style information gets parsed and calculated to generate the \u201crender tree\u201d. In the interest of efficiency, the rendering engine may split up the work that needs to be done, and certain parts of the render tree may be built before all the HTML is done being parsed.\nExample of a call tree of an initial page load\nLayout\nOnce the render tree","record_index":0},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 is completed, the position and size of elements is recursively calculated from the top left corner to create the layout. This computation can generally be accomplished in one pass, however, it may require additional passes depending on the flow of elements. Element placement is highly interdependent. In order to optimize the necessary work, the browser tracks changes and marks these elements and their children as \u201cdirty\u201d. However, due to the heavy correlation between elements, any layout changes will be quite expensive and should be avoided.\nPaint\nAfter the layout is created, content is displayed on the screen by the painting process. During this step, visual styles are used to paint the page according to the correct visual formatting model. Similarly to the layout process, \u201cdirty\u201d elements are tracked and consolidated into one larger rectangle. Re-paints will only occur once per frame to redraw this \u201cdirty\u201d region. As such, re-paints require a large amount of work and should be avoided. \nComposite\nIn the final step, all the painted elements are composited. By default, all elements are painted into a single memory layer; however, by separating elements onto compositor layers, updates will only affect the elements on their respective layer. The CPU draws the layers, while the GPU generates the layers. Hardware-accelerated compositing is incredibly efficient at basic drawing operations. The separation of layers allows for non-destructive changes. As you may have guessed, changes to GPU-composited layers are the least expensive.\nGetting Creative\nAs composite-level changes are the most performant, the only properties that should ever be changed are those that only trigger compositing. These properties are opacity and transform. Seriously, that\u2019s it. However, this isn\u2019t as limiting as it might first seem \u2014 you will just have to get a bit creative.\nTransformation\nTransformation allows for endless possibilities of visual changes to an element: you can","record_index":1},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 position it (translateX, translateY, or translate3d), scale it, rotate it, skew it, or even apply a 3-dimensional matrix transformation. In some cases, you might need to shift your thinking and consider how a change that would cause a re-layout or re-paint could be achieved with a transformation.\nAs a quick contrived example, consider the case of a \u201cbox\u201d element that is shifted to the left 10 pixels when an \u201cactive\u201d class is applied. \nInstead of changing the \u201cleft\u201d property (like below):\nView the code on Gist.\nConsider translating the element across the x-axis:\nView the code on Gist.\nOpacity\nBy changing opacity level, you can easily show and hide elements (similar to changing \u201cdisplay\u201d or \u201cvisibility\u201d properties, but much more performant). Consider the case of a mobile menu toggle animation: in its open state, the menu will have an opacity of \u201c1\u201d. However, when it\u2019s closed, its opacity will be \u201c0\u201d. It\u2019s also best practice to define pointer-events as \u201cnone\u201d to ensure a user doesn\u2019t accidentally interact with the \u201chidden\u201d menu. The \u201cclosed\u201d class should be toggled as a user clicks the either \u201copen\u201d or \u201cclose\u201d buttons. Here\u2019s the respective code:\nView the code on Gist.\nAdditionally, opacity changes allow you to control the visibility level of an element. Again, this requires a bit of thinking outside the box, but can be quite powerful. For example, rather than animating an element\u2019s box shadow directly, you can instead alter the opacity of a pseudo-element containing the box shadow. Here\u2019s an illustrative code snippet:\nView the code on Gist.\nThis can instead be written as:\nView the code on Gist.\nForcing Promotion\nThere\u2019s even better news\u2014 you have control even beyond which properties you select. It\u2019s possible to manually promote elements to their own compositor layer. By forcing promotion, you can ensure an element is always painted and ready. This is an easy way to inform the browser which elements will","record_index":2},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 need a little more help \u2014 that is, anything that is paint expensive. This includes any element that will be changed (i.e., animated in some manner). Certain styles (such as: position: fixed and overflow: scroll) are quite expensive as well. You\u2019ve probably seen bugs come up in the past with elements that \u201cshimmy\u201d, \u201cflicker\u201d, or simply don\u2019t behave as expected. It\u2019s common to see fixed headers on mobile blink as a user tries to scroll down the page. Promoting these elements is an easy fix for these types of issues.\nThe Hacky Method\nPreviously, developers were forced to \u201ctrick\u201d the browser into promoting elements by applying styles that would trigger a new compositor layer but wouldn\u2019t necessarily have a visual effect. Generally, this was achieved with either backface-visibility: hidden or transform: translate3d(0,0,0).\nThe New, Shiny Method\nFortunately, browsers now provide an explicit property to inform browsers what types of optimizations will be needed ahead of time called will-change. You can provide varying values such as a list of properties (\u201ctransform, opacity\u201d), \u201ccontents\u201d, or \u201cscroll-position\u201d. Perhaps most useful is \u201cauto\u201d, which will apply the default, standard optimizations. A quick example:\nView the code on Gist.\nAs in all things, however, moderation is important. There is such a thing as too many composited layers. The browser already does its best to optimize, and will-change optimizations are resource heavy. Using will-change implies an element is always moments away from changing. This is especially important on mobile \u2014 many composited layers can have a noticeable negative performance impact.\nAnimation Methods\nThere are two ways to animate elements: using either CSS (declarative) or JavaScript (imperative). Which one you choose depends on how you need to best accomplish your goal.\nDeclarative Animations\nAs CSS animations are declarative (you tell the browser what to do), the browser knows the full extent of","record_index":3},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 the operation, and thus the end point. As a result, it can make optimizations. Additionally, CSS animations run off the main thread, and prevent blocking more potentially important work. Generally speaking, CSS animations will be the more performant option. Keyframes in combination with animations can provide many powerful options for interesting visual effects. Here\u2019s a small code snippet for rotating an element infinitely:\nView the code on Gist.\nThat said, CSS animations lack the expressive power of JS. One possible solution is to use JS to listen for user input and toggle a class based off the action. This class can then contain the necessary styles for the animation. Here\u2019s a quick example that shows how to toggle a \u201cclass-name\u201d on a box when it has been clicked.\nView the code on Gist.\nIt\u2019s also worth mentioning that if you\u2019re operating on the bleeding edge, the new Web Animations API allows you to harness the performance of CSS, within JS. Using the API, you can easily handle synchronization and timing of animations while still taking advantage of the perks of the CSS approach.\nImperative Animations\nImperative animations tell the browser how to perform the animation. In cases where CSS animations would grow too unwieldy or when more control is needed, JS should be used. It should be noted that, unlike CSS animations, JS animations will be run on the main thread (and as a result are more likely to drop frames), and are generally the less performant option. That said, there are a few options to consider when JS animations are needed.\nrequestAnimationFrame\nOne option to allow the browser to optimize for performance is requestAnimationFrame. You can consider it the setTimeout of the modern age; essentially, it\u2019s a native API for running an animation. It will theoretically be called at 60fps, however, in practice it requests animation drawings at the next available opportunity \u2014 there is no set interval. The browser optimizes by grouping changes into","record_index":4},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 a single repaint, which saves on CPU cycles. \nIt can be called recursively:\nView the code on Gist.\nAdditionally, you should consider leveraging requestAnimationFrame for performance-intensive events like resize or scroll (as opposed to binding directly to the event).\nScroll Performance\nSpeaking of scroll, achieving smooth scrolling performance presents its own challenge. Fortunately, a few recent additions to the specs provide further options for fine-tuning. \u00a0Passive event listeners enable you to improve scroll performance by indicating to the browser that preventDefault will never be needed (which eliminates the need for scrolling to block on touch and wheel event listeners). Usage is as easy as specifying {passive: true} on a listener.  });\nView the code on Gist.\nStarting in Chrome 56, this option will actually be default for touchmove and touchstart.\nAdditionally, the new Intersection Observer API allows you to easily determine when an element has entered or exited the viewport, or intersected with another element. Rather than clogging up the main thread with event handlers waiting on an explicit intersection, the Intersection Observer allows you to execute any work needed only when monitored elements cross paths. This is particularly useful for creating infinite scroll or lazy-loading experiences.\nRead, then Write\nThe correct term for the side effects of back and forth reading and writing to the DOM is \u201cforced synchronous layouts\u201d, and when done in quick succession it\u2019s known by the more illustrative term \u201clayout thrashing\u201d. As mentioned previously, the browser tracks \u201cdirty\u201d elements and queues up changes until necessary. By reading certain properties, you force the browser to perform premature calculations. This back and forth of read \/ write will cause reflows. Fortunately, this anti-pattern has an easy fix: read, and then write.\nTo illustrate this, here\u2019s a contrived example where the layout is read\/written to rather harshly:\nView the code","record_index":5},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 on Gist.\nInstead of reading and writing at every iteration, it\u2019s better to read outside the \u201cforEach\u201d method:\nView the code on Gist.\nThe Future of Optimization\nBrowsers are continuing to add more and more options for fine-tuning performance. One new property (currently supported in Chrome and Opera!) called contain allows you to indicate that an element\u2019s subtree is independent from the rest of the page. Essentially, this is an easy way to tell the browser that it\u2019s safe to optimize an element. You can specify as values strict (for all rules), content, size, layout, style, or paint to limit the scope of any changes. This will ensure that DOM updates in the subtree do not trigger reflows on the parent document. In particular, this can be useful for third party widgets that you lack control over. A quick example:\nView the code on Gist.\nTesting Performance\nWhile knowing how to optimize is all well and good, it\u2019s important to test your app\u2019s performance as well. The best tool (in my humble opinion) is by far Chrome developer tools. Hidden away under \u201cMore Tools\u201d, the \u201cRendering\u201d pane offers several options including tracking \u201cdirty\u201d elements, calculating frames per second, and highlighting layer borders and scroll performance issues.\nOptions available to you under the \u201cRendering\u201d pane.\nAdditionally, the \u201cTimeline\u201d tool (under the \u201cPerformance\u201d tab) allows you to run animations and then drill down into problem areas. The main gist here is: \u201cred\u201d is bad, \u201cgreen\u201d is good. You can actually click into red areas and determine which functions are most problematic.\nAnother interesting option (hidden under the \u201cCapture Settings\u201d section) is \u201cCPU throttling\u201d. Using this setting, you can apply a slowdown multiplier to measure the impact on performance. This is particularly useful for testing how an animation will run on mobile. While the FPS might be quite good on desktop, it\u2019s always important to consider other less powerful","record_index":6},{"post_id":6625,"post_title":"Performant Web Animations and Interactions: Achieving 60 FPS","post_date":1502237953,"post_date_formatted":"Aug 9th 2017","author_name":"Emily Hayman","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/be3c5d0149cc8eebfdb18615fb8438f8?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/performant-web-animations\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/performant-web-animation-1-1-360x200.png","time_to_read":14,"content":"\u2026 devices.\nScreenshot of a healthy timeline.\nTest and Iterate\nThe easiest way to optimize animation performance is to reduce the amount of work needed in each frame. The most efficient way to reduce this workload is to only need updates to elements on composite layers, as these changes will be non-destructive. That said, sometimes you\u2019ll have to get a little hacky and think outside the box. Performance fine-tuning is often a game of testing and iterating \u2014 but in the end, your users will thank you.","record_index":7},{"post_id":6605,"post_title":"Making Search Talk: Connecting Algolia and Alexa","post_date":1501739376,"post_date_formatted":"Aug 3rd 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/amazon-alexa-voice-search\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Algolia-Alexa-Adaptater_Illustration-360x200.png","time_to_read":5,"content":"Mostly, I just wanted to stop typing so much. \nA couple of years back, I bought an Amazon Echo. It was on a whim, really. I had been buying all of my food on Amazon Fresh, Amazon announced a special Prime-only smart speaker, and I decided that Jeff Bezos needed more of my money. The speaker was nice, but I was dying to extend it, because, again, I was tired of typing, and instead I wanted to use my voice to interact with the world.\nDon't get me wrong: I can type with the best of them. I was the best typer in my third grade class. But I really only enjoy typing when it's either a competition or in the service of something else, like chatting with my girlfriend or railing against the designated hitter in baseball. \nWhen I speak with other developers it's the same thing. Some might love refactoring code, others building new products, and still others writing clean documentation. But I haven't heard one who would say: \"I typed 5,000 characters today. What a good day!\" Mostly, through shortcuts, snippets, and the like, we want to type less.\n\nOver the years, I noticed a couple of things. The first was that the service provided by my new employer, Algolia, was a really good fit for what I often did with the skills. It was easier to work with than Amazon\u2019s DynamoDB (the general go-to data store for skills hosted on AWS Lambda), it was fast, and it could be leveraged inside AWS Lambda (where the majority of the Alexa skills reside) as well as it could be on the web.\n\nThe second thing I noticed was that I was typing the same things over and over. For Alexa, there would always be code that looked something like this:\n\nView the code on Gist.\nAnd with Algolia, I was always initializing the client, providing my credentials, etc. In defense of my colleagues who work on this, the amount of setup necessary is remarkably small when you consider what Algolia does. And, they've made it even easier with InstantSearch. \nTaking inspiration from InstantSearch.js, I wondered to myself if","record_index":0},{"post_id":6605,"post_title":"Making Search Talk: Connecting Algolia and Alexa","post_date":1501739376,"post_date_formatted":"Aug 3rd 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/amazon-alexa-voice-search\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Algolia-Alexa-Adaptater_Illustration-360x200.png","time_to_read":5,"content":"\u2026 there might be something similar that could connect Algolia with the Alexa Skills Kit.\nYou see, it\u2019s become clearer than ever that voice as a user interface has arrived. Voice-enabled assistants such as Alexa, Cortana, Google Assistant, Siri, and more are making an impact and changing the way we interact with our computers. \nAmazon is certainly sitting in the pole position. Through the Echo line and partner devices, Amazon is expected to control 70% of the voice-enabled speaker share in 2017. This is a market that will see 130% growth in 2017, with voice-enabled assistants growing 23.1% on their own.\nAll this means that millions of new customers are purchasing Alexa-powered devices and expecting Alexa skills that just work. They expect the skills to be fast and tolerant of ambiguity. Often they're looking for information, as we can see in popular skills such as those from Kayak or Boston Children's Hospital.\nThe developers creating Alexa skills want to be able to create them quickly and trust that they'll work.\u00a0This sounds like something to which Algolia is well suited.\nLet\u2019s take a look. Algolia\u2019s got typo tolerance (how often has Alexa heard \u201cNew York\u201d when you meant \u201cnew work?\u201d), synonyms (which frees you from having to use the skill builder for entity resolution), and search relevancy at the forefront of what we do. We\u2019re fast, which is important for \u201cquick thinking\u201d conversational UI. We also don\u2019t require information to come to us in a specific manner. And, Algolia can be set up in the time it takes you to watch the latest Game of Thrones.\n\nIntegrating Algolia with the Alexa Skills Kit was already simple, but how can we reduce the boilerplate--how can we reduce the typing? We decided to make it even easier by creating the Algolia Alexa Skills Kit Adapter. A small JavaScript library, the adapter is intended for use on Amazon Lambda and provides both tools for integrating Alexa and Algolia, as well as a framework for structuring your Alexa","record_index":1},{"post_id":6605,"post_title":"Making Search Talk: Connecting Algolia and Alexa","post_date":1501739376,"post_date_formatted":"Aug 3rd 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/amazon-alexa-voice-search\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/08\/Algolia-Alexa-Adaptater_Illustration-360x200.png","time_to_read":5,"content":"\u2026 skill.\nThe adapter is all convention over configuration--allowing you to stop thinking about the setup and start focusing on the user interface. While there might be more than 15,000 skills available for the Echo, it\u2019s still early days and an intuitive VUI will help raise your skill above the rest. That\u2019s what we wanted to optimize for.\nWe\u2019ve already leveraged it to great effect in building skills that use Algolia. To see an example, check out a skill we built that enables searching for name meanings and origins.\nVoice-first is a nascent but growing technology. There are many opportunities for companies to stake their claims early. Algolia\u2019s here to help. Get started with our Alexa Skills Kit adapter to build your own search into your skills.\n\n ","record_index":2},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"Performance is a core feature of Algolia. Our search engine delivers millisecond results across millions of records. It was originally designed for mobile devices where the resources are very limited, but was after transformed into an online API that now runs as an NGINX module. Our front-end libraries are written in a way that allows us to express that same speed on the client and ensure a high quality experience across devices.\u00a0Algolia\u2019s engine and integrations are a great benchmark and an inspiration for performance, so we wanted to mirror that excellence in performance of our website. \nOur website is the face of our product. It\u2019s the medium that\u2019s used to present ourselves and the product. It is the first impression that many people get of Algolia. And to embody that core feature of speed we wanted to make sure we go an extra mile on our website as well. \nReducing the payload\nNo bit is faster than one that is not sent; send fewer bits.\n\u2014 Ilya Grigorik, web performance engineer at Google; co-chair of W3C Webperf WG.\nAs Ilya says, the fastest way to have a fast website is to send as little data as possible. Our website was relying on plenty of external dependencies, which increased the total size that needed to be transferred and delayed the load time. A total of 180 requests and 2.4MB had to be transferred by our website visitors.\nTotal payload of 2.4MB and ~180 requests sent to load the website\nDrop JavaScript dependencies\nRemoving dependencies such as jQuery, Underscore and a few others reduced our total JS size to about 20% of it\u2019s initial size. This didn\u2019t have a big impact on the way our front-end code was written, but it forced us to write it using the native browser API such as the querySelector API vs. using jQuery. \nTo ease the process, I integrated transpiling of our code so that we could write ES6 code and transpile it using Babel. This made writing the code more productive and faster.\nRuntime performance\nTo avoid the jank behaviour once","record_index":0},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"\u2026 the website is loaded and ready for user interaction, it\u2019s very useful to profile JavaScript code to find bottlenecks. \nChrome Developer Tools has a very useful rendering tab inside its console drawer, which show areas on your website that are regularly being updated and might be the cause of jank.\nAnalyze page performance by using Chrome dev tools\nThis mostly meant rewriting our scroll event listeners using the new IntersectionObserver API, making sure our DOM operations are not too frequent and our animations only cause composite browser operations (see csstriggers for a list of operations needed to update a property).\n\nThis reduces the heavy lifting needed by the browser to paint and repaint the screen, which in return provides a silky smooth experience for your users.\nReduce CSS size\nBecause the front-end team didn\u2019t have a real convention of writing CSS, the size and specificity of it was growing fast as we added new pages. To tackle that, we wrote a small set of our own helper classes and adopted a way of writing the HTML using those classes. Doing it reduced the CSS file size to ~60% of it\u2019s initial size, and paved a good way of adding new pages to the site while not increasing the CSS size further.\n\nThe pesky time-consuming part was done; now it was time to make sure the users actually see the page as fast as possible. \nPrioritizing first paint\nTo prioritize first paint I needed to determine which assets are critical for our website to render. That meant asynchronously loading all of the render blocking assets except a few very small images like our logo and our main application.css file which weighs about ~50KB.\nThe goal of this is to show the website on the screen faster by loading the rest of the assets in the background. \nBelow is what the loading looked like before the critical assets were optimized.\n\nThe optimized experience:\n\nThis optimization results in a faster perceived performance experience whereas the total loading time stays about the","record_index":1},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"\u2026 same.\nAlong with having as few critical assets as possible, it is also a good optimization to have those assets hosted on your domain. Otherwise each of the requests made to different domains will have to go through the DNS lookup, connection and SSL negotiation phase, which will accumulate on the round trip time needed to perform the request.\nFor instance, if you are using Google fonts from their CDN and your server supports HTTP\/2 protocol, it\u2019s probably better to host the fonts yourself on the same domain as the initial request. This will bring significant improvements for the visitors coming from mobile networks, where the signal quality is poor and request round trip times are higher. \nIn our case, self hosting a font instead of loading it from google fonts CDN improved load time by about 1s on 3G connection.\nBefore (loading font from fonts.googleapi) vs. after (self hosting the same font)\nIf you look closely, you can also see that the fonts.googleapis request actually requests a CSS file which contains the @font-face rules that then create the actual request to load the font files. This means that, by including @font-face rules in our application.css file, we also save an additional request\u200a\u2014\u200aa double win. If you are looking to do a deep dive into font loading strategies, Zach Leat from FilamentGroup wrote a very helpful overview of the improvements you can do today.\nAdding WebP support\nWebP is a new type of image format which enables better lossless and lossy compressions. The support for it is growing, so I decided to test it.\n\n40KB WebP vs. 57KB JPG\nI ran a few compression tests and saw that it was able to compress the file size to about an average of 75% of the original, which saved us hundreds of Kilobytes.\nWhen looking to integrate WebP support into our build process, I found a simple way to do so using Cloudflare and their Polish option. I saw that they allow automatic WebP image compression through their Polish feature, which took complexity of","record_index":2},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"\u2026 integrating WebP out of scope; enabling it was as simple as clicking a button.\nAfter the Polish option and WebP compression are enabled, Cloudflare does the heavy lifting. It checks if the image request contains accept header with values image\/webp or *\/*, as seen below. If the header matches, it converts the original image into WebP format and adds a content-disposition header with the value of inline; filename=\u201dpath\/to\/image.webp\u201d\u00a0instructing the browser that the file will be displayed inline on the page and giving it the file path to the resource.\n \n\nAccept header with webp support\u200a\u2014\u200aimg\/webp and *\/*\n\nResponse header with content-disposition\nIn our case, Cloudflare\u2019s solution worked well which meant I didn\u2019t have to update server configuration and integrate WebP conversion at build time. However, if that is not the case for you and you want more flexibility, Ilya Grigorik wrote a sample config for detecting WebP support, and there are multiple libraries that you can use to convert images to WebP format.\n\nUsing HTTP\/2 server push\nOne of the great things with HTTP\/2 is that it has features like multiplexing connections and server push, which are substantial performance improvements to HTTP\/1.1.\nMultiplexing connections allow browsers to send multiple requests through a single connection, which significantly reduces the number of required connections between the client and the server. \nServer push is a feature that allows the server to start sending assets that the client has not yet requested, but knows that the client will need, and so it eliminates the extra time the client would otherwise take to parse the response and request the assets. \nYou can implement server push either by adding custom HTTP headers, or by adding the link rel=\u201dpreload\u201d and as=\u201d<type>\u201d to the asset source in your HTML, in which case you will need to polyfill the behaviour.\nTo additionally improve the time to first paint, I decided to avoid polyfilling link","record_index":3},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"\u2026 rel=\u201dpreload\u201d and set Link headers for our remaining render-blocking assets. This resulted in faster load time of assets and improved time to first paint by about ~400ms (depending on connection quality).\nTo validate the assets were server-pushed, check the developer tools network tab, where you can see that the request was not initiated by the browser after parsing the document, but was rather pushed by the request for index.html.\n\nInitiator\u200a\u2014\u200aPush\/(index) indicates asset was server pushed\nIf you are looking for a good hosting solution with advanced features like HTTP\/2 server push, have a look at Netlify\u200a\u2014\u200athey just added server push support and their hosting is very solid.\nThe hidden bottleneck\nAs I was optimizing our website, I looked for the obvious quick wins, but there is one thing I didn\u2019t really look at\u200a\u2014\u200athe HTML document size.\nThe compressed size of our index.html was 60KB.\nThe reason for that were inline SVG assets. Inlining SVG is often advised in the web community because of its flexibility. There are plenty of articles that advocate for it, but they are often flawed in that they recommend it as a universal solution, whereas it should depend on the use case. There are often better ways to load inline SVG assets than inlining them straight into the document. \nInlining SVG files bear two major consequences:\n\ndocument size increases\nassets are not cached\n\nIf you are accessing a website where the index.html file size alone is ~60KB, it will take time to fetch the document itself and after it\u2019s finished, you still need the rest of the critical request to render the page. \nBy combining SVGs into a store, asynchronously loading and injecting them into the document, I was able to reduce the size of our HTML file from 60KB to ~15KB + as an added benefit, we were now caching those\u200a\u2014\u200aa double win again.\nMeasuring results and changes\n\nThroughout the development I used two main tools to measure the performance impact of our","record_index":4},{"post_id":6569,"post_title":"Improving Web Performance to Mirror Engine Speed","post_date":1501052314,"post_date_formatted":"Jul 26th 2017","author_name":"Jonas Badalic","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/474f66479fc9f6e259367362ea3fc234?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/improving-web-performance-to-mirror-engine-speed\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/improving-performance-360x200.png","time_to_read":13,"content":"\u2026 work\u200a\u2014\u200aChrome Lighthouse and webpagetest. The first tool\u200a\u2014\u200aLighthouse\u2014 can either be accessed through Chrome developer tools under the audit tab, as a CLI tool or as a Chrome extension. It provides valuable information and front-end metrics, whereas webpagetest can be used to go deeper into the network audit itself. \nThe results\nWe have seen a big improvement in loading performance: our website now loads much faster even on poor connections, ensuring that our visitors get a fast experience both when browsing our content and using our engine.\nThe total size of the website is now ~700KB compared to the original 2.4MB, with about ~300KB of external dependencies that we decided to keep for now. The amount of total requests is now in the 70s range compared to ~180. \nIn addition, our team was able to improve runtime performance and add more logic and animations to the website without having a negative impact on page performance.\nTo sum up\nThese improvements have helped stay on track of providing a unified and fast experience to all of our users and visitors (our documentation and our community page have also been updated with performance in mind). \u00a0\nI have had the chance to do a presentation of the topic to my Algolia co-workers, raising performance awareness within the company. A few weeks after, I did the same talk at a PWA Paris Meetup that we hosted in our Paris office. For those interested, the video is available on YouTube. \nLast but not the least, I\u2019d love to hear your comments and suggestions on the topic: @JonasBadalic. Thanks for reading \ud83d\ude42","record_index":5},{"post_id":6555,"post_title":"Computing Statistics on Terabytes of Data on Multiple Machines","post_date":1500966060,"post_date_formatted":"Jul 25th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/computing-statistics-terabytes-data-multiple-machines\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Computing-stats-data_Illustration-1-360x200.png","time_to_read":5,"content":"At one point, every company needs to compute some basic statistics on its data. I\u2019m not speaking about advanced statistics but simple ones, like means, top values, etc. The algorithms to compute those metrics are fairly straightforward, but can take a lot of memory. So what happens when the amount of data is so big that it can\u2019t fit in a reasonable amount of RAM? \nToday at Algolia we generate 2 TB of logs per day, so computing metrics on this doesn\u2019t fit into any of our machines. Furthermore, the data team would like to have those metrics in pseudo real time, so we need to process the logs in real time.\nHow to compute metrics on terabytes of data\nFor some of our data, it was OK not to have an exact value but an approximation. For example, instead of having an exact average of 3.9, we get 4.0, and that\u2019s ok. Thankfully, some algorithms and data structures are able to lower their precision to have a lower memory footprint.\u00a0Those data structures are what we call probabilistic data structures (or algorithms), and they all share some properties:\n\n\nHave a small memory footprint \u2014 we are talking KB of memory instead of TB.\nUsable in a single pass: each piece of data is processed once by the algorithm.\nNo statistical hypothesis on the data being processed.\nHave a precision within an acceptable error margin. Most of the time the margin is a parameter of the algorithm.\n\nWe\u2019ll see later how they work.\nHow to compute metrics on a stream of events\nAs we have logs arriving continuously, the team thought that we could leverage this and compute metrics on the fly. There are algorithms designed to process data on the fly, and they are called streaming algorithms. Sure, the name is fancy, but we code them every day. Let\u2019s take an example. \nLet\u2019s say we have an array of integers, and we would like to compute the sum of each element. In pseudo code we would probably do something like:\nView the code on Gist.\nNow imagine that our array is infinite (what we call a","record_index":0},{"post_id":6555,"post_title":"Computing Statistics on Terabytes of Data on Multiple Machines","post_date":1500966060,"post_date_formatted":"Jul 25th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/computing-statistics-terabytes-data-multiple-machines\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Computing-stats-data_Illustration-1-360x200.png","time_to_read":5,"content":"\u2026 stream). Our code won\u2019t change, and at any point in time, the variable sum would contain the sum of all elements we saw. This is a streaming algorithm: an algorithm that can process an infinite number of elements and that is able to give a coherent state after each iteration.\nBonus point: most probabilistic data structures are streaming algorithms.\nHow to compute metrics across multiple machines\nWe are able to compute simple metrics with a small amount of RAM and from a stream of data. But what if your stream is so big that one machine can\u2019t handle it? As we found a way to reduce the memory footprint, couldn\u2019t we find another trick for CPU? \nOne simple option would be to split the workload across multiple CPUs. For this, we will use a mathematical property. Yes, math. Bear with me a few seconds, you won\u2019t be disappointed.\nLet\u2019s say you have a set of elements. On this set you have an operation, let\u2019s call it +, that takes 2 elements of this set and gives back an element of the same set. We say that this set and this operation is a monoid if:\n\nFor any a, b, c that are elements of this set, this holds: (a + b) + c = a + (b + c)\nThere exists an element e of this set where: e + a = a + e = a\n\nLet\u2019s take some examples:\n\nIntegers and addition, where e is zero\nStrings and concatenation, where e is the empty string\nLists and concatenation, where e is the empty list\nBooleans and &&, where e is True\n\nWhy did I bother you with this?\nLet\u2019s take a simple example, with integers and addition. If you want to sum 1, 2, 3 & 4, you can ask Google or you can spread this sum on multiple CPUs, let\u2019s say 2. Because this is a monoid you know you can do sub additions, for example: 1+2+3+4 = (1+2) + (3+4). So, you can ask the first CPU to compute 1+2, the second one 3+4, and only then sum those sub sums. And voil\u00e0, we have our final sum, which is 10.\nTherefore, if some set and operation validates 2 properties, we have a sound way to spread the computation on","record_index":1},{"post_id":6555,"post_title":"Computing Statistics on Terabytes of Data on Multiple Machines","post_date":1500966060,"post_date_formatted":"Jul 25th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/computing-statistics-terabytes-data-multiple-machines\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Computing-stats-data_Illustration-1-360x200.png","time_to_read":5,"content":"\u2026 multiple CPUs or machines.\nBonus point: most probabilistic data structures and streaming algorithms are monoids.\nConclusion\nAlgorithms like the ones above are not new \u2014 far from it. The first probabilistic data structure was invented in 1985 by Philippe Flajolet (hence the pf* commands in redis). Monoids are even older. \nHappily for us, a lot of these algorithms are already implemented in big data software (think Spark, Apache Beam, Algebird, etc.). \nThe main thing we should remember is that some simple mathematical properties give us many nice coding features. Perhaps now you feel sad if you slept through your math classes :).","record_index":2},{"post_id":6511,"post_title":"Introducing InstantSearch iOS: Create Great Search UX with Swift and Objective-C","post_date":1500519559,"post_date_formatted":"Jul 20th 2017","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-great-search-on-ios\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/IOS-InstantSearch_Illustration-360x200.png","time_to_read":9,"content":"We\u2019re excited today to be releasing InstantSearch iOS, a library of views and helpers for building a great search experience on iOS. It is built on top of Algolia's Swift API Client to give iOS developers a way to quickly build a search experience that will delight their users.\nBefore we dive into how InstantSearch iOS works, let's talk about why we need this library in the first place.\nImplementing search on iOS today: hard and complex\nMobile screen sizes are small by nature, which makes it hard for end users to see and filter their results. It is therefore expected that developers invest a lot of their time into building a search experience. Let\u2019s explore possible ways to implement search on iOS today, as well as their costs and benefits.\nLocal search\nThe most basic search experience that you can provide is by creating a local search on the data that you have on your device. This data can reside in one of the following places:\n\nIn memory, such as in an array or a dictionary.\nOn disk such as using CoreData, SQLite or Realm.\n\nIt is fairly straightforward to implement a basic string matching algorithm. Just follow a tutorial such as the Raywenderlich one, and you\u2019ll get something working in an hour or so. They even add a way to filter by one refinement by scoping with a scope bar. \nBenefits:\n\nNo 3rd party dependencies, you\u2019re in complete control\nEasy to implement and maintain\nFree\n\nCosts: \n\nLow quality. This is a basic search implementation; no advanced features such as typo tolerance, custom rankings and synonyms.\n\nImplement search on your backend\nAnother popular way of implementing search on mobile is to provide a search functionality from a backend. If you do it this way, the mobile app would request search results by simply making a network request to your backend and adding the query and the filters in the request.\nThis is the most complex way to implement your search since you have to take care of every part of your search architecture: the search","record_index":0},{"post_id":6511,"post_title":"Introducing InstantSearch iOS: Create Great Search UX with Swift and Objective-C","post_date":1500519559,"post_date_formatted":"Jul 20th 2017","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-great-search-on-ios\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/IOS-InstantSearch_Illustration-360x200.png","time_to_read":9,"content":"\u2026 business logic on the backend, the communication between the backend + iOS, and finally the presentation and update of your search UI experience. \nBenefits:\n\nNo 3rd party dependencies, you\u2019re in complete control\nReuses the search business logic across more platforms such as Android and web\n\nCosts: \n\nComplex implementation for getting relevant, typo-tolerant search results with custom rankings\nNeed to host it and maintain it on your server\n\nSearch-as-a-service \nThe last way to implement a search experience is to rely on a 3rd party search-as-a-service to provide you search results and refinement capabilities. That way, the mobile app will just have to call an API over the network to get all the search results. This means speed superior to other solutions\u2019, plus the search backend is taken care for you in this case. \u00a0\n\nBenefits:\n\nPremium search features that work out of the box: speed, relevance, typo-tolerance. No need for search expertise on your part.\nReuse the search business logic across more platforms such as Android and web.\nSearch backend taken care for you, no need to worry about hosting and availability. \n\nCosts: \n\nReliance on 3rd party company\nCosts some money to use SaaS\n\nThe common pitfall \u00a0\nAll of the three methods above will help you build the search business logic of your app, but they have something missing: you still need to build the search UI on the iOS app. If you\u2019ve tried to build a complex UI search experience, you quickly found out that it can be tough to get it right: from dealing with all the states of the search lifecycle to updating all components as results come or filters change, it can be quite a challenge to implement a search UI correctly.\nIntroducing InstantSearch iOS \n\nInstantSearch iOS offers a way to easily build search UI experiences on iOS by using Algolia as the backend search-as-a-service. It lets the developer focus on the look and feel of their app, without having to worry about Algolia API calls, or query crafting","record_index":1},{"post_id":6511,"post_title":"Introducing InstantSearch iOS: Create Great Search UX with Swift and Objective-C","post_date":1500519559,"post_date_formatted":"Jul 20th 2017","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-great-search-on-ios\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/IOS-InstantSearch_Illustration-360x200.png","time_to_read":9,"content":"\u2026 and state management. \nSearch is its own domain of expertise\nSearch is very complex if you want it to be done in a correct way. A good search has to provide speed, relevance and typo tolerance, which is a lot to ask, especially to mobile developers whose primarily skill is building apps, not writing algorithms for fetching the most relevant search results. \nTranslate the concepts of search into UI\nThinking in terms of terms such as disjunctive faceting offers a lot of cognitive complexity to the developer. InstantSearch makes it easy since developers will only have to think in terms of \u201cand\u201d, \u201cor\u201d, \u201c<\u201d, \u201c==\u201d instead of difficult concepts such as the difference between conjunctive and disjunctive faceting. \nHeavy lifting of the implementation\nDealing with the search state can be quite cumbersome. For example, how would you handle the state when requests come out of order? Also, you have to handle all parts of the search lifecycle, be it the initial default state, the no result state, or the new result\/error state. Finally, you have to deal with all the data binding and keep all UI components in sync with the actual search state. InstantSearch iOS has been built to take care of all of that for you.\nWidgets offered with UX best practices\nInstantSearch iOS provides you with numerous UI components called widgets, which are \u201csearch aware\u201d components built on top of UIKit objects such as UITableView, UICollectionView and UISearchBar. These widgets follow search best practices on small mobile screens that will help the developer build great UX.\nPan-platform UX\nThe nice thing about InstantSearch iOS is that it has many siblings such as InstantSearch Android, InstantSearch.js, and more. What this means for you is that you get to use one search engine with myriad possibilities, while having a uniform and consistent UX across all platforms.\nPlug-in architecture: an extensible library\nAs you use the library more and more, you might stumble upon a case","record_index":2},{"post_id":6511,"post_title":"Introducing InstantSearch iOS: Create Great Search UX with Swift and Objective-C","post_date":1500519559,"post_date_formatted":"Jul 20th 2017","author_name":"Guy Daher","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/048f2a8c3c8a8561a20a6329196603b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-great-search-on-ios\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/IOS-InstantSearch_Illustration-360x200.png","time_to_read":9,"content":"\u2026 where none of the provided widgets fit your needs. InstantSearch iOS has been built following Plug-in Architectures standards, which means that the library is modular, customizable and easily extensible. \nThe most important offering is the creation of custom widgets by implementing a few protocols on your UIView class. These protocols will provide you the relevant information that the widget needs, such as new search results coming by, or notifications of refinement changes happening in other widgets.\nGetting started \nThe Getting Started Guide will take you through the main steps to use the library. The library is open source and can be found in this repo. The example application is a great addition to the guide as it\u2019ll  help you get an idea of what you can build with InstantSearch iOS. \nIf you use a Backend-as-a-Service to power your mobile apps, there are guides on how to sync your data between them and Algolia, such as the Firebase guide and the Graphcool guide.\nTo share your thoughts or ask for help, you can reach out on Algolia Community or Stack Overflow. Please share your projects, too! We look forward to seeing what you\u2019ll build with InstantSearch iOS.","record_index":3},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"Building great search UX is one of the hardest problems in engineering, even for companies that have a large team working exclusively on search. \nThere are three essential hurdles to clear:\n\n1) engine performance\n2) interface intuitiveness\n3) result relevance\nWe\u2019ve addressed search performance in several of the previous articles of this series; this article will focus on the link between user interface and relevance. It will cover a wide range of use cases \u2014 from common to very specific. If users complain that they cannot find what they are searching for quickly, it is probably caused by a mix of non-intuitive interface and poor relevance; this is why it is important always to consider them together.\nProviding good relevance for the most common cases\nWhen people are searching for something very general like a company name on LinkedIn or a person on Twitter, you should make sure you are using a notion of popularity on the results: a result with more followers has more chance to be what the users are searching for. Solving this problem has been our primary focus at Algolia from day one. We have addressed this by developing a configurable but easy to understand ranking algorithm where developers have full control of their search configuration. You can configure the popularity via Custom Ranking, make sure you can address a complex problem like typosquatting, promote some featured items, etc.\nThere is an endless way to configure the ranking, depending on the use case. We could even help Twitter solve their search issues :), as you can see in the following example:\n\nSearching for the official Barack Obama account on Twitter with a typo does not return the official account because of typosquatting (May 2017)\nProviding good relevance for advanced search use cases\nSolving advanced search use cases is an entirely different problem where you need to let the user refine search results via filters. We will cover in this section the three main approaches that are used to","record_index":0},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"\u2026 facilitate an advanced search.\nUse of faceting\nFaceting is probably the most standard way to address advanced use cases. The principle is to let the user filter the results based on categories to reduce the size of the results set and remove all false positives or ambiguity. This feature is used in a wide variety of use cases, and there are a lot of different ways to present faceting results, as you can see in our faceting documentation.\nFaceting example: filter on genres and rating\nThat said, faceting is not the perfect solution in all situations as two significant concerns can hurt your relevance:\n\n1) It requires homogenous data: you need to have the same categories inside all your records to have a good UX. If this is not the case, you might have duplicates in your faceting results that will lead to users not finding their record while filtering.\n\n2) The category that the user wants is hidden: when the number of categories to display is large, you won't be able to show all of them, and you might miss the one that the user wants. It will be a challenge to satisfy this user, who will probably just consider your search not good enough.\n\nSearch in faceting results\nTo fix the second problem\u2014faceting not visibly proposing the category that the user is looking for\u2014you can offer users a search for facet values. This means that the full range of categories does not even have to be displayed. Rather, you can offer the most relevant or meaningful categories, and let the user search for the rest. You can find this type of experience on LinkedIn: you can filter on locations when you are looking for a person or a company. LinkedIn proposes you to search for a particular location that is not listed in the selected facet values, which allows being very granular in your search. \nThe only problem that you might have on Linkedin is that this search is not contextual. You are searching inside all locations but without applying the current textual query and filters. Let\u2019s say","record_index":1},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"\u2026 that you are looking for a Director of Marketing job in Buenos Aires but that there aren\u2019t any available on LinkedIn. After you\u2019ve searched for the job title, LinkedIn allows you to refine your search and select \u201cGreater Buenos Aires\u201d in the location facet value search, although this will not return any results. This is frustrating as you\u2019d naturally expect not to be offered filters that will lead you nowhere. \n\n\nImplementing a search in faceting results while applying the contextual query is very complex, and this is probably the reason why LinkedIn preferred to implement a degraded version. A better search experience is our primary motivation at Algolia, which is why, a few months ago, we released a feature called search for facet values that allows you to develop this type of experience in minutes while applying the contextual query. Here is an example of an Algolia search in the brand value in an e-commerce context.\nSearch inside facet values: example of the brand facet\nAs you can see, the search is completely contextual: all filters\/refinements are related and inclusive of one another. You can find more documentation about how to use it in our developer documentation. \nAdvanced query syntax\nSearch for facet values is very useful when you are looking to expose an advanced search to your user without burdening them with a learning curve, which is mandatory for a consumer product. If you are working on a business product that people use every day, you might want to expose them an advanced syntax inside the search box like Slack does.\n\nExample of advanced syntax in a search box (Slack)\nIn practice, proposing such an experience requires the same feature as searching for facet values. It is just a different way to expose the same feature. The goal of this display is to let your advanced users directly perform their advanced query via the keyword and minimize the number of steps they will need to search for their content. You can implement such interface","record_index":2},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"\u2026 easily with our search for facet values feature. We will release a guide soon to help you through the implementation.\nThe implementation\nWe described in the previous sections how the search for facet values feature is useful for implementing an advanced search interface. We will now focus on the implementation to discover how this feature works internally.\nI am sure you will think this type of search simply requires a change in the way we perform the query, like rewriting the query to specify which attribute we will target. But in practice, this is a significant change as we do not return records but rather highlighted facets with counts. \nLet's take a simple example of a professional social network like LinkedIn to illustrate the implementation. We will take simple records containing only four attributes. Those attributes will contain a string and will all be configured as attributes for faceting:\n\nThe name of the person\nTheir title\nThe company where they work\nThe location of the company\n\nHere is an example of such a record:\nView the code on Gist.\nAt first sight, this seems like a simple problem to solve for any search engine. Let\u2019s look at what you need:If a user performs the query \u201dTwilio\", they will retrieve hundreds of Twilio employees. Let's say the user wants to sell Twilio a service in San Francisco and wants to see all the Director titles to find the closest one to their service (this is, of course, a purely fictional use case :)). \n\n1) The profiles need to work for Twilio, so we can restrict the search on \u201cTwilio\u201d to the \u201cCompany\u201d field.\n2) They need to be a Director, so you can restrict the search on \u201cDirector\u201d to the \u201cTitle\u201d field.\n3) If multiple people have the same title, you want to see the title only once, so you need to deduplicate the results. To achieve this, you can use a facet on the attribute \u201cTitle\u201d, and display the facet values returned.\n\n\nIn theory, this search should provide the results that you want, but in","record_index":3},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"\u2026 practice, it\u2019s more complex than that:\n1. People often have multiple job titles in their profiles. And if someone has both \u201cDirector of Sales\u201d and \u201cAccount Executive\u201d listed as a job title, this query strategy will mention both job titles in the list (because the query relies on faceting). Do you really want to see \u201cAccount executive\u201d in the list of results for the query \u201cTwilio Director\u201d?\n2. Since we\u2019re displaying facet values, and not search results, we cannot highlight the words that match the query. It\u2019s always better to have highlighting, particularly when people type something with typos (Diretcor), or if you search at each keystroke (Direc).\n3. It creates a complex UI\/UX, because users will need to specify that Twilio should only be searched in Companies, instead of allowing them to search Twilio in all fields (which would also potentially improve the relevance, on top of reducing the complexity).\n\nThere is a better solution: search directly in facet values while providing the context of the query.\u00a0Let\u2019s see how we can do this by using the \u201csearch for facet values\u201d feature on the facet \u201cTitle\u201d:\n\nView the code on Gist.\nThis query will be applied in two steps:\nFirst, it will retrieve only the results containing the word Twilio in one of the searchable attributes. From this list, it will extract all of the values for the facet Title (i.e., all of the jobs titles listed in every profile that contain the word Twilio).\nThen, in this filtered list of job titles, it\u2019ll search for the ones that contain \u201cDirector\u201d. This allows us to only retrieve relevant results, by using all of the regular search features (typo-tolerance, highlighting, prefix-search, ranking\u2026). The result is the list of all the Twilio job titles containing \u201cDirector\u201d, deduplicated and ordered by count \u2014 exactly what we were looking for.\nIn other words, this feature requires a two-step process that is only doable efficiently when implemented at the heart","record_index":4},{"post_id":6494,"post_title":"Inside the Engine Part 8: Handling Advanced Search Use Cases","post_date":1500363443,"post_date_formatted":"Jul 18th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-8-handling-advanced-search-use-cases\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/Inside_advanced_search_Illustration-360x200.png","time_to_read":11,"content":"\u2026 of the engine. \nEvolutions\nThe release of this feature enabled all our existing users to build a great advanced search interface quickly. We, of course, do not plan to stop here and are already thinking about the next evolutions of the feature. For the moment, the ranking of results is pretty basic and is only using the frequency of facets, which means you can have a result with typo first. We plan to improve the feature by providing a different way to rank the results.\nIf you want to know more about the internal aspect of the Algolia engine, we recommend reading the other posts in this series:\n\nPart 1\u2014Indexing vs. Search\nPart 2\u2014The Indexing Challenge of Instant Search\nPart 3\u2014Query Processing\nPart 4\u2014Textual Relevance\nPart 5\u2014Highlighting, a Cornerstone of Search UX\nPart 6\u2014Handling Synonyms the Right Way\nPart 7\u2014Better Relevance via Dedup at Query Time","record_index":5},{"post_id":6448,"post_title":"Connectors, Docs & the Future \u2014 a Deeper Look into InstantSearch.js v2","post_date":1499224961,"post_date_formatted":"Jul 5th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-js-v2\/","categories":["Guides","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/InstantSearch_js_v2_Illustration-360x200.png","time_to_read":8,"content":"Back in November 2015, we released InstantSearch.js v1 in order to give you an efficient way to build search UIs. We have received good feedback so far: 1600+ GitHub stars, 600+ upvotes on Product Hunt, and 1700+ live implementations and counting.\nOver the last 18 months, we have learned a lot with your feedback, our experience creating other libraries, and with our internal usage of those same libraries. \nWhat we\u2019ve found is that:\n\nWhen it comes to widget customization there are never enough options.\nThe learning curve has one steep jump in it: when you have exhausted the API options, you have to create custom widgets which requires search knowledge.\nThere are some limitations in the current implementation that we want to overcome so that we can confidently continue to improve InstantSearch.js.\n\nWe\u2019ve done our best to integrate what we\u2019ve learned into InstantSearch.js v2, which we are releasing today. Here are some of the changes we\u2019ve implemented.\nCustomization API for widgets\nBefore getting down to nitty gritty details of this new API, let\u2019s have a look at a practical use case.\nBlog layout as of June 2017\n \nOn our blog we use a menu widget to let the user navigate into the categories. The behavior of this widget works perfectly here because of its unique properties:\n\nOnly one element can be selected at a time\nIt displays all the values with the counts\nIt also displays a special all item which deselects the current category\n\nThis behavior is exactly the same as another well known UI element: the drop-down menu.\n\nSome of you rightfully asked: \"Can we render your menu widget as a drop-down element?\"\nThe problem is that the HTML of both UI menu elements is completely different and can\u2019t be expressed as template options. The two widgets\u2019 HTML implementations are very different:\n HTML rendering of the menu widget and a drop-down\nReflecting on that, we would have to either:\n\nAdd a \"renderAsSelect\" option to the InstantSearch.js menu widget\nProvide a","record_index":0},{"post_id":6448,"post_title":"Connectors, Docs & the Future \u2014 a Deeper Look into InstantSearch.js v2","post_date":1499224961,"post_date_formatted":"Jul 5th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-js-v2\/","categories":["Guides","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/InstantSearch_js_v2_Illustration-360x200.png","time_to_read":8,"content":"\u2026 menuSelect widget (we even considered a PR for that)\n\nBut adding more options does not make an API any good. This problem is actually deeper and can be phrased as: \n\"How can I precisely control the rendering of InstantSearch.js widgets?\"\nAs we discussed during our release of react-instantsearch earlier this year, widgets are made up of two parts:\n\nSearch business logic, which deals with the Algolia engine\nRendering logic, which renders the UI and handles the user interactions\n\nWith the connectors API, we abstracted the search business logic into its own API, making it easier than ever for you to precisely control the rendering of any widget. The default IntantSearch.js widgets are implementations of their respective connectors. \nExample usage:\nView the code on Gist.\nThe connector handles the business logic and exposes a simplified API to the rendering function (items, refine, widget parameters). In this case, it receives a function to change the currently selected value and the list of options to display. This makes the core logic of a widget easily reusable.\nIn v2, all the built-in widgets with a UI rendering now have their own connector. Because this is a significant change in the way the library is used, we needed to rethink the documentation: \n\nRevamping the documentation\nIn v1, the documentation layout was a single page with multiple columns. Visually it looked great but when the API grew, some issues started popping up:\n\nThe content felt constrained and was hard to scroll through\nIt was hard to find space for new topics\n\nAfter InstantSearch.js v1, we iterated a lot on our other documentation websites: places, JS Helper, React InstantSearch. With this knowledge and experience, we found that the best way of organizing documentation for libraries like ours is having multiple pages. For InstantSearch.js v2, we are introducing getting started and guides, and we are reorganizing the API reference into independent sections.\n\nThe getting started tutorial is the part","record_index":1},{"post_id":6448,"post_title":"Connectors, Docs & the Future \u2014 a Deeper Look into InstantSearch.js v2","post_date":1499224961,"post_date_formatted":"Jul 5th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-js-v2\/","categories":["Guides","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/07\/InstantSearch_js_v2_Illustration-360x200.png","time_to_read":8,"content":"\u2026 that helps our new users most. Making it a prominent part of the navigation easily orients them. This is where we want to provide a good overview and real learning value.\nGuides are also very important because we found that you can only do so much with an API. Guides provide a framework to users whose use cases may be too niche or advanced to use an obvious solution. \u00a0\nFinally, we kept and split up the API reference. Like we\u2019ve done since InstantSearch.js v1, this part is completely built on top of the code documentation (jsdoc). We even go to the point of using it to create pseudocode examples to explain the API.\nTo learn more about what we think about documentation, read the blog post that we wrote about the user\u2019s journey learning a library.\nBuilding for the future\nFor the last part of this release, we also wanted to make sure that we are going to be able to continue to improve the library during this major version\u2019s lifecycle. For this, we had to introduce a few breaking changes. All of them are documented in the migration guide.\nOther notable changes:\n\nInstantSearch.js now ships a simple and effective theme for all widgets ?\nThe slider is now based on AirBnB's rheostat slider ?\nThe build is now 33% smaller and uses preact internally ?\nThe searchFunction (an advanced feature) has been improved. It now has all the methods ?\n\nFinal word\nWith this version, we want to:\n\nGive you more power to customize widgets\nSimplify the project to new users\nMake the core of InstantSearch.js future proof\n\nInstantSearch.js v2 is available and you can try it now:\n\nRead the documentation\nMigrate a v1 application\nCheck the NPM package on the Yarn website\u00a0\n\nAs always with Algolia, your feedback is key to provide you the best tools. Encountering a bug? Open a GitHub issue,\u00a0Want to showcase your awesome InstantSearch.js website? Post it to our forum and get expert feedback from our community.\n\n ","record_index":2},{"post_id":6431,"post_title":"Harry Logger and the Metrics\u2019 Stone","post_date":1498631614,"post_date_formatted":"Jun 28th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harry-logger\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/Harry_Logger_Illustration-360x200.png","time_to_read":5,"content":"Introduction\nAs Algolia grows, we need to reconsider existing legacy systems we have in place and make them more reliable. One of those systems was our metrics pipeline. Each time a user calls the Algolia API, whether the operation involves search or indexing, it generates multiple lines of logs.\nWe generate around 1 billion lines of logs per day, which represent 2TB of raw data\nThose logs contain information about the query that, when harvested, can yield insightful results. From those we compute metrics. Those are all the numbers you can find in the Algolia dashboard, like average search time, which user agents are using an index, etc. It\u2019s also used for the billing as it\u2019s computing the number of objects and the number of operations our customers perform. \nAs we are big fans of Harry Potter, we nicknamed this project \u201cHarry Logger\u201d.\nThe Chamber of Logs\nThe first thing we had to do was to transfer, in a resilient way, using as few resources as possible, the logs for the API machines to our metrics platform. The old system was doing a \u00a0fine job, but worked by having a centralized system pulling the logs from each machine. We wanted to go to a push strategy using a producer\/consumer pattern. \nThis shift enabled us to do 2 things:\n\nReplicate the consumers on multiple machines\nPut a retry strategy closer to the logs, in the producer\n\nWe needed something that works reliably and in a clean way, hence we asked Dobby to do the job. For performance reasons, Dobby was developed in Golang:\n\nThe prisoner of SaaS\nOur second job was to compute metrics on top of those logs. Our old system was a monolithic application that ran on one machine, meaning it was a single point of failure (SPOF). We wanted the new version to be more reliable, maintainable and distributed. \nAs SaaS is in our DNA, we went to various companies that specialized in the processing of metrics based on events (a line of log in our case). All the solutions we encountered were top notch, but as a","record_index":0},{"post_id":6431,"post_title":"Harry Logger and the Metrics\u2019 Stone","post_date":1498631614,"post_date_formatted":"Jun 28th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harry-logger\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/Harry_Logger_Illustration-360x200.png","time_to_read":5,"content":"\u2026 company, the quantity of the data we generate on a daily basis presented an issue. As of today, we generate around 1 billion lines of logs per day, which represent 2TB of raw data. And no vendor was able to handle it. At this point, we were back to square one.\nThe Streams of Fire\nAfter much consideration, we concluded that we had to build our own system. As the logs are a stream of lines, we decided to design our new system to compute metrics on top of a stream of data (and it\u2019s trendy to do stream processing). \nWe tried the typical architecture:\n\nAs we didn\u2019t want to maintain and host this architecture (we have enough API servers to maintain), we decided to consider a cloud provider. We managed to find every tool on the shelf, which meant we\u2019d have less software to operate and maintain. As always, the issue was price. This streaming platform was 100 times more expensive than our old system. We had a lot of back-and-forth with our cloud provider to try to reduce it, but unfortunately, it was by design in their system. And again, we went back to square one.\nThe Half-Blood Batches\nDuring our tests, we found that the Stream processing software we were using was also able to work in batch mode, not trendy but maybe it was a way to fix our pricing issue? By only switching to batch mode, the price was reduced by a factor of 10! We only needed an orchestrator to launch the batches:\n\nAfter some development of reliable orchestrator we had a fully working system, but it was still 50% more expensive than we envisioned.\nThe Order of DIY\nOne of our engineers, a bit fed up by the amount of time we took to optimize the system, decided to do a proof of concept without using any framework or PaaS software. After a few days of coding, he managed to develop a prototype that suited our needs, was reliable and had a running cost 10 times lower than the batches!\n\nThe tale of the Wealthy Bard\nMigrating our log processing toolchain yielded many outcomes that were valuable to our","record_index":1},{"post_id":6431,"post_title":"Harry Logger and the Metrics\u2019 Stone","post_date":1498631614,"post_date_formatted":"Jun 28th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harry-logger\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/Harry_Logger_Illustration-360x200.png","time_to_read":5,"content":"\u2026 team. In addition to improving the reliability and the evolutivity of our current toolchain, we also increased our internal knowledge regarding our PaaS provider. The process also helped us identify deployment pain points that we will address later this year. \nFinally, we iterated by using and composing different solutions to solve the same problem. The best solution, for us, was the one closest to the no-compromise with respect to your budget limits. In our case, it was possible to have both by finally keeping the multi-master key\/value storage of our PaaS provider.","record_index":2},{"post_id":6415,"post_title":"Testing for Failure in a 99.999% Reliability World","post_date":1497345941,"post_date_formatted":"Jun 13th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/testing-failure-99-999-reliability-world\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/testing-for-failure-360x200.png","time_to_read":7,"content":"Even in a perfect world where everyone is doing test-driven development (TDD), even when everything is well planned and as a result, those plans succeed - Even in this world, things will fail. Bugs happen. Inevitably, there is always a little thingy that was forgotten. That\u2019s this little thingy that this post is about. \nIn our years of building Algolia, we've had our share of learning experiences when it comes to building reliable infrastructure, especially when it comes to hardware and networks. As we strive to have a\u00a0five 9s SLA, we need to have as many failovers as possible. Part of this failover is done in our API clients, where we implement automatic retries in case of TCP or DNS failures.\nTCP is complex and can fail. The TCP stack of most programming languages is sound, but not all HTTP clients know how to handle failure correctly. That is why we need to thoroughly test how our API clients are behaving in case of network failures.\nIf something fails or is not fast enough, we want to make sure that another method is used. So, the most important factors, in our view, are timeouts. We need to be sure they are handled correctly by the HTTP clients we are using. \nWith our knowledge of TCP, we knew that the timeouts we wanted to enforce were: \n\nConnection timeout: the time to make the initial connection, i.e. the time to initiate the TCP handshake\nRead timeout: the time to wait to read data, i.e. the delay between 2 bytes sent by the server\n\nFor DNS, it\u2019s a bit more complex. Most of the time, it\u2019s not a connected protocol and uses UDP. We saw that it is handled very differently in each programming language, so we needed to make sure our API Clients were behaving in the same way whatever their programming language. Hence, we wanted to enforce only one timeout: the time to resolve a hostname.\nSimulating network errors\nWith in mind what we wanted to test, how could we simulate network errors easily and in a way that is language agnostic?\nFirst, connection","record_index":0},{"post_id":6415,"post_title":"Testing for Failure in a 99.999% Reliability World","post_date":1497345941,"post_date_formatted":"Jun 13th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/testing-failure-99-999-reliability-world\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/testing-for-failure-360x200.png","time_to_read":7,"content":"\u2026 timeout. This one is quite easy, as you only need a host that resolves to an IP that doesn\u2019t answer. Some ranges of IPv4 are reserved, so you only need one host that resolves into \u00a0the private network range, we chose randomly 10.255.255.1.\nSecond, read timeout. For this one, we need a host that resolves to an IP that accepts connections, but never answer when we ask for data.\nThird, the DNS timeout, which is a bit more tricky than the first two tests. To test for this condition, we need a host where its DNS resolution times out. So we created a new domain where the DNS resolution is handled by a server that timeouts. Ring a bell? It\u2019s the same as the connection timeouts. The resolver of our domain is the same IP that the one for the connection timeout: 10.255.255.1.\nWith all of this we could test timeouts in every language possible.\nSimulating user input\nWe are operating a public facing API, so anyone can send us a request. And a small part of those are invalid:\n\nInvalid JSON\nBad UTF-8 characters\nSQL injection, remote code, or attacks of the same kind\n\nFor this, there was already a lot of resources on Internet, so we used them.\nFor JSON, we use YAJL, so we were also pretty confident in the handling of JSON. For various reasons, we tried developing our own JSON parser, so we wanted to make sure it was handling bad and good JSON correctly. We stumbled upon this article and this test suite. We used it to test our JSON parser & YAJL. Funny thing, we discovered that YAJL accepts line feed (\\f) as a valid whitespace character, where the JSON standard doesn\u2019t.\nUTF-8 is a complex encoding format, and it\u2019s quite easy to generate a sequence of bytes that result in a bad UTF-8 character. For this, we aggregated multiple source of bad sequence so we could use them.\nLast, but not least, we evaluated naughty strings. It\u2019s strings that could be a security issue\/flaw: https:\/\/github.com\/minimaxir\/big-list-of-naughty-strings. \nSo with little effort, we manage to add","record_index":1},{"post_id":6415,"post_title":"Testing for Failure in a 99.999% Reliability World","post_date":1497345941,"post_date_formatted":"Jun 13th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/testing-failure-99-999-reliability-world\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/testing-for-failure-360x200.png","time_to_read":7,"content":"\u2026 quite a few tests to ensure that we handle corner cases correctly.\nDiscovering Failure\nFor the previous failures, it was something we knew beforehand. So it was quite easy to know what to test and how to simulate errors. But what happens when the unexpected happens?\nLet\u2019s take an example. We have an internal application that reads logs that are in the following format: \u201ckey1=value1;key2=value2\u201d. This format is quite straightforward. It\u2019s a key\/value separated by semicolons. So, there isn\u2019t a ton of code needed to parse it. But this application is business critical and should handle incorrect logs in a proper way, aka not to crash. \nTo ensure it doesn't crash, we can add some basic unit tests as well on some corner cases we thought, but there was probably a lot more that we didn\u2019t think about.\nOne way to do this is to use property testing. It\u2019s a way to test code where you let the computer generates the testing data. It comes from functional languages, where it works pretty well as all functions are pure and could be described by its inputs and outputs. \nProperty testing works when you describe properties on your code, you describe how to generate the data, and then you let the property test framework generates the data and it checks if those data validates the properties. \nLet\u2019s take a full example with our log parsing application. One property could be \u201cI should not throw an exception if I receive an invalid log\u201d. \nSo what is a log?\n\nIt\u2019s a string\nIt\u2019s a sequence of strings separated by semicolon\nIt\u2019s a sequence of \u201ckey=value\u201d separated by semicolon\nWe could then generate the key\/values we expect, and so on\n\nThen we run the testing framework, and it will test our application with the data that is constrained by what is a log. With this we managed to found some corner cases in our parsing of logs. One field was expecting a IP address but we didn\u2019t check it was in the correct format, for example.\nIn Summary\nTesting for failures is not","record_index":2},{"post_id":6415,"post_title":"Testing for Failure in a 99.999% Reliability World","post_date":1497345941,"post_date_formatted":"Jun 13th 2017","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/testing-failure-99-999-reliability-world\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/testing-for-failure-360x200.png","time_to_read":7,"content":"\u2026 a lot of work if you know what are the areas to look for. As long as you know it, you can find some good documentation of corner cases. For all the rest, with little effort, you can code property tests that will test your software in a new, and unexpected way.","record_index":3},{"post_id":6402,"post_title":"Redefining Incredible; Redefining Search","post_date":1496906438,"post_date_formatted":"Jun 8th 2017","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/redefining-incredible-search\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/series-b-announcement-illustration-360x200.png","time_to_read":4,"content":"Today we're proud to announce that Accel has led a $53 Million Series B investment in Algolia - you can read more about it here.\nWe closed our Series B exactly two years after our Series A. Back then, we knew there would be many challenges to face but we believed we could create something incredible. Since then we've continued to learn, to evolve, and as a result we've become a stronger company. We've grown from 20 to 115 team members, our customer base has increased from 650 to 3,000, and we have 8,000 active implementations, across which we\u2019re powering 25 billion searches each month. We have customers in 100+ countries and we have offices in four cities (San Francisco, Paris, New York City, Atlanta).\nWith the team, I often say that a funding round is just a means to an end. In the end we will not be judged by how much we raised but by how much we delivered. Today is a time for us to take a new look at our future, and define what \"incredible\" will mean to us next.\nIf software is eating the world, SaaS APIs are eating the development world. \nSaaS API companies like Algolia are designed from the ground up to innovate. We hold our roots not only in cloud hosting, but in APIs which sought to optimize the speed with which products communicate with users; that meant APIs to post to social networks, to deliver texts, emails & phone calls. \nNow we\u2019re in a new era: thanks to pioneers like Amazon, Twilio and Stripe, developers are now able to create\u00a0the very building blocks of an application itself - authentication, payments, and, yes, even search -- all through APIs. \nFor companies like Algolia, APIs aren\u2019t a secondary tool for extending the\u00a0core product\u2019s reach, such as Uber\u2019s API, for example, which allows Uber to enable other applications to connect into their ride-hailing functionality. APIs are our lifeblood. They have to be reliable - their SLA payback for anything above %.01 downtime should align with that commitment - and importantly they have to","record_index":0},{"post_id":6402,"post_title":"Redefining Incredible; Redefining Search","post_date":1496906438,"post_date_formatted":"Jun 8th 2017","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/redefining-incredible-search\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/series-b-announcement-illustration-360x200.png","time_to_read":4,"content":"\u2026 continuously innovate. \nThe benefits of SaaS APIs are already visible to the developer community: Stripe\u2019s release of Radar in the past year showed how, without making any changes to one's\u00a0code base, developers could benefit from Stripe's payments innovations. \nDay after day, APIs must get better for their users. What\u2019s great today is merely good tomorrow and eventually will not be enough. We don\u2019t just want to push the limits of what\u2019s possible with search, we\u2019re designed from the ground up to do so. \nAll our resources are geared toward improving our API and making the life of developers easier, empowering them to deliver the most engaging experience for their users. Our core engineers handle support & documentation, as well as internally training the whole company on new features before they are released. Our marketing team focuses on educating the greater developer community on how search works and how Algolia works for them.\nBack to the future: what does \u201cincredible\u201d look like tomorrow?\nSearch must become an effortless conversation between user and product. Today\u2019s search trends are rooted in conversation, and ultimately Algolia\u2019s mission is to connect intent with content in the most frictionless way possible. Search and conversation are ultimately rooted in the same three principles - speed, relevance and user experience. \nWe have invested in building the essential building blocks for creating advanced search experiences, often very nuanced, like search for facet values, a highly valuable feature of search for ecommerce sites and social networks, which end-users ultimately will never notice (unless it\u2019s missing when they need it). As we forge ahead, these building blocks will form the foundation for seamless conversation between user and software. \nWe will continue to make investments in leveraging the advantages of being a hosted API to deliver innovations in search, not only making it possible to have consumer-grade search previously","record_index":1},{"post_id":6402,"post_title":"Redefining Incredible; Redefining Search","post_date":1496906438,"post_date_formatted":"Jun 8th 2017","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/redefining-incredible-search\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/series-b-announcement-illustration-360x200.png","time_to_read":4,"content":"\u2026 reserved only for the titans of tech - Microsoft, Facebook, Amazon, Apple, and Google - but also by bringing search innovations to the public that will redefine search itself. \u00a0\nWe will also continue to improve the developer experience, striving to simultaneously lower the barriers to entry for any developer or product team to achieve consumer-grade search, and to raise the bar for what\u2019s possible with search by deploying new innovations across every API client.\nYour success is our success, and vice versa.\nThe best way to build a community is to be one, and Algolia owes a lot to a lot of different people. To our customers, we owe our ability to grow as a company. To the developer community, without whom Algolia would never have been able to advance and improve so quickly. To consumers, whose expectations for excellence have laid down the foundation for Algolia to succeed. \nWe're on a mission to connect intent with content, so let's get to work!","record_index":2},{"post_id":6367,"post_title":"Building for developers\u2014a tour of new features and resources in 2017","post_date":1496830284,"post_date_formatted":"Jun 7th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-building-for-developers-2017\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/15994409_935842486546596_8761301610453709536_o-360x200.png","time_to_read":10,"content":"New\u00a0API features, libraries, dashboard upgrades and community tools have been rolling out this year at a steady pace.\u00a0In case you missed anything, I'm here to give you a\u00a0panoramic look at what Algolia has made available to\u00a0developers since the year began. Milliseconds matter, so let's go!\nNew API features\nSearch for facet values\nIn March, we announced\u00a0search for facet values. Faceting is an essential element of search but sometimes the number of facet values can be hundreds or thousands. A list of checkboxes would be way\u00a0too long to handle that, but a search box is just the right size.\n\nSearch for facet values requires no additional index configuration by the\u00a0developer, just add the\u00a0InstantSearch widget to your\u00a0UI. To learn more\u00a0check out this video with Solutions Engineer Alexandre Collin, recorded at our\u00a0React-themed Search Party\u00a0in January.\nAlgolia Vault\nDevelopers who work in sensitive fields like government and healthcare have additional\u00a0compliance requirements. Algolia Vault\u00a0is a way to address\u00a0these needs in a convenient but highly secure way. Send us an email\u00a0to schedule a demo.\nLibraries\nInstantSearch React\nSince the launch of InstantSearch React\u00a0last December, we have continued to act fast on the\u00a0feedback coming from the early\u00a0adopters. The latest version, 4.0.0, lets you\u00a0query multiple indices\u00a0simultaneously and also connect an external autocomplete. We also created\u00a0a new video\u00a0that shows how to build a fully working search with React in less than 15 minutes.\nWe are very\u00a0proud of our InstantSearch React\u00a0engineer Marie-Laure Thuret, who was selected to give a talk about testing React UI components with React Storybook at ReactConf 2017.\n Algolia engineer Marie-Laure Thuret speaking at React Conf 2017\nInstantSearch Android\nInstantSearch Android is the\u00a0newest member of the Instant Search family. Read the release announcement to\u00a0learn about the challenges of building search on\u00a0Android and how this library helps you address\u00a0them.","record_index":0},{"post_id":6367,"post_title":"Building for developers\u2014a tour of new features and resources in 2017","post_date":1496830284,"post_date_formatted":"Jun 7th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-building-for-developers-2017\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/15994409_935842486546596_8761301610453709536_o-360x200.png","time_to_read":10,"content":"\u2026 If you\u2019re an Android developer we\u2019d love for you to try it out and\u00a0share your feedback.\nInstantSearch v2\nThe original, framework-agnostic\u00a0JavaScript InstantSearch has just kicked off\u00a0a beta for v2. This represents a significant\u00a0upgrade, adding among other things a connector-style API similar to React InstantSearch, which\u00a0makes working with components much\u00a0more flexible.\nAlgolia Offline\nIn February we announced the release of Algolia Offline,\u00a0a solution for\u00a0providing a mobile\u00a0search experience even where there is little or no\u00a0network connectivity. This is available today for\u00a0both Android and iOS. We\u2019re very happy to count great applications like AllTrails in the community of early Algolia\u00a0offline users - read here about how AllTrails uses\u00a0Algolia offline.\nUnifying Algolia iOS\nWe now support both Swift and Objective-C API clients with the same codebase, written entirely\u00a0in Swift. Read more about the journey to supporting two languages done by our engineer Cl\u00e9ment Le Provost, who deserves a big hand for all of his contributions on mobile.\nDocumentation\nIf code is the heart of the developer experience, then\u00a0documentation is the soul. We understand how critical accurate, complete\u00a0documentation is to our developers. In the last year, we have done one major reorganization on algolia.com\/docs\u00a0and we\u00a0are taking all of your feedback into account as we prepare for another big update soon.\nPlease continue to share your feedback by clicking on the icon in the top right-hand corner of each documentation panel.\n\nDashboard\nThe Algolia dashboard is the go-to destination for configuring relevance, locating API keys, inviting team members and much more.\nMore intuitive\u00a0ranking formula interface\nConfiguring and fine-tuning relevancy is essential to shipping a successful\u00a0search. We\u2019re progressively trying to make it easier\u00a0to understand and debug without taking away\u00a0flexibility. In the last month, we released an enhanced Ranking Formula component which","record_index":1},{"post_id":6367,"post_title":"Building for developers\u2014a tour of new features and resources in 2017","post_date":1496830284,"post_date_formatted":"Jun 7th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-building-for-developers-2017\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/15994409_935842486546596_8761301610453709536_o-360x200.png","time_to_read":10,"content":"\u2026 gives alerts about\u00a0relevancy issues\u00a0and suggests\u00a0optimizations.\n\nFaster\u00a0record explorer\nWe released\u00a0a faster, cleaner record explorer built with\u00a0our own library, InstantSearch React. The record explorer lets you explore the data you've uploaded inside of the Algolia dashboard, and includes the ability to facet and\u00a0narrow down the search by\u00a0all\u00a0filter types.\n\nCommunity\nCommunity forum\nOfficially released in December, the Algolia community forum has grown to over 600 users and now\u00a0averages over\u00a040 new posts per day. The forum is the official place for our developers to ask questions, get help and exchange useful information\u00a0about search.\nThe forum is also the home of the Show & Tell category, where we invite you to share projects that you've built with Algolia. We always love to see what you build. Upon request, we\u2019re happy to provide\u00a0feedback\u00a0or help you promote your project.\nCode of Conduct\nThe Algolia Community Code of Conduct\u00a0is live and applies to all of the Algolia community. We are committed to providing a welcoming, harassment-free space at all of our events and online spaces.\u00a0We want to extend our sincere gratitude to Keen IO, LadyNerds and LGBTQ in Technology\u00a0- our code of conduct draws heavily on their efforts.\nNewly-designed open source\u00a0site\nOur site for helping you discover\u00a0open source Algolia libraries, community.algolia.com, has been updated to match our new design and it now has a\u00a0search! This is just the first in a series of improvements to come. We\u2019ve heard your feedback that you\u2019d like a clearer path to submitting projects to be included - expect a better way to do that soon.\nSearch Party\nSearch Party is a meetup series for exploring the possibilities of what search can be used for and used with. So far, we've had a Search Party in four\u00a0locations around the world, and next up is\u00a0June 22 in Paris.\nFor each Search Party we also invite speakers\u00a0to talk about broader technical topics that our developers are also","record_index":2},{"post_id":6367,"post_title":"Building for developers\u2014a tour of new features and resources in 2017","post_date":1496830284,"post_date_formatted":"Jun 7th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-building-for-developers-2017\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/15994409_935842486546596_8761301610453709536_o-360x200.png","time_to_read":10,"content":"\u2026 interested in. In January, Preact creator Jason Miller joined us in San Francisco to speak about performance and\u00a0the story of Preact, a 3KB drop-in React replacement.\n\nJason Miller, creator of Preact, presents at Search Party\nYarn package search\nJust before the holidays, we continued our tradition of making a yearly contribution to the community by releasing\u00a0an Algolia-powered package search for\u00a0Yarn. Then, in March, our intern (and now employee) Haroen Viaene\u00a0made it easer to find\u00a0by putting a search bar at the top\u00a0of every page. But that wasn\u2019t all - at the end of April we added new package detail pages that show popularity, contributors and file trees from unpkg.com. Here's an example of the detail page for React.\n\nDocSearch\nDocSearch is now used by 250+\u00a0open source documentation sites and API portals. We\u2019re thrilled to have\u00a0Stripe in\u00a0the DocSearch family\u00a0and to help developers get even more out of\u00a0Stripe's amazing\u00a0docs. We're also very excited to power search for\u00a0webpack,\u00a0a team doing great work that we are excited to also support\u00a0with an\u00a0Open Collective sponsorship.\nSwag - a new take\nLast month at Twilio Signal we\u00a0tried something new with swag. Instead of giving out bulky physical items, we donated $10 to Women Who Code for every badge we scanned. All in all, that represents over $3,000, an investment we\u2019re thrilled to make in an organization that connects over 100,000 women working in technical careers globally.\u00a0Anyone who donated received\u00a0a special occasion sticker to show off on their shirt or conference badge.\n\nDevelopers serving developers\nAlgolia only exists because of our customers, many of whom are are developers. The way we organize the company is with this in mind.\nWe now have a dedicated Developer Experience squad as part of our engineering organization. This squad is responsible for documentation, guides, tutorials and building tools and integrations that support our developers and customers.\nWe also\u00a0have a Developer","record_index":3},{"post_id":6367,"post_title":"Building for developers\u2014a tour of new features and resources in 2017","post_date":1496830284,"post_date_formatted":"Jun 7th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-building-for-developers-2017\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/06\/15994409_935842486546596_8761301610453709536_o-360x200.png","time_to_read":10,"content":"\u2026 Advocacy team that now has\u00a0three full-time developer advocates in San Francisco, Portland and Paris. We expect to add several more advocates to the team this year - yes, we\u2019re hiring! \nBeyond these teams, the majority of our engineers actively contribute to the community. This has been\u00a0a tradition since the beginning of Algolia, which we make sure continues today with our culture-defined content strategy and our Speaker Program, which provides a 2-day public speaking workshop\u00a0and CFP assistance to anyone in the company who wants to get up on stage.\nThank you!\nLast but never least, I want to say thank you to all of the developers who are choosing Algolia to build their search. We know that search is one of the most critical features of your application and we will continue to work hard to fulfill your expectations - to be open and transparent about our technology, our roadmap and our goals as a company. Let\u2019s continue to build great things together!","record_index":4},{"post_id":6344,"post_title":"Supercharging WordPress so that everyone can have great search!","post_date":1495086452,"post_date_formatted":"May 18th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supercharging-wordpress-for-great-search\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/wordpress-update-360x200.png","time_to_read":3,"content":"Last year, we released our WordPress plugin, allowing WordPress users to super power their search bars with an instant-search autocomplete dropdown menu.\nOur focus since the beginning has been to offer non-technical users an easy way to enhance the search experience for their end users. We also wanted to keep as much flexibility for WordPress developers to be able to hook into the plugin to customize the behavior.\nWhat we\u2019ve learned\nWe already knew that making a plugin simple to use is a real challenge. This is even truer when trying to interface with a search engine such as Algolia which in fine has a lot of tunable options.\nTo make it as easy to\u00a0use as possible, we ship our integration with default settings for the different search indices, and handle the indexing and synchronizing of content for the users behind the scenes.\nOne thing we learned is that by addressing both developers (by making the plugin extensible) and non-technical users, is that:\n\nTechnical users always want to go a step further than the API you expose\nNon-technical users always want the same than technical users but they want to be able to achieve it via a graphical user interface\n\nHere is what we did to lower the barrier\nTo give more power to our users, we decided to put more focus on our plugin configuration interface.\nIn the latest release, ordering of post types can be done by drag & drop. Header labels can be overwritten using a search input without writing a single line of code and the Algolia powered by logo can also be disabled by ticking-off a checkbox.\nA complete changelog or the latest changes can be found on the plugin\u2019s repository.\nAddressing the technical issues\nBy listening to our users, we realized that most of the reported technical issues were related to our \u201csmart\u201d way of queueing indexing tasks.\nIndeed, until this week\u2019s release, we were making an extensive usage of custom post types and using remote HTTP calls to process the queued indexing tasks.\nThis","record_index":0},{"post_id":6344,"post_title":"Supercharging WordPress so that everyone can have great search!","post_date":1495086452,"post_date_formatted":"May 18th 2017","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supercharging-wordpress-for-great-search\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/wordpress-update-360x200.png","time_to_read":3,"content":"\u2026 mechanism, even if in theory looked like to be an ideal solution, appeared to be the main bottleneck of our search plugin.\nThanks to the whole WordPress community, we managed to take out the complexity of\u00a0the plugin and we now offer a way simpler, consistent and pragmatic solution to index content.\nAs a result, indexing is faster, more scalable and easier to debug and reason about.\nAlso, one big request we had, was to let users tune the relevancy directly from the Algolia dashboard. Until now, this was not recommended because on every new re-indexing process you would loose your settings and|or synonyms.\nWe took this release as an opportunity to meet users expectations, and ranking and synonyms can now be customized directly from the Algolia dashboard without fear of being lost again.\nUpgrade to the latest version of the plugin\nWe refactored about 3k lines of code for the v2 release, yet we tried to keep as much backward compatibility as possible.\nGiven we simplified the way we split records and store them, you should re-index all indices once the plugin has been updated.\nIf you previously customized your frontend search templates, you might need to compare your implementation with the new ones to adapt them.\nHow to contribute\nThe community has played a major role in the evolution of the plugin, and we would like to say how grateful we are.\nFor anyone who would like to participate, we now have a community forum where you can share your project or website to give it some visibility.\nFor developers, pull requests are very welcome.\nNo matter if you are a WordPress user or a developer, all your feedback and contributions are highly appreciated.\nYou can check out the latest version of the Algolia plugin for WordPress on the official plugin\u00a0directory.","record_index":1},{"post_id":6290,"post_title":"Mobile Search UX \u2013 Part Two: Deconstructing Mobile Search","post_date":1494983140,"post_date_formatted":"May 17th 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-two-deconstructing-mobile-search\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/deconstructing_1440-360x200.png","time_to_read":8,"content":"In part 1 I made the case that search UX on mobile is a pretty complex topic, from limited screen real estate, to mobility and pesky connectivity - there are a lot of traps that need to be avoided. In this post, I\u2019ll demonstrate exactly how we can avoid these traps by taking an analytical approach to UX. We\u2019ll deconstruct all the moving parts that comprise Mobile Search UX, and further down the road we\u2019ll see how we can act and be impactful on mobile.\nShow Me That Text Input!\nStep one is simple but crucial: you\u2019ll need to decide where in your app your users will get introduced to search. There are, for the most part, three options:\u00a0Full search bar, Tab bar, Icon.\n\n\nFull Search Bar\nThe full search bar approach typically appears on the main screen, giving instant access to the search feature. If your product or app revolves around giving your users quick and easy access to a lot of content, then it\u2019s the way to go. Search becomes the center of interaction to your content that would typically populate the page below in an \u201cas you type\u201d fashion.\n\n\nTab Bar\nWhen search is not the main access point of your content but you still want to give users a easy access to search throughout the app then a tab bar icon is the way to go. Spotify favors suggestions of albums, playlists or songs on the home screen, and when I want a specific tune I have quick access to the search screen with the tab bar icon no matter where I am in the app.\n\n\nIcon\nThe contextual search icon is among the least intrusive options. It\u2019s easily the most flexible approach - you can display the button only in determined parts of the app that make sense for searching. For Gmail, the main use case is browsing unread mails in the inbox. Then, by order of importance in the UI, creating a new mail would come second with the large icon on the bottom right corner. Finally the search icon in the navigation bar on the top right corner enables users to search through \u00a0all my emails.\n\n\nSearch","record_index":0},{"post_id":6290,"post_title":"Mobile Search UX \u2013 Part Two: Deconstructing Mobile Search","post_date":1494983140,"post_date_formatted":"May 17th 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-two-deconstructing-mobile-search\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/deconstructing_1440-360x200.png","time_to_read":8,"content":"\u2026 Screen\n\nOnce you\u2019ve determined where search starts for the user, the next UI element to analyze is the search screen. This element is mainly a \u00a0transition between the main \u2018non-search\u2019 view and the result page that we\u2019ll describe right after. The search screen is displayed as soon as the user taps the text input search field or the search icon. This action already shows an intent -\u201dI\u2019m looking for something-\u201d and that should be rewarded with useful information. As we saw earlier, each tap is painful, so we must gratify each effort. Instead of a default empty table-view waiting for the user\u2019s query to fetch and display content, let\u2019s guide them!\nHere are a few options:\n\nRecent searches enables fast back and forth between new searches and previous ones. When a user searches for a product, it\u2019s handy to give a quick glance at previous searches for comparison.\nTrending searches guides the user by showing what other users are most looking for in your app. This helps an uninspired customer to discover your content easily by clicking a suggestion.\nCategories guide the user with a number of filter options to start with giving a scope of themes that are searchable, refining directly to a scope of interest and speed up the search process.\n\n\n\nResult Screen & Result Display Strategy\nThe final component of Search UX on mobile is the result screen. Here, there are two prevailing strategies: query suggestions & instant results. They both serve a specific purpose and I\u2019ll explain where & when they are the most relevant. \nBefore diving in, it\u2019s important to discuss an overarching concept to search results: the as-you-type search experience. Today\u2019s users expect search to be instant, not only with respect to how fast results are fetched, but how frequently they are updated. Instead of waiting for the user to hit enter to to type a certain number of characters before displaying results, Algolia encourages product builders to display search","record_index":1},{"post_id":6290,"post_title":"Mobile Search UX \u2013 Part Two: Deconstructing Mobile Search","post_date":1494983140,"post_date_formatted":"May 17th 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-two-deconstructing-mobile-search\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/deconstructing_1440-360x200.png","time_to_read":8,"content":"\u2026 results from the first key stroke. Users today expect instant feedback from their inputs, hence \u201cas-you-type\u201d.\n\nQuery Suggestions\n\nTyping a query won\u2019t immediately show results on the screen but instead a series of more complete queries based on the input. Let\u2019s say I\u2019ve typed \u201ciphone\u201d in the search box, the suggestions displayed will be more complete queries that are known on the product side to be popular or relevant for that given query. I\u2019ll end up with a list of suggestions such as: \u201ciphone case\u201d, \u201ciphone cable\u201d etc\u2026\nGiven that it is an extra step between the user and results, the perfect use case for query suggestions is searching through a huge data set - Amazon is a great example. The user might feel overwhelmed by the volume of possible results and not be sure where to search, what to search, what is available. Query suggestions are an effective way to guide the user and help him quickly access what they\u2019re looking for in a large volume of possibilities by suggesting the complete query of what they had in mind. The added benefit is a much faster time to actual relevant content as the user gets relevant suggestions after only a few characters. \n\nThose query suggestions are most often based on popular queries, an analytics solution behind the search will keep track of what query resulted in a purchase, and with a high volume of uses an index of potential queries can be built.\n\nInstant Results\nAnother option is to fetch actual results with each keystroke and display them on the result page. This approach is most effective when the data set is either more limited or the user might have a more defined idea of what they\u2019re looking for. In the Spotify example we clearly understand, because of the product\u2019s nature, that search will be limited to the realm of albums, songs, playlists and artists. When looking for \u201cbeatles\u201d having suggestions of albums containing \u201cbeatles\u201d could be adding an unnecessary step between my query","record_index":2},{"post_id":6290,"post_title":"Mobile Search UX \u2013 Part Two: Deconstructing Mobile Search","post_date":1494983140,"post_date_formatted":"May 17th 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-two-deconstructing-mobile-search\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/deconstructing_1440-360x200.png","time_to_read":8,"content":"\u2026 and the actual result (e.g. The Beatles). When the scope of search is well defined and well-known by your users, providing instant search results is the shortest path between them and your content. One approach, as in our example, is to show a limited set of hits per type of content (5 artists, 5 bands etc\u2026). The most relevant results will appear in the 5 firsts allowing the user to have a quick overview of matches on different types, and you can add a \u2018show more\u2019 button to allow the user to dive in by category.\nAn alternative of displaying limited results per type would be to have a segmented input right below the search box, enabling you to show more results of a specific type at once.\nFor this instant result strategy to provide good user experience, very fast response time is required, to allow results to be refreshed \u201cinstantly\u201d at each keystroke.\n\n\nSuggestions + Instant Results Combination\n\nThe last option to consider is a combination of both strategies. When you have a very large dataset and want to guide your user but are also able to determine if a good match has occurred then mixing both can work. On the google play store we see an actual app result above multiple query suggestions. That\u2019s a great strategy here, as there are millions of apps but it is also possible to be confident the query \u201cairbnb\u201d matches very well with the app airbnb, so displaying the app here has a great chance of being the result the user is looking for. \n\nThat wraps up part 2 of our mobile search UX series. \u00a0We\u2019ve covered the most important components in mobile search today - From text input placement, to search screen and results. The decision for how you guide your user through the mobile search experience depends on a lot of factors - the size & complexity of your data set, the knowledge that you expect your user to have about the data set they are searching through, and your desire to balance discovery and intent in the search experience. With a good","record_index":3},{"post_id":6290,"post_title":"Mobile Search UX \u2013 Part Two: Deconstructing Mobile Search","post_date":1494983140,"post_date_formatted":"May 17th 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-part-two-deconstructing-mobile-search\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/05\/deconstructing_1440-360x200.png","time_to_read":8,"content":"\u2026 overview of the structuring elements specific to search on mobile, we\u2019re ready to go even deeper to provide best practices based on those components.\nBig thanks to Lucas Cerdan:\u00a0this series\u00a0relies a lot on the\u00a0research into\u00a0search and mobile UX that he has done at Algolia.","record_index":4},{"post_id":6127,"post_title":"Mobile Search UX - Part One: 8 Obstacles, 1 Pocket-Sized Nightmare","post_date":1492948808,"post_date_formatted":"Apr 23rd 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-8-obstacles\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Ux-mobile-nightmare_1440-360x200.png","time_to_read":6,"content":"Editor\u2019s note: this post is first in a three-part series on creating amazing mobile search experiences. Check back in the coming weeks for part 2 where we\u2019ll deconstruct mobile search success elements and part 3, which will explore mobile search best practices.\nWhatever your app development stage \u2014 a brand new idea, iterating on a v2 or finally tackling that long-neglected feature \u2014 mobile search can be the decisive element of your app\u2019s success. When executed correctly, search gives your users incredible access to all of your content. Done wrong, it will quickly cripple your app, making it a digital ball of stress and frustration. Before diving into how you can design successful mobile search, let\u2019s analyze some of the pitfalls of mobile user experience (UX) that can make implementing search a real nightmare.\n1- Tiny Screen, Fat Fingers\nMost difficulties developing for the mobile platform come from the extreme limitation of space to display information on the user's screen. A common rookie developer mistake is wanting to display all the data at hand, similar to our desktop experiences. With mobile, more so than anywhere else, rule #1 is less is more.\n2- Every Single Tap Hurts (the User and Your App)\nWhen the time comes to transfer information from your brain to a digital device the most common input method today is the keyboard. You can usually count on having your ten chubby fingers to do this... unless you are using a mobile device. Then you have at most two thumbs available. On mobile, information output is dramatically reduced.\n3- Mobile Users Are, Well, Mobile\nWe use our mobile phones in less than optimal conditions \u2014 on a shaky train, on our bike at a red light, while walking (when we should probably pay attention to the intersection ahead). Those environments are not conducive to good focus and can lead to harder information input\/processing on the user\u2019s end.\n4- Network, or the Likely Absence of It\nWe take our phones everywhere and we want","record_index":0},{"post_id":6127,"post_title":"Mobile Search UX - Part One: 8 Obstacles, 1 Pocket-Sized Nightmare","post_date":1492948808,"post_date_formatted":"Apr 23rd 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-8-obstacles\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Ux-mobile-nightmare_1440-360x200.png","time_to_read":6,"content":"\u2026 them to work as well on the top of a mountain as from our bed. Coverage won\u2019t be perfect, even in the busiest cities. Poor connectivity is extremely frustrating. As mobile developers, we must anticipate\u00a0for the user being offline and plan accordingly.\n5- Information Overload\nLess is more... That\u2019s great in theory but not that simple for our search use case. You\u2019ll need more than a simple text input to create a powerful search experience. At a bare minimum, you\u2019ll also need to display results. How many results you display and how much information you provide about each result is the difference between information overload and a successful search experience. Limiting the volume of displayed results on page means an even higher constraint on relevance \u2014 you can\u2019t allow a single poor hit if you only show 5 total hits. The amount of information per hit should be kept to a minimum to fulfill two simple goals: is this hit relevant to the query? And is this hit interesting in itself?\nThen comes the trickiest part: filtering and refinement inputs. On a desktop, the sidebar provides a liberal amount of space for including plenty of filters and facets; however, when screen real-estate is precious and UI is touch-based, you have to think about the UI sliders\/counters, and prioritize one attribute over another.\u00a0Our mobile devices give users the impression of having access to everything at their fingertips; but you need to provide the tools to find a needle in your mobile app\u2019s haystack of information. \n\n6- Search is an Active Process\nApps today are optimized to provide the most frictionless experience to users. We strive to minimize the amount of taps, scroll, and inputs. When you are simply consuming content in a semi-passive way, tapping a title, scrolling, reading an article or swiping through a few pictures, then a touch screen is perfect. Search isn\u2019t that simple unfortunately. While the end goal is identical: providing relevant content with the least","record_index":1},{"post_id":6127,"post_title":"Mobile Search UX - Part One: 8 Obstacles, 1 Pocket-Sized Nightmare","post_date":1492948808,"post_date_formatted":"Apr 23rd 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-8-obstacles\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Ux-mobile-nightmare_1440-360x200.png","time_to_read":6,"content":"\u2026 amount of user effort, for search, a number of steps are needed at every query. The user has to first think of an appropriate word that represents what he\/she\u2019s looking for, type it letter by letter, evaluate the result and finally reformulate his\/her query if results aren\u2019t satisfying enough. In a few words, this process is far from a one tap action.\u00a0We must serve relevant content with the least reformulations possible from the user.\n7- High User Expectations\nWe, users, have been spoiled by a few products without noticing it. For years, Google has been serving super relevant as you type query suggestions in a matter of milliseconds. We are so used to this functionality that we don\u2019t even notice anymore, it\u2019s part of a natural flow, day-to-day experience and we expect great suggestions at the first keystrokes. \nOn a different use case, Amazon completely revolutionized the way we shop. To give us access to their huge and constantly growing catalog, they implemented a blazing fast autosuggest dropdown menu in addition to powerful dynamic refinements.\u00a0To provide what will be perceived as a great search experience, we must meet or exceed users expectations previously set by big players.\n8- Virtual Keyboard, Actual Typos\nVirtual keyboards are awful. They feel very uncomfortable to use and are prone to generate typing mistakes. From a mobile search UX standpoint, it means that without a typo-tolerant search engine that returns results at the first keystroke, you have an extremely high probability of frustrating users that will inevitably hit the wrong letter and get zero results.\u00a0To ease user inputs and search process, we must be forgiving of typing mistakes and implement a typo-tolerant search solution.\nDeveloping a great user experience for your mobile app is a challenge in itself. Add search to the equation and things get even trickier. Yet, while implementing search on mobile is a demanding task, it is not mission impossible. There are definitely some great","record_index":2},{"post_id":6127,"post_title":"Mobile Search UX - Part One: 8 Obstacles, 1 Pocket-Sized Nightmare","post_date":1492948808,"post_date_formatted":"Apr 23rd 2017","author_name":"Raphael Terrier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/62f753625bcf00a16e7a144d719aac55?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-8-obstacles\/","categories":["UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Ux-mobile-nightmare_1440-360x200.png","time_to_read":6,"content":"\u2026 examples to explore today and we will get some inspiration from them in the following posts.\nIn Part 2, we will deconstruct all the moving parts of a good search on mobile; this will give us a solid ground to finish by exploring the best practices in Part 3.","record_index":3},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"Earlier this year we announced the release of Algolia Offline, which compacted the power of Algolia down to an offline search experience while retaining most of its features. One of the biggest constraints of packaging a search engine into a mobile library is the \u201cbinary size\u201d: how much space the compiled library occupies. Fitting an entire search engine under 1\u00a0MB was an exciting adventure!\nOn mobile, the binary size of your application is doubly important, because it impacts users twice: first when they download the app, and again by eating space on their device\u2019s storage.\nOn iOS, Apple prevents users from downloading an application \u201cOver The Air\u201d (i.e. from a mobile network, not Wi-Fi) if it weighs over 100\u00a0MB compressed to prevent runaway fees on data connections\u2014that limit used to be 50\u00a0MB prior to iOS\u00a07.\nEven if you are below that threshold, many users are still reluctant to install apps if they are too big. Mobile storage is far from being the most affordable medium, and competition for space is fierce. Remember the last time your phone\u2019s disk was full: probably the first thing you did was open up your application list and scroll through those taking the most space to look for apps you could afford to get rid of.\nAs a result, mobile developers strive to make their apps as streamlined as possible. This constraint naturally translates to all mobile libraries, including Algolia Offline.\nTrick #1: There are no magic tricks\nWhile Apple imposes restrictions on your applications, they also provide a useful tool to make them lighter for the end user: App Thinning. Introduced with iOS 9, App Thinning ensures that users only download the code and resources that are relevant for their device (hardware architecture and resolution). It became almost inevitable with the advent of Retina HD displays, which come with 3x resolution, meaning 9x as many pixels!\nUnfortunately, Android has yet to come out with a similar tool. Google Play has a feature called","record_index":0},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"\u2026 \u201cMultiple APK\u201d, whereby you can publish different variations of your app for different device configurations. Contrary to App Thinning, however, it is entirely manual, making deployment a lot more cumbersome. The official documentation itself states that \u201c[this] is not the normal procedure\u201d.\nEven with App Thinning, binary size is still a concern: the problem has been mitigated, not eliminated.\nCloud and mobile: two different species\nBack in 2012, Algolia started as an offline SDK, so one could be forgiven for thinking that shipping a small library was easy enough for us. However, from 2013 on we switched to a cloud-hosted engine, and its rules are very different from mobile: it\u2019s a different animal altogether.\nWith a cloud-hosted product, you control every part of your environment. You have big, fat servers with terabytes of disk: storage space is much less of an issue. Conversely, you have hundreds of thousands of customers to serve in milliseconds, so speed starts to be your main headache. It\u2019s a well-known engineering problem: the space\u2013time tradeoff. With a cloud-hosted product, the cursor is set firmly on \u201ctime\u201d.\nIn addition, since our move to a cloud product, our search engine has been augmented with lots of new features. Each feature means more code, which means bigger binaries in the end.\nAs a result, when we resurrected the Offline SDK to become Algolia Offline, it weighed around 3\u00a0MB. The challenge was now to bring it down to around 1\u00a0MB.\nTrick #2: Trust your compiler\nWe didn\u2019t have to reinvent the wheel to shrink our application by two thirds. Compilers have been around for decades\u2026 since the times when binary size was always an issue. Compilers provide lots of nice options to squeeze your code into the fewest possible bits.\nOur CTO Julien Lemoine posted a great article back in early 2013 explaining how to reduce binary size with the Android NDK. A lot of the tricks he mentioned still apply. Let\u2019s review them.\nThe first obvious","record_index":1},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"\u2026 step is to ask the compiler to optimize for space ( option, instead of the default ). In our case, this alone saves 419\u00a0kB.\nTrick #3: Resist the urge to inline\nNext, let\u2019s be careful with inline expansion. Inlining code can bring significant performance benefits, not only because it saves a function call, but also because it makes other optimizations possible\u2014optimizations which would have been impossible across a function boundary. As a consequence, though, the compiled size grows bigger, because the same code is duplicated in several places in your binary.\nAlgolia relies heavily on inlining, even forcing it for specific functions (typically those used within tight loops). In Algolia Offline, however, we disable this behavior and let the compiler decide. This saves another 144\u00a0kB.\nTricks #4\u20136: Exorcise that Dead Code\nOptimized code is nice, but what if it\u2019s not actually used? When a linker produces a library, it merely takes every object file (i.e. the machine code equivalent of every source file), and links them together. But what if some portions of the files are not actually used? Dead code stripping is the answer. It removes from every object functions that are not useful to the library\u2019s business. In our case, this saves 312 kB.\nThere\u2019s a twist to dead code stripping in the case of a dynamic library. In a static library, you only want to strip private symbols that are not useful to public functions. You cannot know which of these functions will be used, because a static library is merely a collection of reusable pieces of compiled code. In a dynamic library, however, you know which public functions will be used: only the functions exported by the library! An important step is therefore to ensure that you only export the minimum viable set of functions to serve your purpose. By tightly controlling the exports, we save another 288 kB.\nWe can take dead code stripping one step further with Link-Time Optimization (LTO). By looking at the binary as a","record_index":2},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"\u2026 whole, LTO can optimize away dead code across object file boundaries, which a regular linkage cannot do. It results in a more compact binary: 57 kB saved\u2014at the cost of a significantly longer link phase during the build.\nTrick #7: Give the boot to bloat\nWe chose to ban the C++ standard I\/O stream library from Algolia Offline, as it is still a major cause of bloat, especially on Android. We have compile- and link-time checks to make sure we don\u2019t rely on it. The gain is far from negligible: one single, seemingly innocuous  in our code adds 144 kB to the binary size!\nIt\u2019s worth noting that we now use exceptions. We avoided them in the past because they require some limited form of Run-Time Type Information (RTTI), so compilers used to trigger full RTTI support when exceptions were enabled\u2026 and RTTI causes binary bloat. Modern compilers are better optimized, so we can enjoy exceptions without paying the price of full RTTI.\nClang lagging behind on Android\nI am a huge fan of Clang, LLVM\u2019s C\/C++ compiler. LLVM is an awesome piece of software, one of the best surprises of the 2010 decade, when it started to get traction, in particular with Apple backing it.\nWe\u2019ve been compiling Algolia with Clang on Linux for months\u2014it resulted in both a non-negligible performance increase (5%) and more compact binaries.\nOddly enough, we are still stuck with GCC on Android! Although Clang is Android NDK\u2019s default toolchain since r11 (latest version is r14), it still suffers from a number of drawbacks that are deterring us from relying on it at the moment. In particular, I had trouble making LTO work on our project with Clang\u2014and without LTO, Clang cannot compete with GCC in terms of binary size.\nIt is worth noting that we already use Clang\u2019s Standard Template Library (STL), though.\nCode is not everything\nHowever smart they are, compilers can only act on your code. But when you dissect our library, you will see that code does not account for the entire size of the","record_index":3},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"\u2026 binary. A substantial part of it is occupied by data tables.\nHere, no compiler magic can save you; only proper software engineering can. (Remember that \u201cData Structures 101\u201d course?)\nOur data falls into five broad categories: Unicode, HTML entities, stop words, plurals and CJK segmentation.\nPlurals and CJK segmentation are way too big to fit in a mobile library: 5\u00a0MB for segmentation and 48\u00a0MB for plurals. We had to discard those features altogether in Algolia Offline. (As you might have guessed, I did not include them in the initially advertised size of 3 MB\u2026) With regard to the other three data tables, we found a way to make them more compact.\nThe main idea behind the compaction of our data tables is to transform random access structures into a big blob with an auxiliary index. A random access data structure is typically an array: all elements have the same size; therefore, computing the address of a given element is trivial (a multiplication). But if elements have different intrinsic sizes, you are forced to make them as big as the biggest element, which results in a lot of wasted space.\nIn a blob, by contrast, all elements are stored contiguously with no wasted space between them. This makes the structure more compact, but random access becomes impossible: computing the address of a given element is no longer easy. To solve that, we rely on an auxiliary structure (called an \u201cindex\u201d, even if it has little to do with indices in Algolia\u2019s engine) that contains the offset of each element in the BLOB. Because an offset is an integer and is typically much smaller than the actual element, this additional data structure doesn\u2019t overcome the benefit of the blob.\nBy applying this principle to our data, we saved:\n\n96\u00a0kB on stop words\n124\u00a0kB on HTML entities\u2014the structure used in SaaS is extremely sparse for very fast lookup\n476 kB on Unicode data\n\nOverall, the binary shrunk by 669\u00a0kB. Of course, we had to trade some CPU time for this: around 5% for","record_index":4},{"post_id":6207,"post_title":"Keeping mobile apps lightweight: how we shrank Algolia Offline by 69%","post_date":1492752336,"post_date_formatted":"Apr 21st 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/keeping-mobile-apps-lightweight-shrank-algolia-offline-69\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/inside-offline-360x200.png","time_to_read":11,"content":"\u2026 Unicode and HTML entities\u2014a lot more (around +50%) for stop words, but they are accessed only a few times per query, so their performance doesn\u2019t really matter.\nA streamlined library\nCombining all the above optimizations on both code and data brought the library back at around 1\u00a0MB per hardware architecture (with slight variations across architectures and platforms). While it might still sound big, it is actually an acceptable overhead in most use cases, especially these days where high-resolution images sometimes account for the major part of an application\u2019s bundle.\nHere is a summary of our efforts:\n\nInitial size = 3,003 kB\nOptimize for space \u2192 -419 kB\nDon't force inlining \u2192 -144 kB\nDead code stripping \u2192 -312 kB\nControl exported symbols \u2192 -288 kB\nLink-Time Optimization \u2192 -57 kB\nRemove STL I\/O streams \u2192 -144 kB\nCompact HTML entities data \u2192 -124 kB\nCompact stop words data \u2192 -96 kB\nCompact Unicode data \u2192 -476 kB\nResulting size = 943\u00a0kB\n\nThe above numbers are for the hardware architecture armeabi-v7a on Android but, despite small variations, they would be similar on iOS or for other hardware architectures.\nWe hope that these tips & tricks will be useful for reducing the size of your mobile libraries or applications. There are always trade-offs to be made between speed and space, but with sensible compiler and linker settings, and carefully crafted data structures, you can achieve dramatic space savings without sacrificing much of execution speed.\nIn the end, Algolia Offline performs a typical search query under 50 ms on a mid-range device. That\u2019s more than the cloud-hosted engine, of course; but thanks to the lack of any network transmission, the elapsed processing time is about the same. This makes instant search a reality on mobile\u2014even offline.","record_index":5},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"A relevant search returns the best results first. Google\u2019s famous \u201cI\u2019m feeling lucky\u201d button is a standing invitation for users to test their relevance by jumping through directly to the top hit. It\u2019s a bold gesture when you consider that Google\u2019s index contains trillions of documents, yet it works. The top result is often highly relevant and the user ends up right where they wanted.\n\u201cI\u2019m feeling lucky.\u201d - Average internet search user\nIf we want to understand what a consumer expects from their search, luck is a good place to start. If we want to understand what it takes to build that search, luck is a very bad place to start.\nGreat relevance is within reach for any search project, but it requires a careful understanding and application of complex techniques, several of which we'll look at in this article. Whether you're using Elasticsearch or Algolia, the goal should be to give your users a search they'll love, a search so good you'd even consider putting an \"I'm feeling lucky\" button next to it!\nWould I put an \u2018I\u2019m feeling lucky\u2019 button next to my search box?\n\nIn Part 1 of this series, End-to-End Latency, we discussed the perfectly unreasonable search expectations of the average internet user, starting with speed. Global end-to-end response times must be lower than 100 milliseconds for search-as-you-type experiences to feel natural, with less than 50ms being ideal.\nIn this part of the series, Relevance Isn\u2019t Luck, we\u2019ll look first at consumer search behavior and why it challenges traditional approaches to relevance. Then we\u2019ll look at how developers are adapting to these changes using the available tools, looking first at Elasticsearch and then Algolia.\nIntuitive\u00a0user experiences require\u00a0smarter relevance\nConsumer-quality search experiences are so streamlined that it's hard to imagine what's going on underneath, even for engineers. Yet many things that we take for granted as users have serious consequences for engineers trying to","record_index":0},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 achieve good relevance.\nSingle input\nIn the early days of the web, most searches required you to separate your query into different boxes, one for each field you wanted to search in. Today, this is only the case for advanced searches, with the irony being that it\u2019s actually simpler for the search engine to find and rank records when the query is split up into different fields.\n\nInstant search\nSearch-as-you-type experiences have become the standard since Google Instant was introduced in 2010. Users love the immediate feedback and responsiveness of instant experiences. Product designers love instant search because it\u2019s an efficient way for users to discover the content on their site.\n\nInstant search introduces a brand new set of relevance challenges. The search engine can\u2019t wait for a full word to be typed, it has to find and rank matches using only the first few letters.\nInstant search vs. instant suggest\nNot all instant searches are created the same. A variation called instant suggest is a multi-step process. Each new keystroke produces suggestions for keywords but not actual search results. Search results can only be reached by clicking on a suggestion or hitting the enter key.\nInstant search produces real search results with every keystroke, no clicks or enter keys required. The user benefits from a streamlined experience and the development team has less moving parts to tune and maintain. But as we\u2019ll see later, achieving the necessary speed and relevance is much harder for instant search than instant suggest.\nForgiving mistakes\nOn-the-go users misspell, omit and juxtapose words, and frequently get distracted and stop typing mid-sentence. Yet, when they gaze back down at their screen, they still expect to see the contact, movie or product they were searching for. This requires search engines to be much more forgiving than they were originally designed to be.\n\nMobile keyboards make forgiveness a must-have for two reasons: it\u2019s easier to make mistakes and","record_index":1},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 it takes longer to correct them. To make the search engine more forgiving, we\u2019ll need to apply a variety of text and language processing techniques, designed to handle everything from typos to languages like Dutch and German that dontalwayshavespacesbetweenwords.\nPromoting Results\nThousands of pages on the internet can tell you how to treat the common cold. How do you know which one to trust? Text-based signals aren\u2019t enough\u2014whether the page mentions the word cold 5 times or 50 times makes it no more or less trustworthy. To solve this problem for web search, Google created the PageRank algorithm which determines a page\u2019s importance based on how many other pages link to it (and what those pages\u2019 rankings are).\nMost search engines, including Elasticsearch and Algolia, provide a way to promote results based on signals that aren\u2019t related to the text itself. High-level attributes like trustworthiness, popularity, and affordability are commonly distilled down to a set of numbers, which are then used to sort records.\nA particularly challenging part of relevance design is combining text-based relevance with strategies for promotion. The text represents the user\u2019s intent while the promotion represents yours. These have to be considered in the right order.\nFor a great search UX, what the user is looking for must come before what you want them to find.\nRelevance is a journey, not a destination\nRelevance can be broken down into requirements, design, implementation, testing and maintenance phases. Now that we know what some of the requirements are, let\u2019s jump into design and implementation, first with Elasticsearch and then with Algolia.\nAbout Elasticsearch\nIf you\u2019re not familiar with Elasticsearch, you can think of it as a set of tools for building a broad range of search and analytics applications. At the core of Elasticsearch is Lucene, a widely-used open source search engine first released in 1999. Part of Lucene\u2019s wide applicability lives in its ability","record_index":2},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 to apply very different similarity models to calculate relevance, including Okapi BM25 and TF-IDF, the new and former defaults used by Elasticsearch. In this sense, Lucene is a swiss-army knife, not a machete.\n\u201cRegardless of how easy Elasticsearch has made it, Lucene-based search is a framework, not a solution.\u201d \u2014Doug Turnbull, author of Relevant Search, June 2016\nThe job of a relevance engineer is to choose tools from the framework for the application in question. For consumer-grade search, tools are needed to handle:\n\nMulti-field matching - Testing each query word against each searchable field\nPrefix matching - Matching word parts for as-you-type search\nForgiveness - handling typos, concatenations, plurals, omissions\nPromotion of records based on non-textual signals like price or popularity\nA predictable, debuggable way to combine and rank all of these signals\n\nAs we proceed, we must keep an eye on how each new layer impacts the relevance and the performance of the search as a whole. Like with most things in software, getting each component to work is easy. The hard part is making them all work together.\nPoor applicability\nThere are a few features of Elasticsearch that should be ruled out early on, even if they look promising at first glance. This isn\u2019t because they\u2019re bad features, but because they\u2019re not going to be powerful or fast enough for what we want to achieve.\nFuzzy queries\nFuzzy queries allow forgiveness for typos and omissions, but for search-as-you-type they are simply too slow. Strategies to speed them up, like increasing prefix_length or reducing max_expansions, will cause other problems. In the case of setting prefix_length to 3 or more, we lose the ability to correct typos in the first 3 characters of the word.\nThere are also relevance concerns. There is no straightforward way to rank exact matches above fuzzy matches - the match query gives all fuzzy matches a constant score of 1, regardless of how inexact the query was. This has the","record_index":3},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 consequence of making some records unreachable, even by exact queries, when typo-free variants are scored higher for other reasons. Fuzzy matching doesn\u2019t work with phrase matching, and it also can\u2019t be used with cross_fields in multi_match queries, which prevents us from specifying field-specific boosts.\nPrefix, match_phrase, and match_phrase_prefix queries\nPrefix queries allow you to match the first letters of a word but don\u2019t handle phrases, where at least one full word has already been typed. match_phrase only works with whole words. match_phrase_prefix does work with prefixes but on the last world only, and it fails to find matches when the prefix is short (1-2 letters) and the index is large. For this reason, Elasticsearch advises to avoid it.\nTF-IDF\nIf the fields you\u2019re searching tend to be short\u2014names, titles, descriptions, sentences or short paragraphs\u2014you probably want to turn off TF-IDF\/BM25 altogether. Term frequency (TF) won\u2019t be very helpful, as several occurrences of the same word in a short span isn\u2019t usually a meaningful relevance signal. Inverse-document frequency (IDF) can unfairly penalize high-signal terms simply because they\u2019re popular, like \u201cPlato\u201d will be in a search for philosophy book titles. You can read a very detailed explanation of this phenomenon from Doug Turnbull and the really sharp team at OpenSource Connections called Title Search: when relevancy is only skin deep.\nThere\u2019s one other general advantage with not using IDF, which is that the relevance of a single document is self-contained. A new influx of documents with previously unencountered proportions of keywords can\u2019t cause a relevance regression in the documents that were already present.\nHigher applicability\nWhile the previous Elasticsearch concepts might not prove that useful in the end for consumer-grade relevance, the next few can add more value, provided that we understand their limitations.\nminimum_should_match\nThis important configuration knob","record_index":4},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 lets a query return results even if some query words aren\u2019t in the record. For example, if the user searches for \u201cmy quick brown fox\u201d, the query can return a record that only contains \u201cfox\u201d but neither of the other 2 terms, if the minimum_should_match is 25% or lower. This scenario is especially important in instant search where not all query words are present until the end (or perhaps never at all).\nA minimum_should_match value that is too high will exclude too many potentially good results, leading to zero-result scenarios more often. If the value was 50% in the example above, the \u201cfox\u201d result would not have been returned. A value that is too low will cause irrelevant results to appear. At 25%, the above query would also return results where \u201cmy\u201d was the only word that matched. Finding the right value for this parameter is important, and requires you to understand both your data and the way that your users are searching.\nCompletion Suggester\nThe Completion Suggester can be used to create an instant suggest experience, as described above. It cannot be used to create a full-record instant search experience, as suggesters are limited to searching in only one field. Still, instant suggest has advantages that make it a popular choice, namely that it\u2019s fast. Suggesters have a completely different in-memory execution path which helps make them fast enough for as-you-type search.\nOn the relevance side, suggesters have weighting and can do fuzzy matching. On the other hand, clients have to de-duplicate data and advanced features like filtering and faceting can\u2019t be used. Read more about the pros and cons here: Quick and Dirty Autocomplete with Elasticsearch Completion Suggest.\nn-grams\nOne of the most versatile tools in the Elasticsearch toolbox is n-grams, where words are broken down and indexed as smaller groups of letters. The \u201cbrown\u201d would be indexed with the tokens [\u201cb\u201d, \u201cbr\u201d, \u201cbro\u201d, \u201cbrow\u201d, \u201cbrown\u201d, \u201cr\u201d, \u201cro\u201d,","record_index":5},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 \u201crow\u201d, \u201crown\u201d, \u201co\u201d, \u201cow\u201d, \u201cown\u201d, \u201cw\u201d, \u201cwn\u201d and \u201cn\u201d]. \nA subset of n-grams called edge n-grams are a better choice for instant search, because all tokens start from the beginning of the word and because there will less tokens overall, improving the speed of the search. The edge n-grams for \u201cbrown\u201d are just [\u201cb\u201d, \u201cbr\u201d, \u201cbro\u201d, \u201cbrow\u201d, \u201cbrown\u201d]. With n-grams a query for \u201crow\u201d would match \u201cbrown\u201d. This is called infix matching and it usually does more harm than good for relevance\u2014another good bad example is a search for \u201cnathan\u201d matching \u201cjonathan\u201d. Edge n-grams only afford prefix matching, so \u201crow\u201d will only match \u201crower\u201d or \u201crowboat\u201d, not \u201carrow\u201d.\nSimilar to the limitation that fuzzy matching has with ranking exact above inexact matches, n-grams don\u2019t come with a way to rank full word matches higher than partial ones. N-grams don\u2019t store information about whether the token represents a full word. Without that, we can\u2019t guarantee that \u201cmotor\u201d will be ranked above \u201cmotorbike\u201d for the query \u201cmotor\u201d.\nShingles\nAnother limitation of n-grams is not being able to use them for phrase matching, where we want to giving ranking preference to records where the terms match in the right order. A name search for \u201canne marie\u201d should rank results for \u201cmarie anne\u201d after \u201canne marie\u201d. This is very important when there are many people named \u201cmarie anne\u201d, because the user may never be able to reach the \u201canne marie\u201d record they were looking for otherwise. N-grams don\u2019t store the position or order of tokens, so it\u2019s impossible to satisfy this ranking requirement with n-grams alone.\nShingles are often used instead of (or in concert with) n-grams because they do retain information about word proximities. Shingles are similar to n-grams in that they represent tokens, but instead of breaking up words into letters, they break up phrases into words. The shingled tokens","record_index":6},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 of \u201cquick brown fox\u201d are [\u201cquick brown\u201d, \u201cbrown fox\u201d].\nFor search-as-you-type relevance, shingles will have to be used alongside n-grams because shingles won\u2019t help us do prefix matching. An Elasticsearch implementation using both techniques is referred to here by DogDogFish. However, the increase in index size and amount of scanning at query time make the combination of shingles and ngrams almost certainly too slow for full instant search. Presumably for that reason, and also to handle typos, the DogDogFish team ended up with a mix of instant suggest for keywords, instant suggest for spelling correction, followed by keyword search to find results.\n\u201cIf there is a particular search found to yield bad results, it can be easy to optimise towards improving and fixing that search, but then other searches end up suffering as a result.\u201d\u2014DogDogFish\nDelivering relevance with Algolia\nUnlike Elasticsearch, Algolia is not a swiss-army knife for implementing a myriad of front-end and back-end search use cases. But nor is it a single blade capable of only one kind of search\u2014the API is highly customizable and individual search implementations vary widely by data, user interface and audience.\nA good way to think about Algolia is a set of wrenches, each a different shape or size, purpose-built to tighten a specific nut or bolt of a fully instant consumer-grade search.\n\nMulti-field\nAlgolia is designed to index structured data, not large blocks of text, so searching in multiple fields of a record is the default behavior. The list of fields to be searched is specified in the searchableAttributes property of the index. It can be changed at any time in the Algolia Dashboard or API.\nThe list of searchable attributes is ordered, with the most important attributes being first. The ordering is considered at query time by the ranking formula, which is the foundation of how Algolia computes relevance. Algolia\u2019s ranking formula contains a handful of different criteria,","record_index":7},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 each meant to handle a specific relevance need. For multi-field-aware ranking, matches in some fields need to count more than others. This is done by the formula\u2019s attribute criteria.\nUnlike Elasticsearch, there are no term or match queries that operate only against single fields; conceptually everything is multi_match. Likewise, there are no operators like bool or dis_max to combine relevance signals from multiple fields\u2014 this is handled by the ranking formula itself.\nThe ranking formula, including the attribute criteria, doesn\u2019t require developers to specify how much more important one criteria (or one attribute) is than another, only that it is simply more important. In contrast to Lucene-style boosting, no coefficients need to be provided.\nAlgolia uses a tie-breaking algorithm, not the computation of a score, to rank documents. The algorithm considers each criteria, one after the other, to successively sort the result set into parts. Read more here about the tie-breaking algorithm and ranking criteria.\nPrefix matching\nFor the query \u201csun\u201d, exact matches like \u201csun\u201d need to be ranked above partial matches like \u201csunflower\u201d. Otherwise, records about the sun will be impossible to reach if there are many records about sunflowers\u2014there is no way for the user to be more precise than \u201csun\u201d when they are looking for records about the sun.\nShowing a sunflower to a consumer when they wanted the sun will cause a ton of frustration, so much that we want to guarantee it won\u2019t happen. The exact criteria in the Algolia ranking formula makes sure that full-word matches count more than partial-word matches. The criteria not only can rank one prefix against one word, but it also combinations of multiple full words and multiple prefixes.\nForgiveness\nThe typo criteria in the ranking formula guarantees that exact matches for query words count more than matches where this a mistake. The reasoning is because sometimes a typo is not actually a typo, it just","record_index":8},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 depends on what the underlying data is.\nTake a query for \u201cjon\u201d. This is either an exact match for \u201cJon\u201d or a 1-letter omission typo for \u201cJohn\u201d or \u201cJoan\u201d. The search engine needs to guarantee that people named \u201cJon\u201d appear before \u201cJohn\u201d, while still showing \u201cJohn\u201d and \u201cJoan\u201d afterwards. The \u201cjon\u201d query is simultaneously a typo for some records and not a typo for others. The records for which it is not a typo need to come first.\nBoth Algolia and Elasticsearch use the Damerau\u2013Levenshtein distance algorithm for computing typos based on word distance. Spelling mistakes, however, are just a small fraction of dealing with the reality that it is a human in front of the search box, and that humans speak different languages and phrase things in different ways.\nIn Elasticsearch, search teams add analyzers to the indexing path or the query path to handle the quirks of language, and there are quite a few to choose from. With Algolia, queries are put through a two-step tokenization and alternative-word matching process that handles many common cases for a variety of different languages. For a deep dive, check out the Query Processing edition of the Inside the Engine series, but here are a few examples:\n\nAbbreviations and apostrophes: In French, \u201cl\u2019aigle\u201d (the eagle) needs to be indexed as aigle not laigle - simply removing the apostrophe will produce very bad relevance. But if indexing English, \u201cdon\u2019t\u201d should be just one token \u201cdont\u201d.\nAgglutinative languages: Some languages like Japanese, Chinese and Korean don\u2019t have word separators. A statistical approach is required to break words into tokens. Here, typo tolerance is not based on word distance but on synonyms between ideograms, so that you can search for simplified Chinese using traditional Chinese and vice versa.\nPrefix correction: Beyond Damerau\u2013Levenshtein, Algolia keeps a dictionary of words that can be used to correct prefixes. If the user inputs \u201cjhon\u201d the","record_index":9},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 prefix will be corrected to \u201cjohn\u201d automatically, since \u201cjohn\u201d will have been in the dictionary. This situation is often overlooked but is essential to a good instant search experience. Mistakes are very common in the first few characters of a search, as we all know from using our mobile devices.\n\nTyposquatting\nWhen you mistype a domain name and end up on an ugly \u201cDomain For Sale\u201d page, you\u2019ve been a victim of typosquatting. Domains aren\u2019t the only place typosquatting happens. It\u2019s also common on social networks, where it takes advantage of the search\u2019s preference for exact matches. @BarakObama on Twitter, who is not @BarackObama, has 16.3k followers. To prevent typosquatters from getting to the top of the search results, we need to do something that is the opposite of what we usually recommend - we need to consider another signal to be more important than the text. Because the elements of Algolia\u2019s ranking formula can be reordered, this is easy. The recommended solution is to add an isPopular attribute to the data, set it to true for the top ~1% of accounts, and put it above the typo criteria in the ranking formula.\nPromotion\nMulti-field matching, prefix matching, forgiving mistakes and understanding the intricacies of language all fall under an umbrella that Algolia calls textual relevance. Textual relevance is all about what the user wants to find.\nBusiness relevance is about what you want the user to find. When your index contains textually identical people, products or items, Algolia gives you the custom ranking to push the most important records to the top. A custom ranking might be driven by business metrics, popularity metrics, or attributes intrinsic to the record, like price. As with textual relevance, your Algolia custom ranking can take into account many signals, while still giving you a guarantee about the order in which they matter.\nBy default, the ranking formula applies textual criteria first and only then the custom ranking.","record_index":10},{"post_id":6163,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 2: Relevance Isn\u2019t Luck","post_date":1492677174,"post_date_formatted":"Apr 20th 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/feel-lucky-1-360x200.png","time_to_read":22,"content":"\u2026 This provides a guarantee that promotional signals will not override the textual quality of the match, which is one of the most common situations that lead to bad relevance and a bad user experience.\nConclusion\nIn Part 1 of this series, we looked at how instant search forces us to think about latency in a new way. The same is true for relevance. We can no longer search by full words only nor assume the query has made it accurately from the user's head into the search box.\nThe path to great relevance looks different whether you are using Elasticsearch or Algolia. Because relevance and user experience are inextricably linked, the type of user experience you will end up providing may also be different. Recent Elasticsearch additions like the Completion Suggester are designed to help developers build an instant suggest experience that is fast enough for as-you-type search, with a tradeoff of some functionality. For full instant search, Elasticsearch users should expect to make trade-offs between speed, relevance and how predictive or forgiving the user experience is.\nAlgolia is built from the ground-up for full instant search. Tradeoffs center around different ways to model data and finding the most cost-effective ways to integrate, but not around speed, relevance or user experience.\nRead other parts of the Comparing Algolia and Elasticsearch for Consumer-Grade Search series:\n\nPart 1 - End-to-end Latency\nPart 2 - Relevance Isn't Luck\nPart 3 - Developer Experience and UX\n\n\n\nWant to learn more?\nJoin us for our next webinar to learn how you can achieve fast and relevant results from your search implementation.\nWebinar: Achieving Speed and Relevance in Search\n Sign Up\n ","record_index":11},{"post_id":6189,"post_title":"Small but impactful search UX tips for mobile web","post_date":1492495201,"post_date_formatted":"Apr 18th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-tips\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/impactful-UX-tips-360x200.png","time_to_read":8,"content":"As part of our annual tradition of giving a gift to the community, these past few weeks I was part of the team working to\u00a0implement search inside of Yarn. During that time, I noticed some small quick-wins that could make our search boxes better,\u00a0and today I\u2019d like to share a few of them.\nUser experience is important at Algolia - both the developers who use our API and the end-users who use the products they build - and so we look a lot at how the tools we provide to developers can make it easy to create great UX.\nOne of our most recent projects, react-instantsearch, comes with many optimizations and we\u2019ll highlight a few of them in this post.\nSubmit UX\nWe use  because it allows us to take advantage of semantics for screen readers that tell the user it\u2019s a search input, and it will also show a ? (magnifying glass icon) as the return button on Android. In Chrome and Safari this has more advantages, namely that it shows a search button on one side, and a clear button on the opposite side.\nTo get that functionality for everyone, react-instantsearch wraps the search box in a form, and includes two extra buttons: one with type reset to clear the input, and another with type submit that will be used if \u201csearch as you type\u201d has been deactivated -we hide the latter with some css to avoid duplicating functionality.\nSee the Pen Default React InstantSearch layout by Algoliamanager (@Algolia) on CodePen.\n\nWe only want to submit the form when a user types something, so we will put the \u201crequired\u201d attribute on the input. That prevents a user from submitting an empty search. However the default result isn\u2019t too beautiful:\nA required input\nWe can avoid this message showing up, but still avoid the submitting of the form, by adding the \u201cnovalidate\u201d attribute to the form.\nSee the Pen input type text vs input type search by Algoliamanager (@Algolia) on CodePen.\n\nButton UX\nNext, we can go further in improving the search box itself. On mobile, the submit button","record_index":0},{"post_id":6189,"post_title":"Small but impactful search UX tips for mobile web","post_date":1492495201,"post_date_formatted":"Apr 18th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-tips\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/impactful-UX-tips-360x200.png","time_to_read":8,"content":"\u2026 becomes more prominent, since it\u2019s visible as the return key (on both iOS and Android). We would love to have something that has to do with search to be displayed there instead of return. On Android we already solved this, just by using input type=search, but iOS still shows \u201creturn\u201d, because iOS requires the form to have an action. Search is available on the page we\u2019re on, so we can leave the action attribute empty, and then it means the current page. This will cause the submit button to read \u201csearch\u201d instead, and it will be translated to the language of the keyboard you\u2019re using.\nYou can play around with the differences on all browsers in this codepen:\nSee the Pen input type text vs input type search + forms by Algolia (@Algolia) on CodePen.\n\nDon\u2019t forget to Blur\nDon\u2019t fret, we aren\u2019t done with the search button yet. If you have a button that says search, you expect the search results to appear. Because we show the results in realtime, while you\u2019re typing, we don\u2019t need to calculate the results at this point, so pressing search now does nothing at all for users. We fix this by\nblurring (which is the opposite of focusing) the search input. The keyboard will then be dismissed when you hit search, and you can see all of the results.\nNo need for Autocorrect\nBecause Algolia is typo-tolerant, mobile-specific features like autocorrect become a bit obsolete. We don\u2019t want to spend time undoing the suggestions by our browser to suggest other words, especially if you\u2019re in a case like Yarn, where a lot of the packages have unique spelling that we don\u2019t want to be corrected.\nTo disable autocorrect, you\u2019ll need three attributes \u2014 , , and  \u2014 the latter avoids starting a search with a capital letter all the time. With those improvements, a search on a mobile device will look like this:\nThe situation before these improvements\n \nThe situation after these improvements\nA last thing that has been added is the \u201crole\u201d added to the form.","record_index":1},{"post_id":6189,"post_title":"Small but impactful search UX tips for mobile web","post_date":1492495201,"post_date_formatted":"Apr 18th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-tips\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/impactful-UX-tips-360x200.png","time_to_read":8,"content":"\u2026 Since December of 2015 , with html-aria#18, it\u2019s valid html5 to add an additional role to a form, and in this case a search role makes a lot of sense. In screen readers that support it, the search input will be read as a search form with inputs, and that\u2019s exactly what we want.\nWith all those things \u00a0implemented in react-instantsearch in\ninstantsearch.js#1999 and instantsearch.js#2046, we get a search form that looks like this:\nSee the Pen React InstantSearch SearchBox by Algoliamanager (@Algolia) on CodePen.\n\nYou can get a search box like this one by using react-instantsearch to make your search experience, or apply this knowledge yourself.\nWhen making the search for Yarn a few other things came to my attention \u2014I\u2019d like to thank James Kyle a lot, because he kept finding new things to give me feedback on. One of these is stylistic, and that is to always make sure your search input stands out from the background. You can do that by making sure it has a white (or light) background, a placeholder that\u2019s not too light and not too dark. You should add an identifiable \u00a0icon (for example ?) to get extra clarity, and it\u2019s also good practice to give a special border to active inputs. My colleague Kevin Granger made a really cool webapp that allows you to create all kinds of search boxes that follow these criteria at http:\/\/shipow.github.io\/searchbox.\nOpenSearch\nOpenSearch in action on the Yarn site\nAnother thing I learned about while making search for Yarn is the OpenSearch spec. It was developed by a9 (which is a subsidiary of Amazon) a long time ago, and deals with a lot of browser and search functionality. You can make your whole search available in the dropdown that comes up when you use \u201csearch in site\u201d. Making your search available in the rss format that opensearch expects isn\u2019t what Algolia is made for right now, but we can make browsers handle the \u201csearch in site\u201d feature. To do that we have to add a  to the  of every page. In that we add a ","record_index":2},{"post_id":6189,"post_title":"Small but impactful search UX tips for mobile web","post_date":1492495201,"post_date_formatted":"Apr 18th 2017","author_name":"Haroen Viaene","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/48ff3c037bd28b8fccabe52751e9700f?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/mobile-search-ux-tips\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/impactful-UX-tips-360x200.png","time_to_read":8,"content":"\u2026 and , a  is also added to show up in the UI of browsers correctly. The \/opensearch.xml file should contain something like this:\n\n\n\nYou can read more about the implementation of OpenSearch in Chrome,\nFirefox, Safari, Internet Explorer, Yandex and in general.\nThe Devil is in the Details\nI enjoyed discovering these quick fixes that get a search experience from good to great, and hope they help you create beautiful search for your users. If you have any questions about the tips above, or have UX tips of your own, we\u2019d love to discuss this more with you on the Algolia forum.","record_index":3},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"Our road\u00a0to supporting both official iOS languages\u00a0with the same code base\nSince 2014, the Apple ecosystem has had two official programming languages: Objective-C and Swift. For library providers like Algolia, supporting both languages is a must\u2014ideally from the same code base. The traditional approach is to support Swift via Objective-C. Despite the fact that it is much less frequently done the other way round, we chose to travel off the beaten path and have our entire code base written in Swift. \nIt has been a challenging adventure\u2014but also a rewarding journey. Let\u2019s see why.\nObjective-C has been the main programming language for Apple platforms since the release of Mac OS X. It was actually imported along with NeXTSTEP during Steve Jobs\u2019 spectacular comeback. Before that, the official programming languages for Mac OS were C\/C++, and even Pascal at one time.\nWhile Objective-C\u2019s syntax was inspired by Smalltalk\u2014one of the first purely object-oriented languages\u2014its goals and implementation were more pragmatic. Objective-C is mostly a preprocessor on top of C, combined with a thin runtime. In other words, it\u2019s an object-oriented, orthogonal extension of C. (This is why C++ can also be extended in a similar fashion, leading to the three-legged beast that is Objective-C++\u2026 but let\u2019s keep away from horror stories for this article.)\nWhile Objective-C has been criticized by many, mainly for its unusual syntax, it is an elegant programming language. It\u2019s highly dynamic, making difficult software engineering problems like proxying or mocking a breeze. It has a very clean object model, with classes and metaclasses. Though not fashionable any longer, it remains a remarkable piece of software engineering.\nA leap into the future\nYou must be wondering: if Objective-C is such a great language, why did Apple feel the need to move away from it?\nEven though Objective-C has evolved over the years, adding modern constructs like closures (called \u201cblocks\u201d in","record_index":0},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 Apple parlance) and generics (more like type annotations, actually, but let\u2019s not be picky), it still suffers from a few shortcomings.\nIts main drawback is its lack of compile-time checks. The flipside of being a dynamic language is that many things happen at runtime, making it harder for the compiler to detect potential bugs. Also, because every function call translates in a call to the runtime, performance remains intrinsically limited.\nAnd, of course, there is that unfamiliar syntax that can deter newcomers. I actually suspect that this is the main reason why Apple moved away. Not the flaws of the language in itself, but the fear that programmers would steer away from their platforms because they didn\u2019t feel comfortable writing code for them.\nSwift: powerful but complex\nSo, what is Swift all about?\nSwift brings a seemingly familiar, curly-brace touch to Apple programming. Looks can be deceiving, though, because it ships with many modern notions not supported by other \u201ccurly-brace languages\u201d, like optionals, case classes, or pattern matching.\nIt also brings not-that-modern, but still useful notions like true generics (as opposed to C++\u2019s templates or Java\u2019s type-erasure system), value types (supported by C++ from the origins) or default argument values (idem).\nMore importantly, Swift chooses a different trade-off between robustness, speed and dynamicity. It is strongly typed\u2014sometimes bordering on rigidity, as with optionals, but that\u2019s for the good cause\u2014which translates into more compile-time checks\u2026 and less runtime surprises (a.k.a \u201cbugs\u201d). It is fully compiled, instead of just translating into runtime calls, which means it can run faster, especially since it leverages the awesome LLVM compiler infrastructure and all the optimizations it provides.\nAll these benefits come with a cost, though. While its syntax looks easier, Swift is actually a difficult language to master\u2014as complex as C++ is. Also, it uses unique constructs, like  or ","record_index":1},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 conditional assignments, and an odd error handling mechanism\u2014a hybrid between exception throwing\/catching and traditional return codes.\nTherefore, its benefits for new programmers is not that obvious. I have seen junior programmers pick up Objective-C really easily and quickly, while I must admit that I still struggle with Swift at times.\nJumping headfirst into Swift: Lessons learned\nAt Algolia, we are always eager to try new things. It\u2019s part of our DNA. So, when Swift 1 was released, we jumped on the opportunity\u2014that was our first mistake. \nThere was no obvious need for it. Objective-C was still supported: the entire legacy code base, including Mac OS and iOS themselves, was written in it, so it would not disappear in the near future. Because of that, Apple had made sure that Swift could leverage Objective-C code, which means we could have supported Swift without writing a single line of code in that language.\nHowever, some customers at that time were already going full Swift and asking us to follow them\u2014 the adventure looked too exciting for us to pass up. \nWe wrote our Swift API Client, while keeping our Objective-C API client.\nA few months later, Swift 2 was released. It broke everything, but since version 1 was mostly experimental, it was fine to rewrite the library for our next major release..\nMaintaining two code bases hurt. Algolia is a fast-paced company, with new features being released almost every week. Since many of those features need to be reflected in our API clients, those libraries evolve rather quickly as well. Therefore, having two very distinct code bases to support the same platforms began to itch\u2026 especially given a major refactoring was required to pave the way for our Algolia Offline release.\nThe logical consequence was to drop one of the code bases\u2014easier said than done! Deciding which one to drop was tough.\nThe natural choice would have been to keep Objective-C. Its feature set is entirely covered by Swift (but not vice","record_index":2},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 versa), and it\u2019s more mature, with a stable API and ABI. It\u2019s what most people advise, for good reason.\nOn the flip side, Swift offers nice abstractions that better fit our needs. In particular, it can make value types (like integers or floats) optional, which is useful for handling our search parameters. It also provides advanced enumerations, allowing us to deal elegantly with special cases, like our removeStopWords parameter, which accepts both a boolean or a list of strings.\nThe lack of ABI stability is not a problem in our case, since our API Client is open source and delivered through CocoaPods.\nFinally, as stated above, some of our customers already went 100% pure Swift, and we didn\u2019t want to betray them.\nSwift first, Objective-C as a fallback\nSo we chose Swift. It turned out to be a challenging task.\nWe didn\u2019t do it right from the start. We had to fail, and learn from our mistakes\u2014only after version 4 of our Swift API can we speak from a state of confidence about our decision. That said, we may still discover mistakes in the wild out there, waiting to be fixed in future releases.\nBecause the ecosystem still has a huge proportion of its code base written in Objective-C, Apple went to a lot of effort to \u201cbridge\u201d Objective-C and Swift as seamlessly as possible. Whenever some Swift construct has an equivalent Objective-C construct, it is automatically made available from Objective-C\u2014it\u2019s like magic.\nThe problems arise when the magic falls short, simply because there is no equivalent notion in Objective-C for what you are trying to express in Swift. In particular, the aforementioned optional value types or advanced enumerations cannot be bridged.\nThe easiest way to solve this is to use a subset of the Swift language, namely only features that can be mapped to Objective-C. It\u2019s safe, but it negates the purpose of choosing Swift as an implementation language in the first place. If we limit ourselves to what is supported by Objective-C, then we","record_index":3},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 might as well write code in Objective-C directly. Also, by using types or constructs that are not typical of Swift, we condemn ourselves to a poor developer experience (DX).\nBy writing some Swift-specific code and having an Objective-C compatible fallback whenever necessary, we gave ourselves more work up-front\u2014and probably a little more maintenance\u2014but it yields a much better DX in the end.\nLet\u2019s see how it goes in practice with our  class, which gathers our search parameters. Every parameter is embodied by a property with the same name. It is first implemented with the optimal type for Swift. Since most of these types are bridgeable to Objective-C, no more work is required for most of the parameters. For the few non-bridgeable properties, we add an Objective-C compatible fallback. This property is inevitably also visible in Swift, which could create confusion for the developer. However, we use a few tricks to mitigate the impact:\n\nThe Swift name is prefixed with , which makes it obvious that it is intended for Objective-C, and also ensures it appears last in code completion.\nUsing an  annotation, we map this property to the parameter\u2019s name. Since the original property is not visible in Objective-C, no conflict is induced by this. All goes as if the property simply had a different type in Objective-C.\nThe fallback property is not documented, which means it does not appear in the generated reference documentation.\n\nWe looked at other potential solutions, like having an Objective-C specific subclass. This leads to even better name insulation; however, it poses tricky covariance issues when extending the class, like what is done by InstantSearch Core for Swift. Finally, naming tricks were easier to maintain and created less friction for users of the library.\nThe end result is cool autocompletion that works smoothly from both languages:\n\n\nSmoothing out the rough edges\nWe discussed properties, but what about methods? Thanks to named arguments, Swift is actually","record_index":4},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 quite close to what Objective-C can achieve. And thanks to well-established naming conventions, Xcode is able to automatically compute the name of a selector from a Swift function, and this computed name will be suitable 90% of the time. For the remaining 10%, though, explicitly specifying an Objective-C selector name can lead to a better DX.\nFor example:\n\nthe Swift function  would normally translate into the Objective-C selector , but we preferred .\nSimilarly,  should translate into , but we chose to make the first argument more explicit with .\n\nGenerally speaking, it is OK to be more verbose in Objective-C than in Swift.\nAlso, code is fine, but shipping a good software library involves more than just writing code. Extensive documentation is a must.\nThe traditional tool used in the Apple ecosystem, Appledoc, only supports Objective-C. Instead, we use Jazzy, which supports both Swift and Objective-C, but (at the time of writing) only generates Swift documentation from Swift code. Objective-C developers have to guess the selector\u2019s name based on the Swift function\u2019s name\u2014or rely on Xcode\u2019s autocompletion. As you can see, there is no perfect solution yet for cross-language documentation.\nWhen Apple falls on your head\nThe cool stuff with Apple is how predictably unpredictable they are. Just when we thought we had Swift figured out\u2026 Swift 3 was released, and broke everything. Again.\nBeing hit by an apple is not necessarily a bad thing (ask Isaac Newton). Swift 3 brings huge improvements over Swift 2, and was totally necessary. It really looks like \u201cSwift finally done right\u201d. The migration gave us an opportunity to improve our code, sometimes in a backward-incompatible way that would not have been possible without a new major version.\nStill, a forced migration does have some drawbacks. You need to support two branches for a while; more importantly, you don\u2019t control the timeline\u2014and that\u2019s especially true with Apple. The scope kept moving until the","record_index":5},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 final release, too. Swift underwent a tremendous amount of changes between the first and the last beta versions (making them more like alpha versions). There were even changes between the last beta and the Gold Master (GM)!\nRegardless of this constant stream of changes, customers asked for our support from day one. They wanted their app to be updated in the App Store as soon as iOS 10 was out. All this conspired to create a \u201crush effect\u201d\u2014which is never the best way to ship quality software.\nWe managed to handle it pretty well, all things considered. Version 4.0 of our Swift API Client was out on September 14th, 2016\u2014one day after Xcode 8.0 final was officially released.\nAll of this could have been avoided if modules were distributed in binary form, but Cocoapods encourages delivering modules in source form, and anyway Swift doesn\u2019t have a stable ABI yet. Although ABI stability was initially planned for version 4, due in Fall 2017 (see the Swift Evolution Proposals), it now appears that it will be deferred again.\nExploring the jungle\nAs a conclusion, supporting both Swift and Objective-C from the same code base is definitely possible, from either of the two languages. Which one you pick is a question of which compromises you\u2019re willing to make. Objective-C is the safe and easy choice. Choosing Swift is more complicated, and will expose you to more hectic schedules; however, you will be rewarded with a slightly better DX for your Swift users.\nHaving walked this path, and being able to weigh the benefits and drawbacks, I would still recommend sticking to Objective-C for the time being as far as universal libraries are concerned. The decision for a standalone application is an entirely different tradeoff, and Swift does make a lot of sense in this case.\nDespite the huge improvements Swift has undergone, the language is not yet mature enough to write future-proof software\u2014as all good libraries should be. It is surprisingly insufficient in some crucial","record_index":6},{"post_id":6150,"post_title":"A Tale Of Two Languages: Supporting Swift & Objective-C","post_date":1491987097,"post_date_formatted":"Apr 12th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/supporting-swift-objective-c\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/tale-of-two-languages-360x200.png","time_to_read":13,"content":"\u2026 areas, like inter-thread synchronization. Even today, it\u2019s not uncommon to see the compiler crash on erroneous input\u2014instead of cleanly exiting with an error message\u2014or burn 100% CPU for several seconds before finally giving up on evaluating a seemingly trivial expression.\nHopefully, all that will be solved in the future, making Swift a first-choice language for all types of developments. In the meantime, I\u2019ll keep exploring the jungle for you.\nReferences\n\nSwift\u2019s official page\nAlgolia\u2019s Swift API Client\nAlgolia documentation\n\nNEW! InstantSearch Core for Swift: a high-level library to develop rich search experiences","record_index":7},{"post_id":6143,"post_title":"Algolia Vault - Bringing Physical & Digital Data Security to Search","post_date":1491879393,"post_date_formatted":"Apr 11th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vault\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Screen-Shot-2017-04-05-at-11.51.12-360x200.png","time_to_read":3,"content":"At Algolia, we take search seriously. And we take security even more seriously. Search is the means by which users access personal & often private data, and one of the reasons customers trust that data on Algolia\u2019s servers is because we have strong security for all of our customers - even our forever-free Community\u00a0plan used by thousands of websites today.\nGreat search & secure search don\u2019t have to be contradictory\nSome data is more sensitive than others. Last\u00a0year Algolia set out to provide an extra level of security for our customers for whom privacy concerns & secure data are core to their business (or a regulatory requirement of the industry they work in). We\u00a0recently began making that feature available to our Enterprise users upon request, and we're excited to introduce Algolia Vault.\nAlgolia Vault is a dual-faceted approach to making sure that the most sensitive data can't be compromised. Medical records for a digital healthcare service or personal user data used only internally should be searchable and secure. \nAlgolia Vault brings extra physical security by activating\u00a0encryption-at-rest, encrypting\u00a0your data even when it\u2019s not being used. By default, Algolia encrypts all data transfers, but adding encryption-at-rest means that your data is safe in the case of physical theft. \nLikewise, Algolia Vault brings extra digital security by allowing\u00a0customers\u00a0to create an IP Whitelist, preventing any access from non-authorized IPs. IP whitelisting means a tradeoff against access by your dedicated Solutions Engineer & Customer Success Manager; however, Algolia Vault users can limit access to select IPs to make sure their data isn't accessed by unwanted users.\nAlgolia Vault limits\u00a0implementations to back-end, meaning that your search speed becomes dependent on the speed of your servers (and not just ours). Of course, you\u2019re still benefiting from the Algolia engine\u2019s lightning fast search speed and our support team is there to work with","record_index":0},{"post_id":6143,"post_title":"Algolia Vault - Bringing Physical & Digital Data Security to Search","post_date":1491879393,"post_date_formatted":"Apr 11th 2017","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-vault\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/Screen-Shot-2017-04-05-at-11.51.12-360x200.png","time_to_read":3,"content":"\u2026 you to keep your end-users satisfied with the overall Search UX.\nWith Algolia Vault, we\u2019re one step closer to making search available to all product\u00a0builders, no matter the platform, language, or business needs they may have. Sensitive data shouldn't be punished with bad Search UX just because the data needs to be secure - financial institutions, governments, healthcare & providers\u00a0are among those the most affected in terms of end-user experience due to the requirement & expectation of data security. With Algolia Vault, along with our ongoing security certifications and requirements, we hope to bring great search UX to every product, app, website & service.\u00a0\u00a0\u00a0\nIf you\u2019re interested in trying out Algolia Vault today or learning more about how we prioritize security, you can check out some of our other security-related blog posts, or reach out to us directly on Twitter or by email.","record_index":1},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"On March 31st, we announced our CSS API Client, that replicated a search engine with only CSS. While it was only a joke in the spirit of April Fool's, it was a lot of fun to make, and also a lot of fun to see it in the wild.\nI've always been fascinated by how much could be done using only CSS (from flags to FPS games). When I started building the demo, I never imagined that I could build such a full-fledge search experience with CSS. The final demo has some huge drawbacks that make it unusable in production (the massive filesize of the generated CSS being the main one), but it still has some nice features that I'd like to shed some light on in this blogpost.\n\nThe power of selectors\nBy design, CSS does not have some of the features we expect from programming languages. It has no conditions, no loops, no functions, no regular expressions\u2026 Instead, it has selectors that can target elements in an HTML document and apply styling to them.\nUltimately, targeting elements is one of the greatest strengths of CSS, because it can select elements through a combination of tag names, classes, ids, and attributes. With the sibling selectors (,  and ), you can even go crazier in what you can target - this is a big part of how I manipulate CSS for creative CSS projects.\nThe search bar\nWhen it comes to searching, the first thing that comes to mind is the search bar. This is the ubiquitous element where you input your text, and results gets displayed. In HTML, a search bar is nothing more than an . It can thus be targeted by CSS.\nAn  element also has a  attribute. It means I can write a selector that will only match the input when it has a specific value:\n\n\nWhen the value of the input is equal to \"Tim\", I can change the background of the input to green. This might not seem much, but this is the cornerstone on which the whole CSS API Client is built. It lets me change the styling of elements based on the value of the .\nWe'll see how this will be used, but first I need to talk a bit","record_index":0},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 about JavaScript.\nJavaScript: the necessary evil\nI know I\u00a0said this CSS API Client didn't need any JavaScript, but that wasn't entirely true. I still need a tiny JavaScript statement for this to work.\nWhen you use an attribute selector in CSS, it uses the value of the attribute that is set in the HTML, at page load. And when you type something in an input, it only updates the \"virtual\" value of the input (the one you can read with JavaScript), not the one in the HTML. It means that by default, CSS does not catch changes in the value of an input.\nAs the whole demo is based on that fact, that's not very convenient.\nThat's why I need JavaScript. I want our  attribute to be updated whenever something is typed in the input. This is mandatory to get this as-you-type feeling of results instantly displayed.\n\nThis simple JavaScript statement can be read as: \"Whenever the value changes, set the value equal to the value\". It might seem obvious but is actually needed to pass the correct value from the JavaScript world to the CSS world.\nNow that I've taken care of that, let's see how to display the results.\nDisplaying results\nAt that point, I can apply some styling to the input based on its current value. It's a good start, but what I really want is to display results based on what is typed in the input.\nUsing the  sibling selector, I can target the  that is just following the  in the markup. And by using the  pseudo-selector along with the  declaration, I can add some generated content to it.\n\n\nWhenever I type Tim, Josh or Vincent, the next div will be populated with the full name.\nPrefix search\nThat works well. but a good search engine shouldn't have to make you type the full keyword before displaying results. I should see \"Tim Carry\" as a result as soon as I type \"Ti\", or even just \"T\".\nThat's when the major mental shift occurs:\n\nAs a user, when I am searching, I want the answer to the question: What results will I have if I type \"Tim\"?.\nBut here, as a builder, the question","record_index":1},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 becomes: What should I type to have \"Tim Carry\" as a result?.\n\nIn this example, the answer would be \"T\", \"Ti\", \"Tim\", but also \"C\", \"Ca\", \"Car\", \"Carr\" and \"Carry\" if I want to search in both first name and last name. Those variations are called n-grams, which means all the strings that would match a specific record.\nNot wanting to do this task by hand, I wrote a Ruby script that generates the n-grams for any word, and outputs the CSS selectors to a new file.\nDisplaying several results\nNow that I had it automated, I started feeding my script with the list of all Algolia employees, to see how it performed. That's when I realized that displaying only one result was too limiting.\nFor example, we have 5 people named \"Alex\" in the company (at some point, 20% of the company was actually named Alex), so I wanted to display all of them when searching for \"Alex\", not only the first one.\nI changed the HTML markup and instead of having only one  for the result, I created as many divs as the number of employees. I gave each div a unique incremental  (to select them more easily), and pre-filled each of them with the full name using the same  and  trick.\nThose s were all hidden by default, and only made visible when the value of the input was matching them.\n\n\nStyling\nNow that I had the full data-set to play with, I decided to give it a bit more styling, to make it look like the actual \"About Us\" page we have on the website. Because I'm only using one  per result, I had to resort to a few more CSS tricks.\nI added a picture for each of us as a background image. I used Cloudinary to make all images round, resized and grayscale. I added a fun fact on hover, using the  pseudo element and generated content.\nTo display both the name and job title, I used the special character  to add a new line in the generated content. I also added  so both lines were actually displayed separately. Then, using  I was able to style each line differently.\nEdge cases\nBy playing with an actual dataset, I","record_index":2},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 could start to see some edge-cases that I didn't expect at first.\nThe first one was handling case sensitivity. I wanted both \"Tim\" and \"tim\" to match the same results. Fortunately, CSS handles that directly without needing me to generate more n-grams.  will match \"tim\" in a case insensitive way. This works on all major browsers except Edge, unfortunately.\nAll the other challenges had to do with my co-worker, R\u00e9my-Christophe Schermesser. R\u00e9my-Christophe has a very interesting name when it comes to edge-cases.\nThe first is that his first name is actually composed of two names and I wanted to be able to find him either with \"R\u00e9my\" or \"Christophe\". In addition to splitting the name on spaces (to identify first name and last name), I also had to split them on dashes.\nThe second part was that his name actually uses an accented character (\u00e9), and I wanted to be able to find him either by typing \"Remy\" or \"R\u00e9my\". I had to normalize the names, and generate n-grams for the normalized version as well.\nFinally, his last name, \"Schermesser\", is difficult to spell correctly on the first try. I wanted people to be able to find him even if they wrote \"Shermesser\" (without a \"c\"), or \"Schermeser\" (with only one \"s\"). Typos are very common in search, especially on mobile (thanks to the fat-finger effect), so a good search engine should auto-fix them as much as it could.\nTo handle the typos for this specific word, I had to generate 40 more n-grams, one for each variation of the word with one letter removed (except for the first and last one). \"Ser\", \"Sherm\", \"Scermes\", \"Schrmesser\", etc were all valid new n-grams to find \"Schermesser\".\nThe funniest thing in this story is that, even in the office, almost no-one ever calls him R\u00e9my-Christophe. Everyone calls him \"RCS\", and I wanted to replicate this in the engine, through the use of synonyms. Adding a synonym is nothing more than manually adding a new set of n-grams that should match a specific result. In my case, \"r\", \"rc\" and","record_index":3},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 \"rcs\" were now also valid keywords to find R\u00e9my-Christophe.\nHighlighting\nAt that point, I could search for people either with their first name or last name, including typos, or through synonyms. I decided to also add finding them through their job title. This adds a lot of possibilities, and sometimes it was not obvious why a specific result was displayed.\nA good search engine should never let the user wonder why a result is displayed. It must be clear instantly why a result is relevant, and that's where highlighting comes into play.\nThe Algolia API handles highlighting by wrapping matching text with the  tag. But because I'm using generated content, I cannot add HTML into it, so I had to find another way.\nThe trick I used was to generate a custom font that is a merge of Raleway Regular and Raleway Bold. I kept the regular font as a basis, but added all the bold characters into the private UTF-8 space. To highlight a word, I replace its letters with their bolded version. CSS lets you use specific UTF-8 characters by using the  syntax.\n\n\nOrdering results\nAnother really important part of a search engine is that the most relevant results should be displayed first. My current setup wasn't taking relevance into account at all when displaying results.\nThis was obvious when searching for \"Co\". I had Nicolas and Julien, both Co-founders, displayed before ry Dobson. This was not what I wanted. Cory should have been the first result because the match was in his first name and that was more important.\nThe Algolia engine handles this by sorting results through a tie-breaking algorithm. We define a list of criterias, and we check each result against those criterias to sort them into buckets. I applied the same logic in my script; my first criterion was that a match in the name was more important than a match in the job title. The sorting should then create two buckets, one for a match in the name, another for a match in the job title.\nIf there were several results in a specific","record_index":4},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 bucket, it would apply the second criterion to this bucket to further split the results in more buckets.\nMy second criterion was the position of the match. A match in the first word was more important than a match in the last word. If it would still have ties, it would then sort by the last criterion, the custom ranking. Here I decided that results at that point should be displayed by order of arrival in the company.\nYou can see it in action with the query \"St\" for example:\n\n\nWe have a first bucket with the match on the name, and a second with match in the job title (Full-Stack Software Engineer)\nThe first bucket is then split into two more buckets, a match in the first name (Steven Merlino), or a match in the last name (Alexandre Stanislawski, David Strader)\nFor each bucket with more than one item, results are ordered by arrival date in the company (Alexandre joined before David, Vincent joined before Maxime, etc)\n\nNow that I know in which order items should be displayed, how do I do that in CSS? This one was actually much easier than anticipated. As soon as you start using flexbox-based display, you can apply an  to any element to change its position in the display.\n\nFaceting\nThe last feature to add was the filtering (or faceting, as we call it in the search industry). Faceting is extremely important when the number of results you get is very large, and thus hard to comprehend. It is important to let your users filter it down.\nI added a way for users to filter results by team. I used most of the same tricks (generated content, flexbox ordering, etc) that I already used on results. The one thing that is important here is that the facets are actually  elements, each linked to an invisible  button.\nIn HTML, you can link a  with an . Whenever you click on the label, the radio button will get selected, wherever it is in the page. I used that to my advantage, by linking each  to a different radio button that I put in the markup right before the main search bar.\nUsing","record_index":5},{"post_id":6102,"post_title":"How we built the real demo for our fake CSS API client","post_date":1491355101,"post_date_formatted":"Apr 5th 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/real-demo-fake-css-api-client\/","categories":["Community","News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/04\/css_aprilsfools_reveal-360x200.png","time_to_read":14,"content":"\u2026 the  pseudo-element and the adjacent sibling selector, it let me create a \"global state\" to my page, based on which  is currently checked.\nFiltering is applied here by hiding all the results that do not match the selected facet. If I select the team \"Engineering\", then I need to hide all results that are not in the \"Engineering\" team.\n\n\nPerformance\nThe generated filesize can get huge quickly, as I need to create new rules for each new n-gram. I used one-letter ids as much as I could as it both reduces the filesize, and makes the browser parsing of the CSS much faster. I then minified the file by grouping selectors sharing the same rules together. Still, the final file is 4Mb.\nI've also removed the  keyword (taking too many characters) by artificially boosting the specificity of my rules instead ( is more specific than  for example but will still target exactly the same elements).\nConclusion\nBy splitting a complex problem into small chunks, and using the strength of the language that was at my disposal, I managed to build the illusion of a search engine in CSS, which made for a great basis of some April Fool's fun. While the demo could never be applied in production - the file size is too big, the results cannot be selected nor clicked and it lacks many features a real search engine should give you - it was a great opportunity for me to stretch what\u2019s possible with CSS, and for us at Algolia to bring a smile to our community and a few new faces who discovered us from our prank. You can have a look at the source code on GitHub.\nSearch is all about speed, relevance and UX, and we've only scratched the surface of those three pillars here because we were limited by the CSS language; however, we're glad the prank & demo sparked some great conversations in the CSS community and we look forward to discussing more with you.","record_index":6},{"post_id":6057,"post_title":"Goodbye JavaScript: Introducing our CSS API Client","post_date":1490947227,"post_date_formatted":"Mar 31st 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/js-is-dead-all-hail-css\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/css_search-360x200.png","time_to_read":4,"content":"Editor\u2019s note: This blog post is an April Fool's faux product launch. However, to see how we actually built the real demo for 'launch', see our full explanation.\nAt Algolia, we are always pushing the boundaries of search. Today is an important day for us as it marks the official release of our 11th API Client: the CSS API Client.\nCSS is an awesome language. It only takes you a few years of practice to be able to style a minimalist website in a matter of days.\nLately, we\u2019ve seen more and more discussion about its place in regards to JavaScript. One side thinks that CSS and JavaScript have very different goals, and should be kept in separate parts of your code, to have a clearer separation of concerns. Others argue that one cannot live without the other and that CSS\u00a0should be inlined directly inside JavaScript.\nAt Algolia, we've decided to take a stance to stop this never-ending debate once and for all. We came to the conclusion that both sides were wrong, and that CSS was a language so powerful that you do not actually need any JavaScript.\nThat's right: we decided to get rid of JavaScript altogether.\nJavaScript: 1993-2017\n\nLook, JavaScript is an impressive language. We even considered rewriting our whole engine with it at some point. JavaScript is asynchronous, so it is fast by definition.\nUnfortunately, the language is not mature enough for us. A new version of it is released every year, which shows how unstable it is. On the other hand, CSS3 was released in 1998, without any new version since then. We think it sends a clear signal that CSS is the mature and stable tech.\nCSS: superiority through simplicity\nCSS also has none of the features that bloats other programming languages. It has no conditions, no loops, no functions and no regexps. It is pure. You can write concise selectors like  that clearly expresses your intent at first glance.\nWe\u2019ve exposed our entire search engine using just CSS, and you can try it live.\nSeriously, it actually exists. It\u2019s","record_index":0},{"post_id":6057,"post_title":"Goodbye JavaScript: Introducing our CSS API Client","post_date":1490947227,"post_date_formatted":"Mar 31st 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/js-is-dead-all-hail-css\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/css_search-360x200.png","time_to_read":4,"content":"\u2026 typo-tolerant, handles synonyms, highlighting, faceting & more.\nTry it live!\nA truly offline search for your web browser\nThe best part of the client is that it works offline. There are no actual requests done to our servers, everything is directly handled by your browser. It means that once you've download the initial CSS file, you can unplug and search away. That's what we call Offline Search!\nNo call to the API also means that you have unlimited operations. You can search all day long in the same 100 records, and it won't cost you a cent. Every request being handled by your browser means that you are now using your 8 cores at their maximum potential. We've actually started shutting down 2\/3 of our datacenters because we are anticipating a much lower load thanks to this CSS release.\nOne API Client to rule them all\nYou know the saying: \"Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away\". So, we removed a lot of features.\nYou won't need data instantly replicated all around the globe to reduce latency because the data now lives in your browser; you can't get any closer than that.\nAt Algolia, we also take great pride in having extensive documentation and code examples for all our API clients. We'll remove those other clients, obviously, because we know you won't need any other API client than the CSS one.\nWe are so confident in the quality of this release, that it will work flawlessly, everywhere, every time, and that there will never be any bugs, that we decided it\u2019s no longer necessary to offer support for it. This shows how committed we are.\nFinally, because there are no calls to the API, we decided to remove Analytics and the 99.999% SLA from our services. It was a hard decision to make, but when we weighed the pros and cons, it was clear that we needed to remove all the features that didn't bring any value. Instead, we added support for flexbox.\nNext Steps: CSS Everywhere\nThis whole experiment opened","record_index":1},{"post_id":6057,"post_title":"Goodbye JavaScript: Introducing our CSS API Client","post_date":1490947227,"post_date_formatted":"Mar 31st 2017","author_name":"Tim Carry","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/23b877d87b8a3d381a63601e8b5b1ec0?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/js-is-dead-all-hail-css\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/css_search-360x200.png","time_to_read":4,"content":"\u2026 our eyes on a brand new world. We've already starting working on our new Machine Learning processing pipeline in CSS. Stay tuned!\nTry our new CSS API Client live","record_index":2},{"post_id":6072,"post_title":"InstantSearch Android: Optimized Search Components for Android","post_date":1490669714,"post_date_formatted":"Mar 28th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-android\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/android-instantsearch-illustration-360x200.jpg","time_to_read":7,"content":"We\u2019re excited today to be releasing InstantSearch Android, \u00a0a library of Views and helpers to help you build the best instant search experience on Android. It is built on top of Algolia's Android API Client to give Android developers a powerful solution to quickly build the right search experience to delight their users.\n\nBefore we dive into how InstantSearch works, I\u2019d like to talk about why we invested so heavily in the Android developer ecosystem.\nImplementing Search on Android today: not for the faint of heart\nCreating great search for your Android app is hard. Search is a complex problem in general, but, for an Android developer, adding search to their application comes with unique challenges: some are common to all mobile app developers, but the Android platform brings a few hurdles of its own.\nLimited Resources\nThe first source of difficulty comes from the constrained resources of mobile devices themselves. Because many apps are fighting for a limited amount of RAM, you won\u2019t be able to keep all your data in memory. You cannot afford to put thousands of products in your user\u2019s RAM if only a few are relevant to them, and you can\u2019t store them in the device\u2019s storage either, as it suffers from the same issue: tens if not hundreds of apps are squeezed in the small hard drive of a mobile device. This limitation hinders your ability to store data on your user\u2019s device, which makes it more impractical to build a local search solution.\nEven if you don\u2019t have a large dataset, you won\u2019t be able to implement all the advanced features that a powerful search needs (typo-tolerance, synonyms handling, ...). With the CPU constraints of a mobile platform, most complex processing logics cannot be implemented locally if you build from scratch - at least not easily or quickly.\nNetwork Stability\nThe previous performance limitations are easily removed if you use a hosted search engine, as you can offload space and computational resources to your","record_index":0},{"post_id":6072,"post_title":"InstantSearch Android: Optimized Search Components for Android","post_date":1490669714,"post_date_formatted":"Mar 28th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-android\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/android-instantsearch-illustration-360x200.jpg","time_to_read":7,"content":"\u2026 provider.\nHowever, mobile devices are used in a specific context: your users move from a network to another, connect and disconnect to Wi-Fi networks, lose connectivity when passing under a bridge. This mobility context brings two difficulties to mobile app developers.\nWhen the user moves around or simply has bad connectivity, their device suffers long latency. The network requests can suddenly take a very long time to resolve.\nMoreover, even when the connectivity is good, the network bandwidth can still be pretty small. Although a request can be sent quickly, the response\u2019s payload can take a long time to load.\nThese two network effects will have a negative impact on your app\u2019s UX if you don\u2019t consider them when developing.\nReal Estate & Screen Constraints\nThe main challenge of designing great mobile apps comes from the available screen space of the devices. At best, a tablet will offer around 10 inches of screen space for your apps, but most mobile devices will only have a few inches of screen space (on average 4.5\u201d.)\nThis small space for your application\u2019s interface will limit what you can display on the screen at any given time, which makes developing complex interfaces like a search UI even harder on mobile.\nIt is even harder if you consider the \u201cFat Finger Effect\u201d: because it is hard to tap precisely with a touch-screen, small buttons are hard to action and nearby commands are easily mistakenly clicked. This brings another layer of complexity when designing your interface.\nAndroid developers also have to address the platform fragmentation. You want your app to display nicely on a variety of devices, each with different dimensions and resolution. It complicates further the testing of your interface and the development of new UI components.\nLack of framework for search interfaces on Android\nOne of the more peculiar parts of building on top of the Android platform is that there are no building blocks for creating beautiful search experiences on","record_index":1},{"post_id":6072,"post_title":"InstantSearch Android: Optimized Search Components for Android","post_date":1490669714,"post_date_formatted":"Mar 28th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-android\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/android-instantsearch-illustration-360x200.jpg","time_to_read":7,"content":"\u2026 Android.\nThe Search Overview Guide will tell you where you should perform the search, but not how:\nThe process of storing and searching your data is unique to your application. You can store and search your data in many ways, but this guide does not show you how to store your data and search it.\nThe only practical solution that this guide tips at is using a full-text search algorithm if your data is stored in a local SQLite database. But what if your data is not stored like this, or if you want more powerful search capabilities like typo-tolerance, prefix search, or query processing?\nDeveloping a great search interface for an Android application is therefore difficult. You will face harsh performance and design constraints, user expectations of advanced functionality from your search engine, and your design needs to work across the various screen sizes and network conditions that encompass today\u2019s mobile ecosystem.\n \nAlgolia: Improving the Android developer experience\nAt Algolia, we seek to provide every developer with the best tool for their job.\nIn 2015 we brought such a tool to web developers with instantsearch.js: a widget-based library that abstracts the complexity of building a search interface and powering it with Algolia.\nInstantsearch.js fulfilled a real need, as the community reaction has quickly shown: today the project is used by 1530 projects and has almost 1,500 stars on GitHub!\nThis works great for web developers, but Android developers need their own native solution.\nIntroducing InstantSearch Android\nLet\u2019s dive in and see what InstantSearch Android does for developers.\nWith the Algolia engine under the hood, your app can leverage powerful features like typo-tolerance or geosearch, without bearing the CPU and storage costs that would come with an embedded search engine. We\u2019ve also got you covered for RAM consumption: InstantSearch Android has been optimized so your app can run seamlessly on any Android device, using various memory","record_index":2},{"post_id":6072,"post_title":"InstantSearch Android: Optimized Search Components for Android","post_date":1490669714,"post_date_formatted":"Mar 28th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-android\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/android-instantsearch-illustration-360x200.jpg","time_to_read":7,"content":"\u2026 optimization techniques like View recycling or LRU Caches when relevant. Your users will be able to enjoy your app\u2019s great UX without sacrificing their precious RAM!\nPrepared for adverse Network\nWith the hundreds of mobile applications using Algolia, we have had our fair share of network issues and limitations to work with. We leveraged this experience while developing InstantSearch Android, which includes the most useful tools for handling bad latency or low bandwidth easily: out-of-the-box progress indicators to let your user know that you did process their query, customizable caches to avoid sending queries when it is useless, and parameters to ensure the server\u2019s response only contains the data that you need.\nDesigned for Mobile\nWe\u2019ve distilled down our experience building search into mobile apps to bring you optimized components for building your own search experience - if you\u2019d like to learn more about Mobile Search Patterns, you can check out this talk we gave earlier this year at Mobile World Congress.\nWe released InstantSearch Android with a set of Widgets: UI components to help you build faster your search interface. A widget is based on an Android View, made aware of the surrounding search interface and configured to follow the best practices of mobile search.\nEach widget follows the recommendations that are common to all mobile apps so you don\u2019t lose time, while leaving you the room for customizing its look and feel for your users\u2019 exact needs.\nYou can enable autofocus on a SearchBox in one line, or make it so that scrolling the Hits can automatically close the keyboard: we want every pattern that is useful to app developers to be a no-brainer with InstantSearch Android, so you don\u2019t waste time on mundane things and can dedicate your time to what makes your app awesome.\nThis toolbox is now open-sourced and available on GitHub: you can start using it today to build great search interfaces in no time!\nTo give you an idea of what you will be","record_index":3},{"post_id":6072,"post_title":"InstantSearch Android: Optimized Search Components for Android","post_date":1490669714,"post_date_formatted":"Mar 28th 2017","author_name":"Paul-Louis Nech","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e62ff7c92ea53758de27616b9a4b9ab?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/instantsearch-android\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/android-instantsearch-illustration-360x200.jpg","time_to_read":7,"content":"\u2026 able to build with InstantSearch Android, we created two example apps based on some well-known search interfaces : the Media app will show you how to build a video search interface while the Ecommerce app shows a more \u00a0complex e-commerce search interface.\n\n \n ","record_index":4},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"Before comparing Algolia to Elasticsearch, it\u2019s good to understand a few things about the nature of search.\nSearch architecture is unique\nThe type and quality of search experience you can deliver depends heavily on your choice of search engine, hardware, datacenter region and front-end web and mobile development frameworks. It\u2019s important to make the right choice for each part of the stack but it\u2019s equally important to make a set of choices that work together as a whole. Because search user experience goals are so demanding, a vertically-integrated approach to architecture is more important than for other types of applications. Latency, for example, is not only a function of the search engine but of every step between the search user interface and backend infrastructure.\nSearch is mission critical\nSearch is one of the hardest features to get right, both because users benchmark search experiences against Google & Amazon, and because search is a balance of multiple disciplines, not limited to UX, relevance tuning\u00a0and performance optimization. Development teams often delay building search because of a lack of confidence in getting it right and the fear that it will take longer than expected. Yet search is often the most mission critical feature\u2014the quality of an application\u2019s search has a big influence on the perception of application\u2019s overall quality. In domains like e-commerce, introducing a search bug can cost millions of dollars.\nThe combination of these factors make search one of the riskiest areas of development for business and consumer applications. When comparing different ways of delivering a solution, like Algolia and Elasticsearch, we want to look at how each approach specifically addresses the full, end-to-end set of risks. In this comparison, we will look not only at the search engine but the full search architecture, starting with end-to-end latency.\n\nMission-critical search for a global user base\nThere are many different types of","record_index":0},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"\u2026 search applications. To focus the comparison of Algolia and Elasticsearch, we want to hone in on a specific family of use cases which we refer to consumer-grade search. Consumer-grade search is the type of experience delivered by companies like Google, Amazon and Facebook to billions of people worldwide. It connects people with products, content and key pieces of structured data. It is fast, reliable, works on multiple platforms and the results are highly relevant.\nThe search tolerates misspellings, alternate phrasings or user mistakes. Relevance is not caveat emptor, it\u2019s caveat venditor - the search must adapt to the user, not the other way around. Consumers have high expectations of relevance but equally demanding expectations for the user interface. They expect an effortless, multi-faceted search and browsing experience, the kind pioneered by sites like Amazon.\nConsumer-grade search doesn\u2019t just apply to consumer-facing applications. Today\u2019s business application users have become just as demanding, in part because many business applications are now distributed in app stores and compete directly with consumer versions.\nThe expectations of the average user can seem unattainably high, but this is why Algolia exists. Algolia is laser-focused on helping customers meet the perfectly unreasonable search expectations of the average Internet user.\nAbout Elasticsearch\nAs a search engine that also functions as a scalable NoSQL database, Elasticsearch accommodates many different types of applications while not being opinionated toward one specific case. Elasticsearch is used for search but also log processing, real-time analytics, running map-reduce and other distributed algorithms, and even as an application\u2019s primary database. The breadth of Elasticsearch is impressive and it does things that Algolia is not well suited for - streaming logs, map\/reduce querying, complex aggregations and operating on billions of documents at a time. Algolia itself has used","record_index":1},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"\u2026 Elasticsearch internally for tasks like storing logs and computing rollups.\nIn this comparison, however, we are focusing on consumer-grade search. This is the most common situation we are asked to compare. Building a consumer-grade search application with Elasticsearch requires a nontrivial amount of backend and front-end software engineering. There are many more steps than just provisioning and operating an Elasticsearch cluster.\nIn this series we\u2019ll dive into what some of those steps are; however, you can already take a look at how Algolia solves for these steps in our Inside the Engine series. In it we explore implementation details like I\/O optimization, query tokenization, multi-attribute relevance, highlighting and synonym handling. These are features that must be accounted for in any search project, including those with Elasticsearch at the core.\nEnd-to-end latency budget\nThe first feature of search is speed. Whole-transaction latency, from keystroke to visible search result, is what forms the user\u2019s first impression of a search. A search application architect needs to have this in mind from the beginning, as a huge number of factors can affect the end-to-end latency.\nTo make things more difficult, for consumer-grade search the upper bound on satisfactory end-to-end latency is very, very low. Most consumer search experiences, including Google, Facebook and those of Algolia customers, deliver new results with every keystroke. This type of experience, known as instant search, is loved by users for its interactive feel but it only works if search results can be returned in the blink of an eye. Less, even: a human eye blink takes 300-400 milliseconds. An instant search starts to feel laggy at only 100 milliseconds.\nFor as-you-type search to be as satisfying as possible, Algolia recommends the end-to-end latency be no more than 50ms. This is the speed at which search feels truly real-time, where the user feels in full control of the experience. Under these","record_index":2},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"\u2026 conditions, users are likely to keep reformulating their query until they find what they\u2019re looking for, rather than abandon or bounce.\nIf you\u2019re using Elasticsearch or Algolia to power an as-you-type search, these are the important numbers to keep in mind as you design your architecture. It is possible to consistently reach these numbers if you know 1) where latency is likely to accumulate and 2) how to reduce or eliminate it.\nThat\u2019s what we\u2019ll look at in the following side-by-side table: how Algolia reduces latency in each layer of the stack, where latency can accumulate inside of Elasticsearch, and what work can be done inside or on top of an Elasticsearch implementation to reduce the risk of added latency.\n\n\n\n\n\nAlgolia\n\n\nElasticsearch\n\n\n\nGlobal User Base\nLow device-to-datacenter latency requires infrastructure in multiple regions.\nTip: add 1-2ms for every 124 miles of distance over fiber.\n \nAutomatically replicate indices to any of 15 regions throughout the world using our\u00a0Distributed Search Network (DSN).\n \nIt is possible to cluster Elasticsearch across multiple data centers, but not recommended. The recommended solutions involve replicating manually via a messaging queue to clusters that are not aware of each other.\n\n\nRAM\nIf a query's data is not all in RAM, it may have to load data from the much slower disk.\n \nAlgolia indices are stored in RAM (256GB or more) and memory mapped to the nginx process. No\u00a0pre-loading (warming) is required to get great performance for the first query.\n \nThe ES cluster must have enough RAM and be properly tuned to make sure large indices stay in memory.\u00a0If you are also supporting an analytics workload, you risk large analytics queries evicting data for searches.\n\n\nVirtualization\nIn sharing hosting environments like AWS, performance can fluctuate because of contention with other customers.\n \nAlgolia runs on bare metal hardware with high-frequency 3.5GHz-3.9Ghz Intel Xeon E5\u20131650v3 or v4 CPUs.","record_index":3},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"\u2026 Clock speed is directly related to search speed.\n \nElasticsearch can be deployed on bare metal and optimized hardware, but at a premium cost compared to AWS or cloud-based solutions.\n\n\nSorting\nBefore results are presented to the user, they have to be put in the right order.\n \nAlgolia presorts results at indexing time according to the relevance formula and custom ranking. There is a minimal sorting step at the end to account for dynamic criteria like the number typos and proximity of words.\n \nSorting is done at the end of each query. Depending on the number of results to be sorted, this can impact latency.\n\n\nRelevance\nSpeed is often traded off to get better relevance.\n \nTokenization required for partial word matching and typo tolerance is done mostly at indexing time.\n \nAdvanced techniques like ngrams, shingles and fuzzy matching make indices larger and also require analysis at query time.\n\n\nDNS\nDNS can be slow before it's cached by the user's device. If a DNS provider is under DDOS, requests will be slow or fail to complete.\n \nAlgolia uses two DNS providers to increase reliability. Logic to fallback from one to the other is part of all API clients.\n \nElasticsearch does not provide out-of-the-box support for redundant DNS, but you could build\u00a0it yourself.\n\n\nLoad Balancing\nLoad balancing & coordination can cause network congestion and add latency.\n \nAlgolia API clients connect directly to the server with data on it. There is no network hop or single point of failure for reaching a cluster.\n \nAn ES cluster needs the right ratio of data nodes and coordinating nodes to avoid adding latency.\u00a010G network bandwidth is recommended for large\u00a0clusters.\n\n\nGarbage Collection\nApplications running in the JVM require momentary pauses to free up used memory. During these pauses, requests are queued.\n \nThe Algolia engine is written in C++, it does not use the JVM.\n \nThe JVM can be tuned to reduce the frequency and impact","record_index":4},{"post_id":6029,"post_title":"Comparing Algolia and Elasticsearch For Consumer-Grade Search Part 1: End-to-end Latency","post_date":1490197954,"post_date_formatted":"Mar 22nd 2017","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-v-elasticsearch-latency\/","categories":["Infrastructure","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/screen_shot_2017-03-21_at_19.37.12_1024-360x200.png","time_to_read":11,"content":"\u2026 of GC pauses. The tuning depends on the workload and server resources available. This is a painstaking process about which much has been written.\n\n\nSharding\nSharding allows data to be scaled across multiple indices. Overloaded shards exhibit degraded performance.\n \nAlgolia handles any required sharding behind the scenes, it is invisible to the user. Shards can be dynamically rebalanced to avoid hot spots.\n \nIf original shard assumptions are wrong, such as the choice of a shard key, an Elasticsearch cluster will have to be rebuilt or rebalanced down the road to alleviate performance hotspots.\n\n\nHeavy Indexing\nLarge indexing operations can negatively impacts search performance because they compete for the same CPU and memory.\n \nAlgolia splits search and indexing into separate\u00a0processes with\u00a0different CPU priorities.\n \nThe Elasticsearch cluster must be configured to use different nodes for searching and indexing.\n\n\n\nConclusion\nLatency can creep in from any number of places. Great care needs to be taken at each layer of the stack to avoid exceeding the latency budget and causing users to abandon. Algolia\u2019s hosted search approach means that we can give\u00a0our customers the benefit of our expertise in reducing latency.\u00a0For users of Elasticsearch, latency needs to be understood\u00a0and addressed by the implementing engineering team.\nRead other parts of the\u00a0Comparing Algolia and Elasticsearch for Consumer-Grade Search\u00a0series:\nPart 1 - End-to-end Latency\nPart 2 - Relevance Isn't Luck\nPart 3 - Developer Experience and UX\n\n\nWant to learn more?\nJoin us for our next workshop to learn how you can achieve fast and relevant results with your search implementation.\nWorkshop: Achieving Speed and Relevance in Search\n Sign Up\n ","record_index":5},{"post_id":5986,"post_title":"A search inside a search: how we brought Inception to Algolia","post_date":1489389180,"post_date_formatted":"Mar 13th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-for-facet-values\/","categories":["News","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/search-for-facets-360x200.png","time_to_read":6,"content":"Today, we are\u00a0thrilled to announce\u00a0search for facet values, a new feature to give end-users full control to explore and actually use those hidden values, enabling you to deliver on your promise to connect your users with what matters most to them.\nFacets are a key feature for a search engine, because they serve as\u00a0a way to filter data by a specific value on an attribute - a good example of a facet is \u201cbrands\u201d in an e-commerce website. With Algolia, each set of results for a search query contains the most common occurrences for each facet. This means if you have many values, some will be hidden. The simple solution to this problem is to fetch more values, but it may not be enough and the user would need to look for the value manually. Our previous solution for this problem was to recommend developers create a new index for each facet they wanted to make searchable -\u00a0which was a process that created overhead.\nIs faceted search still not clicking\u00a0with you? Let's dig further into the e-commerce example by assuming that\u00a0you\u2019re looking for a new pair of khaki pants from your favorite indie brand. If the brand isn\u2019t well known, then even if you know that a store carries this brand, it may not appear in the first results or even in the list of brands among the facets if it\u2019s not in the top ten brands of this site. You would have to filter other parameters in order for the results to be more focused on the brand you want. And that\u2019s only if you are persistent, because we all know that Googling the website name & the name of the brand will give you some results - that\u2019s where search for facet values comes in. It lets you filter the brands in order to find the one you want, from the very beginning of the search.\nSee it in action: redoing TED search\nTo demonstrate this new feature we built a brand new demo based on TED talks. In this search, we have set filters based on events, the kind of conference (original, TEDx, and so on), topics and languages.","record_index":0},{"post_id":5986,"post_title":"A search inside a search: how we brought Inception to Algolia","post_date":1489389180,"post_date_formatted":"Mar 13th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-for-facet-values\/","categories":["News","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/search-for-facets-360x200.png","time_to_read":6,"content":"\u2026 Each of those filters contains more entries than we can display which makes them perfect for using search for facet values.\nTry it out!\n\nBehind the scenes\nUntil the addition of Algolia's search for faceted values, implementing this feature meant extra work preprocessing the data. For each faceted attribute, you had to build (and maintain) a dedicated index with the occurrences of the elements. This index had to be kept up to date, as records were added, deleted or updated.\nBut not anymore! Our new functionality only requires two elements:\n\nConfigure the facet to be searchable: searchable(attribute) instead of attribute in the attributesForFacetting\n\n\nUse the new method of the API to actually search in the values\n\nThis new feature added to our API is available in all our API clients using the method called searchForFacetValues(), as well as in the JS Helper. That method accepts a specific query to specify the search and the whole set of filters that you can send to index queries. This allows the search for facet values to be contextualized. \u00a0This filters the values to only those that make sense for the current query.\nFor example, in an ecommerce website for which we have two faceted attributes: type of good and color. If you search for \u201capples,\u201d you don\u2019t want purple to be proposed because there are no purple apples (not that I know of, at least).\nBaked in our widgets\nA feature is only as valid as its ease-of-use. Since we provide an end-to-end developer experience from the REST API to the UI, we wanted to implement the searchable refinement list right in our widget libraries instantseach.js and react-instantsearch.\nWhen I first I looked at the LinkedIn implementation, something didn\u2019t seem quite right. It felt as if the feature was hidden from the users and the space could be used more efficiently. We prototyped two versions of the search refinement list and we ended up with two simple implementations that we tested internally:\n\n\n\n\u00a0\n\n\n\n\nWe then conducted a","record_index":1},{"post_id":5986,"post_title":"A search inside a search: how we brought Inception to Algolia","post_date":1489389180,"post_date_formatted":"Mar 13th 2017","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-for-facet-values\/","categories":["News","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/search-for-facets-360x200.png","time_to_read":6,"content":"\u2026 study with those two prototypes, and it turned out that for us the second version was the best. It was especially true for those who had no prior knowledge of the LinkedIn implementation.\nThis implementation is now available to you directly in instantsearch.js and in react-instantsearch. It is not a new widget but just a new option to pass to the refinement list which will transform the list into a search one.\nGoing further with search for facet values\nSearch for facet values is a great feature and with our current implementation baked into instantsearch libraries (vanilla js or react) we take advantage of it with few tweaks. But the truth is that we are only scratching the surface. This feature actually opens the door to implementation like smart search bars (like you see in Slack) that let the user refine facets within the search box.\nIf you\u2019re wondering how to get started, we\u2019ve updated our guides (because we <3 guides):\n\nLearn how to use this new feature in our Instantsearch result page guide\nAnd more in detail in the filtering and faceting guide\n\nLet us know what cool things you\u2019ve built in the Algolia community forum.","record_index":2},{"post_id":5947,"post_title":"Algolia: Picking up where Google Site Search left off","post_date":1488467653,"post_date_formatted":"Mar 2nd 2017","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/google-site-search-alternative\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/Gss-Copy-360x200.png","time_to_read":6,"content":"Recently Google has decided to sunset Google Site Search (GSS), their paid service for powering search inside of websites. The service will cease on April 1st 2018 and stop active development on April 1st 2017.\nAlgolia helps product teams create lightning fast, relevant search, and since the announcement we\u2019ve gotten a lot of questions and interest around how Algolia could serve as a replacement to GSS.\nIf you don't have time to read this post, join us for our next webinar and learn more about the benefits of moving from GSS to Algolia...\nWebinar: Comparing Google Site Search and Algolia\n Sign Up\n \nA jack-of-all-trades search solution\nFor website owners, GSS was a jack-of-all-trades solution, notoriously easy to set up and as powerful as you\u2019d expect a Google search product to be. From website crawling to the actual search UI, everything was handled automatically - all you needed was a small snippet of code.\nGSS will be dearly missed, if only for the fact that no technical skills were required to go live with a powerful search.\nWe\u2019ll talk a little more later about how Algolia compares (spoiler: we require developers 99% of the time), but let\u2019s first look at what GSS did & didn\u2019t do.\nKeeping your search up to date\nBecause GSS was using Google\u2019s default crawling mechanism, you couldn\u2019t control the refresh rate of your pages. As a result, it could take several days for your search results to reflect the latest content on your website, and up to 24 hours when using their (paid) on-demand indexing API.\nIn contrast, Algolia provides you with an API you can use to push & update your content in realtime. That API is available in many different ways through several\u00a0libraries and API clients. For e-commerce platforms (Magento, Shopify, etc.) & Content Management Systems (WordPress), we provide out-of-the-box plugins which include automatic indexing & updating requiring no additional software development.\nRelevance Fine-tuning\nBy design, GSS","record_index":0},{"post_id":5947,"post_title":"Algolia: Picking up where Google Site Search left off","post_date":1488467653,"post_date_formatted":"Mar 2nd 2017","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/google-site-search-alternative\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/Gss-Copy-360x200.png","time_to_read":6,"content":"\u2026 didn\u2019t provide a lot of control over relevancy in search results. The product started under the assumption all websites want to prioritize the same relevance as Google.com does, and therefore provided a single algorithm for every website. Unfortunately, it\u2019s rare enough to see two eCommerce websites with the same relevancy algorithm, let alone every website on the web. GSS users could tune search relevancy in three ways: using keywords, weighted labels, and scores. But those knobs and dials were not ideal for improving\u00a0your global site rankings - in fact you could negatively impact global results relevancy while trying to fix a specific query's results.\nAlgolia has a unique way to combine textual & business relevancy, making sure your users always find the right content. In Algolia you have complete control over what we call the \u201cranking formula\u201d and you can influence more \u201ctraditional\u201d things like \u201ctypo tolerance\u201d or \u201cword proximity,\u201d but also custom criteria, such as your product popularity in the case of an e-commerce company or the number of comments if you were indexing article content.\nUpdating your design\nGSS has several built-in search page layouts and lets you select the one you\u2019d like to have appear on your website. Most of the time, you want to go further than a simple integration: you want to have the same UI & UX patterns on all your website pages, including the search results page. GSS unfortunately only provides an API to fetch the results, you need to handle the full UI rendering on your end, from scratch.\nAlgolia provides you with powerful frontend libraries (like instantsearch.js, react-instantsearch or autocomplete.js) to help you build a beautiful & effective search results page, query suggestions, or dropdown menu in no time. Our goal is to reduce the time it takes you to implement incredibly powerful and beautiful looking search. We take User Experience very seriously, whether you\u2019re a developer, product","record_index":1},{"post_id":5947,"post_title":"Algolia: Picking up where Google Site Search left off","post_date":1488467653,"post_date_formatted":"Mar 2nd 2017","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/google-site-search-alternative\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/03\/Gss-Copy-360x200.png","time_to_read":6,"content":"\u2026 manager or website owner.\nSearch is a Conversation - Autocomplete and Instant Search\nBoth GSS & Algolia enable autocompletion, helping your users to find the right content as they type. GSS's autocomplete feature was very simple: rendering a list of query suggestions as you type.\nWith Algolia you can build a rich dropdown menu displaying query suggestions but also include actual search results. Algolia goes even further and provides search results in real time with each keystroke, so search feels like a conversation & not an order request).\nNavigation & Refinement\nGSS provides you with a simple way to categorize your content and refine your end-users search through those categories. To define your categories, you could either add them through your page's meta-data or by defining an annotation file. Unfortunately, this provides you limited refinement capabilities and makes it almost impossible to create a flexible search experience for all your users.\nTo solve this, Algolia provides you with a full-featured faceted-search experience, letting you define all the dimensions you want to use for the refinements. Building an Amazon-like refinements sidebar takes a matter of minutes.\nIn conclusion\nAlgolia can be a great option for GSS users to consider if you're serious about search; However it all comes down to what you want to do with Search. If Search is core to your user experience, or you believe it could be with a better search experience, putting in a little extra effort and a few lines of code may be worth it. Outside of our one-click integrations with major content & ecommerce platforms, our team can help guide you through the transition and the choices to make. Sign up below to learn more.\nWant to learn more?\nJoin us for our next webinar and learn more about the benefits of moving from GSS to Algolia.\nWebinar: Comparing Google Site Search and Algolia\n Sign Up","record_index":2},{"post_id":5933,"post_title":"With Algolia Offline, Take your Search Bar Everywhere.","post_date":1486365612,"post_date_formatted":"Feb 6th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/with-algolia-offline-take-your-search-bar-everywhere\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/offline-360x200.png","time_to_read":7,"content":"Mobile apps are everywhere, occupying an ever-growing part of our daily lives. Ironically, this pervasiveness has taken apps to areas where network coverage is still not that great: public transport, underground areas, big malls\u2026 or outside big cities. With offline usability becoming an increasingly shared concern among users and developers alike\u2014and justly so\u2014, we at Algolia decided to work at improving both the online and offline search experience. Today we\u2019re proud to introduce Algolia Offline, the easiest way to add offline search to your mobile app.\nEven a cool mobile application can quickly become frustrating without decent network connectivity. How many times have you been stuck staring at a spinning wheel while in the subway? Even when coverage is available, the latency inherent to mobile networks can make responsiveness an issue. An application that runs smooth as silk on Wi-Fi can inspire a ragged feeling when connected to 3G\u2014or even 4G.\nWouldn\u2019t it be great if apps kept working, whatever the network conditions? Well, that\u2019s definitely possible. Let\u2019s see how!\nBeyond caching and prefetching\nApps traditionally approach offline usage through two basic techniques: caching and prefetching.\nCaching means saving content locally when the user accesses it online, so that it remains available while offline, or just loads faster, without the need for network access. A cache is usually limited in both size and lifespan. This is typically how your web browser handles images, for example.\nPrefetching involves guessing what content the user might access later, and pre-downloading it so that it is readily available when needed. For example, a music app could download the next song in a playlist, or a news app could download today\u2019s edition. Sometimes, the user explicitly chooses the content to synchronize\u2014in Google Drive, for example, you can choose to pin documents for offline use; music apps typically sync favorite artists or albums.\nThose","record_index":0},{"post_id":5933,"post_title":"With Algolia Offline, Take your Search Bar Everywhere.","post_date":1486365612,"post_date_formatted":"Feb 6th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/with-algolia-offline-take-your-search-bar-everywhere\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/offline-360x200.png","time_to_read":7,"content":"\u2026 techniques work well for direct access to content\u2014you know which album you want to listen to, or which document you want to read\u2014but what if directly accessing the content is too tedious? For example, you might not remember which folder the document is in. Or what if you don\u2019t know exactly what you are looking for? For example, when browsing through your previous notes to try and recapture this great idea you had.\nThis is when you need offline search. Search makes content accessible, and is key to a great user experience\u2014this is as true online as offline.\nThe difficulties of offline search\nSolving the offline search problem isn\u2019t easy.\nPowerful search technologies are usually online only, while embedded technologies (like SQLite\u2019s FTS extension) don\u2019t provide the same experience for users (e.g. typo tolerance). And in any case, you need to code twice, once for online search, and a second time for offline search, with two different technologies, using different paradigms.\nToday, great user experience comes at the cost of a rather painful (and costly) developer experience\u2026 Luckily, solving developer pains is Algolia\u2019s raison d\u2019\u00eatre.\nAlgolia Offline: It\u2019s like Algolia, but offline\nQuite simply, Algolia Offline is a full Algolia search engine embedded into your mobile app, directly on your users\u2019 devices. We cross-compiled the same code that powers our SaaS product to iOS and Android, magically shrunk to fit in around 1MB (per CPU architecture), and made it available to all native applications. (\u201cHybrid\u201d JavaScript-based applications are not yet supported by Algolia Offline.)\nAs you may know, Algolia started as an offline SDK, before we pivoted to a hosted cloud offering\u2014this is a return to our roots.\nAll Algolia\u2019s great features are now available without any network: typo tolerance, filtering, faceting, geosearch\u2026 \nThere are a few notable exceptions: in order to reduce data size, we have disabled some features, like plural","record_index":1},{"post_id":5933,"post_title":"With Algolia Offline, Take your Search Bar Everywhere.","post_date":1486365612,"post_date_formatted":"Feb 6th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/with-algolia-offline-take-your-search-bar-everywhere\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/offline-360x200.png","time_to_read":7,"content":"\u2026 dictionaries (which would incur over 48 MB of overhead!), or segmentation for Asian languages (5\u00a0MB overhead). You still get the core of Algolia\u2019s magic: powerful textual and business relevance combined with unparalleled speed.\nSeamless integration with our existing Mobile SDKs\nAlgolia Offline synchronizes your online indices (or a part of them\u2014you decide) to your user\u2019s device. Once they are synced, they can be queried just like online indices, using the same API Client that you already use to search online. In fact, Algolia will seamlessly transition from online to offline based on the responsiveness of the network. Alternatively, you can choose to explicitly search offline, which can be useful to build fast autocompletion, as we\u2019ll see in the example below.\nOur approach was centered around the idea that the developer should always be in control: \n\nYou can choose what to sync: which indices, how many objects, which ones are the most relevant. \nYou can also choose when to sync: how frequently, or based on the current network conditions (e.g. only when connected to Wi-Fi)...\nYou control both storage and bandwidth usage\u2014keeping your app in line with the recommended best practices.\n\nIt\u2019s Easy. It\u2019s Fast. What else do you need?\nThe first obvious benefit is that you just need to code once, with a consistent API. It saves time, and also saves the sweat of duct-taping together heterogeneous search technologies.\nSecondly, all the synchronization happens in the background: your UI responsiveness is not impacted (did we mention milliseconds matter?). You can even search during a sync and still get consistent results.\nFor users who want to search content that isn\u2019t available online (by design or by requirement), Algolia Offline lets you create indices directly on the user\u2019s device, without syncing with an online index. This is useful if you have user-generated content that is input on the phone.\nAlgolia Offline in action\nA good use case to start with is","record_index":2},{"post_id":5933,"post_title":"With Algolia Offline, Take your Search Bar Everywhere.","post_date":1486365612,"post_date_formatted":"Feb 6th 2017","author_name":"Cl\u00e9ment Le Provost","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/a1333c0aa8991ac198a4ec5b72fffa95?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/with-algolia-offline-take-your-search-bar-everywhere\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/offline-360x200.png","time_to_read":7,"content":"\u2026 searching prefetched content, which we talked about earlier. Say you have a music app downloading favorite songs, or a news app fetching the latest articles before your user wakes up. With Algolia Offline, users can now search this content wherever they are\u2014be it quietly sipping a coffee at home, backed up by a good Wi-Fi, or wandering through the subway, enduring an intermittent, laggy connection.\nA second, more subtle use case, is to improve UI responsiveness. Even when network coverage is available, latency may be too high to accommodate instantaneous, as-you-type search experiences. But when your online index is synced locally, you can provide instant results at each keystroke. It doesn\u2019t have to be the complete data: you may use the local index as an autocomplete source, for example. Any strategy is imaginable: you could provide offline as-you-type results, then go online when the user stops typing or presses the \u201cSearch\u201d key.\nWe want to see what you dream up!\nWe truly believe that the potentialities of Algolia Offline go well beyond what could be described in this article. You will probably come up with many more interesting use cases.\nSo, go ahead! Try it now, and share your feedback with the community, so that we can keep improving on this product. And stay tuned for updates: this is just v1; more awesome features are to be expected in the future.","record_index":3},{"post_id":5927,"post_title":"Personalized Search at the Speed of Algolia","post_date":1485934860,"post_date_formatted":"Feb 1st 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/personalization\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/personalization-illustration-360x200.png","time_to_read":4,"content":"Search is more than a query and results - search is a conversation your users have with your product. When done right, you feel understood by the engine - and, in turn, the product - which reacts to your needs and adapts to what you\u2019re saying as you\u2019re saying it.\nPerhaps one of the most magical moments is when the search engine speaks to you - an individual person, with search results that are catered to your personal tastes and expectations.\n\"Personalization, yes! But not at the cost of speed.\"\nOne of the core focuses of Algolia is speed. Every time we\u2019ve looked at how other search engines handle personalization, it always came at the cost of speed - we don\u2019t think personalizing the search results makes sense if it makes the search slow. Who wants a slow, personal conversation?\nWe rolled up our sleeves earlier this year and got to work, and we\u2019re proud today to announce two features that make personalization of search results a breeze, without compromising on the speed Algolia is known for. We call these two API features Optional Filters and Filter scoring - and here\u2019s how it works.\nBecause the feature can place high demands on CPU performance, it is enabled by default only in our Enterprise plans. If you\u2019re interested in trying it out on your plan, send us a note.\nGetting started\nLet\u2019s take the example of searching through an index of movies. In this hypothetical website, users can add movies to their \u201cWatch later\u201d list - a familiar feature - and this information is added to the index, in the form of an attribute watch_later with an array of user IDs of every user that adds this movie.\nA Sample JSON entry:\n\nWhen we search this index, we want movies that are added to a watch list to appear before others.\nA regular search would look like:\n\nWith personalization, it would work like this:\n\nWith only one added parameter, suddenly the results are personalized: the search retrieves all the movies related to \u201ckarate\u201d, but if a user has added one of","record_index":0},{"post_id":5927,"post_title":"Personalized Search at the Speed of Algolia","post_date":1485934860,"post_date_formatted":"Feb 1st 2017","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/personalization\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/02\/personalization-illustration-360x200.png","time_to_read":4,"content":"\u2026 the movies related to \u201ckarate\u201d to their watch list, this result will show up at the top of the results.\nGoing further: multiple levels of personalization\nWhat if you want to have multiple layers of personalization? For example, let\u2019s say you\u2019ve analyzed that a user is a huge fan of Action movies, watches a lot of comedies, but is not restricted to these two genres. You\u2019d like the results to first show Action movies, then Comedies, and then the rest of the results. Here\u2019s how you\u2019d do it:\n\n...And you\u2019ll get the results in the order that you expected! We added a new rule called filters in our ranking formula, which will rank the results according to the matching optionalFilters and filters scores.\nWe\u2019ve been playing around with these features for a few months now, and a few of our customers already use them in production during the beta period, with amazing results.\nThe best part about these features is how versatile they are: there\u2019s virtually nothing you can\u2019t do when it comes to personalization of results:\n\nOn a professional social network, retrieving people that are part of your extended network first\nWhen searching on a shoe store, displaying brands or categories of products the user has bought from before first\nIn a CRM search, displaying leads I\u2019ve contacted in the past first, or the ones that are assigned to me\n\nIf you want to know more about these features, we wrote a guide on personalization. We can\u2019t wait to see what you\u2019ll build with them!","record_index":1},{"post_id":5900,"post_title":"Solutions Engineers: Advocating for the Customer","post_date":1484816385,"post_date_formatted":"Jan 19th 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/solutions-engineers-advocating-for-the-customer\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/se-team-1-360x200.png","time_to_read":4,"content":"We\u2019re the Solutions Engineering team at Algolia. What does that mean exactly? We do a lot of different things, but they\u2019re all tied together by one philosophy:\nWe advocate for the customer inside Algolia.\nBeing a Solutions Engineer requires flexibility above all. On any given day, we speak with customers and future customers, build out proofs of concept that show just what Algolia can do, and even add new features to the product itself. The tasks are varied and no two days are alike. Our priorities are determined by a single question: will this help our customers be more successful?\nWe\u2019re proud of\u2014and enjoy\u2014what we do. We wanted to share it with you, to pull back the curtain and give you insight into this team you may have interacted with.\nA Variety of Job Requirements; A Variety of Backgrounds\nTo make our customers successful, we need to know what makes us successful. When we are evaluating a potential new SE colleague, what skills should they have?\nA successful Engineer on the Solutions Team should have the following qualities:\n\nProgramming aptitude\nCommunication\nUX design knowledge\nOrganization\nIdeation\/innovation\nProactivity\nHigh standards\nTeamwork\n\nIt\u2019s a decently-sized list and the qualities intersect well with each other. An Engineer on the Solutions Team won\u2019t be able to communicate well with our customers and colleagues if she\u2019s disorganized. And we consider good programming and UX knowledge to be siblings of a sort.\nWe\u2019ve worked for startups, we\u2019ve been at agencies, and we\u2019re former educators. No matter how we got here, we use our experience to the benefit of our customers.\nWe Superpower Our Customers\nOn average, 60% of our time has a direct customer impact, with the other 40% spent on projects that look toward the long-term.\nSome work that might fall in the \u201cdirect customer impact\u201d bucket:\n\nA well-known forum wants to search 100s of millions of posts and comments. What\u2019s the best way to organize the data given our","record_index":0},{"post_id":5900,"post_title":"Solutions Engineers: Advocating for the Customer","post_date":1484816385,"post_date_formatted":"Jan 19th 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/solutions-engineers-advocating-for-the-customer\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/se-team-1-360x200.png","time_to_read":4,"content":"\u2026 infrastructure and their search needs?\nWhat\u2019s the best UX for an ecommerce shop that sells books, toys, electronics, lawn equipment, sporting goods, food, DVDs, CDs, & video games?\nA potential customer wants to see what Algolia would look like on their website. Can you flex your technical muscles to blow them away?\n\nHere are a few examples of our 40% long-term work:\n\nThere\u2019s a conference in Australia in front of 200 SaaS founders? Sign us up!\nNeed to redesign the dashboard? Let\u2019s assemble a team and make it awesome!\nHow about an improvement to our instantsearch.js library? We\u2019re on it.\n\nI asked my colleagues what they enjoyed most about their jobs. I thought Raph said something particularly interesting. He joined us in the spring of 2016 and he said he would tell people that:\n\u201cI love being part of an awesome team that\u2019s tackling a real problem: making content more accessible, and to see that it has a real impact.\u201d\nAt the end of the day, that\u2019s what makes our job so rewarding: we are making a real difference in our customers\u2019 lives and their customers\u2019 lives. Be it a well-known business or an e-commerce shop just trying to get off the ground, we are helping businesses be more amazing\nWe Superpower Our People\nOne of the overarching tenets of working at Algolia is that we hire really talented people and then empower them to do their best work. If you don\u2019t like giving talks to big groups, but you love creating video tutorials, we\u2019re going to encourage you to do what you are best at.\nEach person here cares deeply about the success of the others. Take a look again at the key qualities of our team members. Our teammates have high-standards and they value teamwork. We expect each person to look out for the others and to actively assist them when they need help.\nAnd that extends beyond our immediate team. We work closely with every team at Algolia. I often tell people that I\u2019ve learned more in the past year about sales and recruiting than I","record_index":1},{"post_id":5900,"post_title":"Solutions Engineers: Advocating for the Customer","post_date":1484816385,"post_date_formatted":"Jan 19th 2017","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/solutions-engineers-advocating-for-the-customer\/","categories":["Business"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/se-team-1-360x200.png","time_to_read":4,"content":"\u2026 ever thought I would\u2014or would want to. If I ever left Algolia to build a product, I\u2019m in a much better place than I would have been in a strictly engineering role.\nDid I mention that we\u2019re hiring?\nIf I had to sum us up in a pithy way, I\u2019d say we\u2019re engineers who obsess over our customers. It doesn\u2019t quite fit on a business card, but it works.\nIf you read the above and thought to yourself, \u201cThat\u2019s me!\u201d then we want to hear from you. We are hiring in SF, NYC, and Paris and can\u2019t continue to grow without talented people.\nE-mail us at solutions.recruiting@algolia.com and let\u2019s chat.","record_index":2},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"One of the most unique and most-used features of Algolia is the Distinct feature: it enables developers to deduplicate records on-the-fly at query time based on one specific attribute. We introduced this feature three years ago, opening up a broad range of new use cases like the deduplication of product variants\u00a0- a must-have for any eCommerce search. It has also considerably changed the way we recommend handling big documents like PDFs or web pages (we recommend splitting each document into several records).\nThis post highlights the advantages of this feature and the different challenges it represents in term of implementation.\nDocument search: state of the art and limitations\nSearching inside large documents is probably the oldest challenge of information retrieval. It was inspired from the index at the end of books and gave the birth to an entire sub-genre\u00a0of statistics. Among those methods, tf-idf and BM25 are the two most popular. They are very effective for\u00a0ranking documents, but they don't handle\u00a0false positives well\u00a0- they push them to\u00a0the bottom\u00a0of the results.\nTo illustrate this problem, you can perform the \"states programming\" query on Google and Wikipedia. According to Google Trends, this query is as popular as the Rust programming language and seems to correspond to developers that search how to develop an algorithm with a state. The same query on Wikipedia is unfortunately not very relevant! The first issue probably comes from a stemming algorithm or a synonym that considers \"program\" as the same than \"programming.\" In Algolia, you can have an expansion of singular\/plural without polluting results\u00a0with stemming\u00a0by using the ignorePlurals feature, which\u00a0is based on a linguistic lemmatizer.\nThat said, even if you scroll through the first 1000 hits of Wikipedia search, you won't\u00a0find the article which appears first in\u00a0Google. There are hundreds of\u00a0articles that contain both the word \"states\" and \"programming.\" Even the \"United States\" page","record_index":0},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 contains both terms and one is in the title! In this case, tf-idf and BM25 are not the most useful ranking criteria. The position of the two query terms in the document is more important, in addition to\u00a0their relative distance (finding them close together is always better).\nStates programming query on Google and Wikipedia\nWhy do we split large documents\nOne of the best ways to avoid such relevance\u00a0problems is to split big pages into several records - for example, you could create\u00a0one record per section of the Wikipedia page. If a Wikipedia article were to have\u00a04 main sections, we could create 4 different records, with the same article title, but different body content.\nFor example, instead of the following big record with the four sections embedded:\nView the code on Gist.\nWe could have 5 different records, one for the main one and one per section. All those records will share the same source attribute that we will use to indicate the engine that they are all component of the same article:\nView the code on Gist.\nWe can use this\u00a0same approach for technical documentation, as there are\u00a0often just a few long pages in order to improve the developer experience: scrolling is better than loading new pages!\u00a0This is exactly why we have decided to split pages into several records in our DocSearch project; you can learn more about our approach\u00a0in this blog post.\nIt might sound counter-intuitive, but the problem is even more visible when your data set is smaller! While the problem is only visible on well-selected queries on the Wikipedia use case,\u00a0it becomes very apparent when you search inside a\u00a0website with less content. You have a high probability of having false positives in your search that will frustrate for your users.\nThe need for deduplication\nSplitting a page into several records is usually easy as you can use the formatting as a way to split. The problem with several records per document is that you will introduce duplicates in your search results. For","record_index":1},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 example, all paragraphs of the United States Wikipedia article will match for the \"United States\" query. In this case, you would like to keep only the best match and avoid duplicates in the search results.\nThis is where the distinct feature comes to play! You just have to introduce one attribute with the same value for all records of the same source (typically an ID of the document) and declare it as the attributeForDistinct in your index setting. At query time, only the best match for each value of the attribute for distinct will be kept, all the other records will be removed to avoid any duplicate.\nIf we split the United States Wikipedia article by section and subsections, it would generate 39 records. You can find the first four records on this gist:\nView the code on Gist.\nThis split by section reduces a lot the probability of noise on large articles. It avoids the query \"Etymology Indigenous\" to match this article (which would typically match because there is a section called \"Etymology\" and another one called \"Indigenous and European contact\"). To improve the relevance, we have also divided the records into three different types that we order from the most important to the less important via the recordTypeScore attribute:\n\n\"main\", this is the first section of the article, including the title and synonyms (score of 3)\n\"section\": those are the main sections of the article (score of 2)\n\"subsection\": subdivision inside the sections of the article (score of 1)\n\nThe recordTypeScore attribute will be used in the ranking to give more importance to the \"main\" records than the \"section\" and \"subsection\".\nAnother good property of this split is that it allows linking to the right section of the page when you click on the search result. For example, if you search for \"united states Settlements\", you will be able to open the United States page with the anchor text stored in the record.\nHere is the list of the four Algolia settings we applied on this index:\nView the code on","record_index":2},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 Gist.\nYou can set those settings via the setSettings API call or directly on the dashboard:\n\nsearchableAttributes and customRanking can be configured in the Ranking tab\n\nAlgolia Dashboard: Searchable & Ranking Attributes\n\nignorePlurals can also be configured in the Ranking tab. For the moment you can only enable it for all languages in the dashboard (setting it to true), we will improve this setting to let you configure also the language.\nattributesForDistinct can be configured in the Display tab of the dashboard:\n\nAlgolia Dashboard: Group by\nEach of those settings is important for the relevance:\n\nThe customRanking uses the recordTypeScore to give importance to the main chapter in the case of equality on the textual relevance. In case of equality, we use the popularity attribute which represents the number of backlinks inside Wikipedia to this article (all records of the article keep the same popularity)\nThe order of attributes in\u00a0searchableAttributes gives the following decreasing\u00a0importance for\u00a0the matching attributes, title > text > synonyms > sourceArticle\nThe attributeForDistinct is applied on the name of the article, only the best elements will be displayed for each matched articles\nWe set ignorePlurals to \"en\" to consider all singular\/plurals form of English as identical without introducing any noise (you can also set this setting at query time if you have a\u00a0search in a different language\u00a0using the same index)\n\nWith those settings, you can easily request a set of results without duplicates with the query parameter\u00a0distinct=true. You can go even further by requesting several hits per deduplication key. You can, for example, pass distinct=3 in your query to retrieve up to three results per distinct key - this feature is usually\u00a0known as grouping.\nKeeping several records per distinct key\nSearch engines are also used for navigation or criteria-based search due to the fact that those queries are cheaper than on a regular database. Keeping","record_index":3},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 several records per distinct key makes it\u00a0easy to bring a different display to search.\nFor example, if you want to build a list of job offers, you will probably let the user search via different criteria like the role, location, job type, etc - but you might also want to aggregate the search results per company instead of having each job display company information. This is what AngelList is doing when you search for a job on their platform. They display companies along with the three best job offers, this type of display can be built very quickly by:\n\nHaving one record per job offer with an attribute containing the company ID\nConfiguring the company ID attribute as attributeForDistinct in the index settings\nPassing distinct=3 in your query\n\nWith those three simple actions, you can\u00a0build an interface like AngelList below (Note that SafeGraph has only two job offers that match the criteria, the value three is the upper-bound limit on the number of records we want per distinct key). Some customers also implement a \"Show more\" button on each company that allows to display all offers for this company. To implement it, you have just to filter on the company name and specify distinct=false on the query to retrieve all offers of this company.\nAngelList job search displaying several jobs per company\nThe Impact of Distinct on faceting\nFaceting can very quickly become a headache when distinct is enabled. Let's illustrate the problem with a very simple example containing two records sharing the same distinct key.\nView the code on Gist.\nThe first record contains the facet values \"A\" and \"B\" and the second record contains the facet values \"B\" and \"C\".\nLet's imagine that you have a query that retrieves both records and that faceting and distinct is enabled in your query. You can imagine three different behaviors in this case:\n\nComputing the faceting before the deduplication (using distinct). In that case, the possible facet values will be \"A=1\", \"B=2\" and \"C=1\". All categories","record_index":4},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 are retrieved but the count won't fit what you have on screen.\nComputing the faceting after deduplication (distinct). In this case, the result would be \"A=1\", \"B=1\" OR \"B=1\", \"C=1\" depending on the best matching record. In both cases, the result is not what you expect.\nApplying deduplication independently for each facet value. In this case, we would have A=1, B=1 and C=1 (for each facet value, we need to look at all distinct key and perform a deduplication at this level).\n\nThe third behavior is the holy grail as the result of each facet count would be perfect. Unfortunately, the computation cost is insane and doesn't scale as it directly depends on the number of facet values which match. In practice, it's impossible to compute it in real time.\nLet's look at the advantages and drawbacks of the two approaches which can be computed for each query in a reasonable time:\n\nComputing the faceting before the deduplication: all retrieved facets are valid but the problem is obviously the counts are misleading because they include the duplicates. That said, this approach works very well if you do not want to display the count, it handle any case including the fact you can have different facets in the records that share the same distinct key.\nComputing the faceting after the deduplication: this approach works well when you use distinct=1 and all records with the same distinct key share the same facet values, which correspond exactly to the use case of splitting big documents. In other use cases, the results will be disturbing for users as some facet values can appear\/disappear depending on the record which is kept after deduplication.\n\nIn other words, there is no perfect solution solving all problems. This is why we have decided to implement both approaches but as the deduplication can be misleading and cause weird effect, we have decided to use the faceting before the deduplication as the standard behavior. We have a query parameter called facetingAfterDistinct=true that can be","record_index":5},{"post_id":5868,"post_title":"Inside the engine part 7 \u2013 Better relevance via dedup at query time","post_date":1484541526,"post_date_formatted":"Jan 16th 2017","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-7-better-relevance-via-dedup-at-query-time\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2017\/01\/blog-illus-360x200.png","time_to_read":11,"content":"\u2026 added to your query when you are splitting some big records. In this case, you have a perfect result because the facets are identical in all records sharing the same distinct key.\nGreat features take time\nWe introduced the distinct feature in December 2013 and have improved it significantly in July 2015 by adding the support of distinct=N in the engine. The different faceting strategies have been a long-standing challenge and it took us a lot of time to find the best approach.\u00a0The query parameter facetingAfterDistinct=true has been beta tested with real users for a few months before becoming public in December 2016.\nThis feature opened a lot of possibilities for our users and allowed a better indexing of big records that later gave the birth to our DocSearch project.\nIf you want to know more about the internal of the Algolia engine, we recommend to read the other posts in this series:\n\nPart 1\u2014indexing vs. search\nPart 2\u2014the indexing challenge of instant search\nPart 3\u2014query processing\nPart 4\u2014textual relevance\nPart 5\u2014Highlighting, a Cornerstone of Search UX\nPart 6\u2014Handling Synonyms the Right Way\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":6},{"post_id":5821,"post_title":"Introducing the Algolia Community forum and the Pioneer Badge","post_date":1482485173,"post_date_formatted":"Dec 23rd 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-community-forum-pioneer-badge\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/community-badge-illustration-360x200.png","time_to_read":7,"content":"Today we\u2019re making a new resource available to the Algolia Community. It\u2019s a modern take\u00a0on\u00a0one of the Internet\u2019s most fundamental gathering\u00a0tools: the forum.\nSee it live: the Algolia Community forum\nAs the Algolia Community has grown, it\u2019s become very active on Github, StackOverflow and Twitter. As developers, we love these tools but there are gaps as we scale. Sometimes we want to have deeper conversations or conversations not specifically about programming. Other times we want to have larger, ongoing discussions and\u00a0get input from a lot of\u00a0people.\n\nDiscourse, the open source project we\u2019ve chosen to power the new\u00a0forum, is a chance to bring\u00a0everyone together into one place and continue the\u00a0conversation at scale. Discourse has customizable notifications, code highlighting, link expanding, badges and many other goodies right out of the box. It also has an excellent set of community moderation tools. Some very active communities run on Discourse including Github\u2019s Atom, Twitter Developers, and Docker. We\u2019re excited to join them as\u00a0part of the larger Discourse community.\nCustomizations\nDiscourse is open source and has a powerful set of admin tools, so it's really a breeze to customize. One\u00a0customization we've done so far is visual, by applying a minimalist theme that fits in with our overall community fonts, styles and colors.\n\nWe've also replaced some of the default badges with ones that match\u00a0our theme. See them all on the\u00a0Badges page, and keep reading to learn more\u00a0about our first brand-new addition, the Pioneer Badge.\nCategories\nThe top-level organizational unit of Discourse is the category. Categories contain topics which have one\u00a0or more\u00a0posts. Notifications can be turned on and off\u00a0at the category and topic level which means that\u00a0you decide what you want to receive, not us.\u00a0For example, if you're using the Shopify plugin, you shouldn't get emails about the WordPress plugin (unless you want to). Because the\u00a0Shopify and","record_index":0},{"post_id":5821,"post_title":"Introducing the Algolia Community forum and the Pioneer Badge","post_date":1482485173,"post_date_formatted":"Dec 23rd 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-community-forum-pioneer-badge\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/community-badge-illustration-360x200.png","time_to_read":7,"content":"\u2026 WordPress\u00a0categories live separately on the forum, along with about a dozen others, it's possible\u00a0for you to configure what you're watching at a fine-grained level.\n\nThe home page of each\u00a0category has a menu like this where you can configure what you want to receive. For categories you're tracking or watching, you\u2019ll get unread counts, so even if you don\u2019t want notifications sent to you it's easy to see what you\u2019ve missed just by coming back to the site.\u00a0We've created a few categories so far, here's a summary of a few of them.\nThe Announcements category is one to keep an eye on. We\u2019ll use it to post official announcements, including releases and new features, and also events and highlights from around the community. Introductions from each new member also live here inside of a special introductions topic.\nThe Projects category is a place to show off your Algolia implementations, projects and extensions. It\u2019s ok to brag here! The community will love to hear how you configured your search and why you made the choices you did. You can also ask the community\u00a0to try\u00a0your search and give you feedback.\nThe Development category is the place to talk about the Algolia API, SDKs, and anything you\u2019re using to get data into and out of Algolia. If you have questions about the architecture, technical or security design of your integration ask them in here. For anything else, you can get\u00a0access to more viewpoints and potentially more depth by posting here.\nWe still encourage you to use StackOverflow for programming problems where you are expecting a single correct answer and Github Issues for repository-level bugs and feature requests.\nThe Design, UX and Relevance category is a place to\u00a0talk about best practices for designing and implementing the user-facing aspect of your search. Not sure whether to use an autocomplete or instantsearch interface? Curious about what makes a search great on mobile? This is the category to ask in. Small changes to design and","record_index":1},{"post_id":5821,"post_title":"Introducing the Algolia Community forum and the Pioneer Badge","post_date":1482485173,"post_date_formatted":"Dec 23rd 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-community-forum-pioneer-badge\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/community-badge-illustration-360x200.png","time_to_read":7,"content":"\u2026 relevance can have a big impact on the satisfaction of your users, and we encourage you to post in here with questions or lessons you\u2019ve learned.\nThe DocSearch category\u00a0is for the\u00a0200+ maintainers who are using DocSearch to power the search on their documentation. It's also for any of\u00a0the teams we're backing\u00a0via\u00a0our sponsorship of Open Collective\u00a0including webpack.\nThe Site Feedback category exists for you to ask questions and give feedback about the forum itself. All thoughts and questions about organization, moderation, and ways\u00a0to improve are welcome.\nHow to get the Pioneer Badge\nWe want to reward Algolia developers and fans who join the forum in 2016 with a special badge. The entire Algolia team, and especially our developer advocates, really appreciate your help with beta testing and trying new things. To us, you're pioneers who are constantly pushing forward the\u00a0frontier of search.\n\nHere is what you can\u00a0do to get the Pioneer Badge.\n\nHead to\u00a0the forum\u00a0and\u00a0click Log In.\u00a0You'll be authenticated with your algolia.com account, or prompted to log into Algolia if you weren't already.\nIntroduce yourself on\u00a0the introductions thread.\nTell the community about what you're working on in the Projects category.\n\nPlease complete all 3 steps by January\u00a031,\u00a02017. You will receive your virtual badge within a few days. We will follow up to send you a\u00a0physical sticker for your laptop, along with information about extra badge-having benefits you can expect in 2017.\nBadges are fun but\u00a0they also play another important role in community building\u2014badges\u00a0help members establish\u00a0their\u00a0reputation. Reputation helps new members identify experts and empowers experts to take more ownership. Reputation is also a way for developers to build their portfolio and strengthen\u00a0their career. We expect the Pioneer Badge to be the first of many opportunities to build a valuable, market-recognized reputation as an Algolia developer.\nThanks for being a pioneer, let's explore","record_index":2},{"post_id":5821,"post_title":"Introducing the Algolia Community forum and the Pioneer Badge","post_date":1482485173,"post_date_formatted":"Dec 23rd 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-community-forum-pioneer-badge\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/community-badge-illustration-360x200.png","time_to_read":7,"content":"\u2026 more\u00a0frontiers together in\u00a02017!\nBig thanks to Gianluca for deploying and configuring the Discourse instance, Antoine for creating the design, Jonas for implementing the design, S\u00e9bastien for creating the Pioneer Badge, Liam for reviewing this blog post, Raymond for onboarding our WordPress users, Matthieu for onboarding our Shopify users, and everyone at Algolia and\u00a0in the community who has contributed something\u00a0so far!","record_index":3},{"post_id":5796,"post_title":"Harnessing API\u2019s with React: a different approach","post_date":1481165558,"post_date_formatted":"Dec 8th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harnassing-apis-with-react-a-different-approach\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/react-instantsearch-360x200.png","time_to_read":5,"content":"Today we are launching react-instantsearch, a new library to build search interfaces with Algolia in React and React native. This new way of implementing Algolia brings ideas that go beyond the creation of widgets using React - we think they will fundamentally influence the way the JS community builds UI libraries!\nA year ago we launched instantsearch.js, in order to provide an easy solution for building search interfaces The project now has more than 1000 stars on github and 1500 users, far exceeding our expectations. With the release of instantsearch.js, we tried to help front-end engineers as much as possible and because the framework war was roaring hard back then, we went for the universal choice: Vanilla JS.\nIt was our first take at building a complete widget library for building search UI\u2019s. With it, we tackled the problems that our users found when building a search interface, namely the lack of packaged options for search UX patterns, and the cumbersome mapping of concepts from the search realm to the UI realm.\nTo tackle those issues, we took the vanilla JS \/ framework \u201cagnostic\u201d approach. We created a light framework to be able to consolidate all the search parameters that each widget can set. Our final product was a drop-in search addon for all JS front-ends; however, this light framework is nothing without the widgets that are included. Each of them provides options to customise the search behavior and also their UI - it turns out, you can never provide enough options for UI and behavior.\n\u201cYou can never provide enough options\u201d\nWe started with fixed markup, but this was not enough because you might want to use a specific CSS library with its own markup. So we added the ability to customize some parts of the UI with templates, but this wasn\u2019t enough because users wanted customization across the board. We added the possibility to manipulate bigger chunks of the UI, but this created inconsistencies with our own expectations for the markup which","record_index":0},{"post_id":5796,"post_title":"Harnessing API\u2019s with React: a different approach","post_date":1481165558,"post_date_formatted":"Dec 8th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harnassing-apis-with-react-a-different-approach\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/react-instantsearch-360x200.png","time_to_read":5,"content":"\u2026 led to more options\u2026 in the end, the more options you provide, the more complexity you create.\nComplexity has two faces. First, it makes it harder to deal with inconsistencies and bugs. Second, it creates API noise. Dealing with inconsistencies and bugs is hard but it\u2019s always a matter of how much you work on it, and eventually it\u2019ll be fixed. On the other hand, the more options you have, the harder it will be to grasp what are the important or critical options. The amount of options creates an artificial sense of complexity, users get the impression that the learning curve is steep. They had to dig through all the options to find the one they need.\nOverall the API created for supporting UI options made the library harder to learn and use, and therefore the developer experience suffered from it.\nWe began to wonder \u201dWhat if we could separate the two, and provide options that are only meaningful for the search and limit the UI options?\u201d - this is how react-instantsearch was born.\nDecouple search logic from rendering\nFrom the outside, react-instantsearch shares the same philosophy as instantsearch.js. It provides widgets, and each will handle a single part of the search UI. The combination of widgets makes up a search query to Algolia, which in turn updates the UI with new results, which in turn allows the user to further refine the search. Unlike its cousin, however, react-instantsearch handles customization by giving developers complete control over the rendering - the result is worth taking a look at:\nreact-instantsearch is made of a root component <InstantSearch> and other UI components that we call widgets. These widgets are the composition of a dumb react component with a business logic counterpart called connectors.\nAnd this is where the true originality of react-instantsearch is.\n\nConnectors are higher order components that provides a dual API. For the wrapped component, it exposes the minimal API necessary to let it do what it is supposed to","record_index":1},{"post_id":5796,"post_title":"Harnessing API\u2019s with React: a different approach","post_date":1481165558,"post_date_formatted":"Dec 8th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harnassing-apis-with-react-a-different-approach\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/react-instantsearch-360x200.png","time_to_read":5,"content":"\u2026 do: set and read the query if it\u2019s a SearchBox, or set filters and get the filter values if it\u2019s a RefinementList. For example, a minimalist implementation of a custom SearchBox would be:\nView the code on Gist.\nThe connector also provides an external API that let the developer set the search settings relevant to the concerned domain. For example, the `hitsPerPage` parameter is set on the Hits widget but is used by the connector to set the number of results to display on a single page.\nNow we have a clear and powerful API that provides all manners of customization for developers, and at the same time we can provide ready-to-use widgets that have limited options for UI tweaks. Our widgets are the results of our collective experience with search UX and what we\u2019ve learned from users.\nThe end result is that beginner users are not left wondering what kind of experience they should implement, while experienced developers can adapt their search experience accordingly. Developers looking for a quick & powerful solution will find exactly that with react-instantsearch - the deeper they dive, the more they will uncover its flexible underbelly by overriding the defaults we provide .\nreact-instantsearch is a dual-faceted API:\n\nOpinionated widgets that represent our vision of a good search UX. They have fewer options and therefore provide a faster execution\nCustomizable Connectors HOC\u2019s which are UI-free, allowing you to fully adapt them to your desired search experience.\n\nA glimpse of the future for service based UI\u2019s\nAlgolia is, at its core, a search engine. UI&UX are very important elements that make for a great implementation and differentiate an unused search bar from a central navigation tool. Those improvements are very prominent when binding the search capabilities of an engine to the UI.\nThis new \u201cflavor\u201d of instantsearch provides more for the UI by channeling the capabilities of the search into abstract widgets. We provide our way of doing those","record_index":2},{"post_id":5796,"post_title":"Harnessing API\u2019s with React: a different approach","post_date":1481165558,"post_date_formatted":"Dec 8th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/harnassing-apis-with-react-a-different-approach\/","categories":["News","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/12\/react-instantsearch-360x200.png","time_to_read":5,"content":"\u2026 implementations, but we also let you implement new UI widgets through the prism of our experience on how to use our engine.\nIf you want to have a look at what the next version of instantsearch.js will look like, try react-instantsearch and let us know if we have the right balance of feature and customization (on twitter, gitter or by email).\nAs a final word, we would like to express our sincerest gratitude to Alexandre Kirszenberg who built the initial POC during his summer internship. Thanks!!!","record_index":3},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"In any search engine, adding synonyms is one of the most important customization to introduce domain-specific knowledge. Synonyms make it possible to tune search results without having to make major changes to records.\nSupporting synonyms is no easy task especially for multi-word expressions, which introduce complexity in the handling of the proximity measure between matched words. We will describe in this post how we handle this complexity via a rewriting of word positions at query time.\nDifferent types of synonyms\nThe most common way to define synonyms is to use a set of expressions that are all identical. We call such a set an N-way Synonym because any expression of the synonym set is a synonym of another expression.\nFor example, if New York is unambiguously a City in your use case, you probably want to have a synonym set containing \"NY\" = \"NYC\" = \"New York\" = \"New York City\". This synonym set is configured as N-way, which means that:\n\n\"NY\" in the query also matches \"NYC\" or \"New York\" or \"New York City\"\n\"NYC\" in the query also matches \"NY\" or \"New York\" or \"New York City\"\n\"New York\" in the query also matches \"NY\" or \"NYC\" or \"New York\" or \"New York City\"\n\"New York City\" in the query also matches \"NY\" or \"NYC\" or \"New York\"\n\nLess common but still used are asymmetrical synonyms,\u00a0in which you want to apply an equivalency only in one direction - we call this a 1-way Synonym. For example, \"smartphone\" = [\"iphone\", \"android\"] means that you want the \"smartphone\" query to be expended in \"iphone\" and \"android\" because you can have a record containing iphone without having the word smartphone. That said, you do not want the query \"iphone\" to returns all records containing the word \"smartphone\" - that wouldn\u2019t be relevant for the user.\nThe Algolia engine supports both 1-way and N-way synonyms, and both types can contain any type of expression (one word or multiple words like \"New York City\").\nThe dangers of generic synonym sets\nWe are often asked by users when we will","record_index":0},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 include a standard synonym dictionary in the engine. The question is understandable - after all, we are taught in school that most words have synonyms, so it seems easy to package all this knowledge into one generic resource (per language). Unfortunately, it\u2019s not that simple.\nThere are varying degrees to which two words or phrases are synonymous. For example, \"substitute\" is a stronger synonym to \"alternative\" than \"choice\" even if you can find both in a standard English synonym dictionary. Adding \"substitute\" as a synonym of \"alternative\" can be a very good idea for a website that compares technologies or offers; that being said, it can easily add noise on other use cases, especially if the name of a technology contains \"substitute\" in the name like the Substitute or jquery-substitute open source projects.\nPolysemy is present in all Natural languages, which means that a word can have different meanings depending on the context. For example, the word \"crane\" can be a bird or a construction equipment and the context is required to select the correct meaning. Our customers are all in a specific semantic context and the dictionary need to be specific to give the best results. If you are on a tech website you don't want to have a generic synonym of the word \"push\" because there is a strong meaning in tech and using synonyms for the verb or the noun \"push\" would lead to weird results.\nThe most useful synonyms are always very specific to your domain. This is the case of the synonym set [\"NY\" = \"NYC\" = \"New York\" = \"New York City\"] we mentioned before. This synonym set is valid if you only have cities in your use data set, it could introduce a bad relevance if you also have States!\nEven specific synonyms can have a different meaning in another use case. For example, I saw the synonym T-shirt = Tee on several ecommerce websites. This synonym makes sense as a lot of users are typing \"tee\" to search for a t-shirt. That said, this synonym is dangerous if you have a sports","record_index":1},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 website as you can have a query for \"tee ball\" or \"golf tee\" and won't like to return t-shirt!\nWith all those constraints in mind, it is close to impossible to package a set of synonyms that would make sense for all use cases as it is too much use case dependent. In practice it would lead to a bad relevance and results would be hard to understand for your users.\nThis is why we do not plan to package a generic synonym dictionary in the engine. That said, there are other resources that we already package in a generic way like the singular\/plural forms of a language that we expose in the ignorePlurals feature. We plan to continue introducing such resource in the future that provides a lot of value without having the drawback described before.\nComputing synonym at indexing or query\nOne of the main question with synonyms is to make the choice between pre-computation at indexing time or expansion at search time. Both approaches have advantage and drawbacks:\nWith pre-computation at Indexing time, if you have a synonym \"NY\" = \"NYC\", you will index the document with both words when one of the two words is found in a record. This method is the best for performance as there is no OR between the two synonym words at search time. The biggest issue with this approach is to introduce the support of multi-words synonyms, which require introducing a new knowledge in the inverted list.\nFor example, if you have the text \"New York Subway\", the word \"New\" will be indexed as position 0, the word \"York\" will be indexed at position 1 and the word \"Subway\" will be indexed at position 2. If you have a synonym \"New York\" = \"NY\", you will have to introduce in your index the token \"NY\" at position 0 with a padding of 2 positions so that the query \"NY subway\" will have correct positions. This additional information has a big impact on the index size and performance that usually nullify a lot the gain of indexing in most use cases.\nConversely, with computation at search time, you will search for","record_index":2},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 all synonyms at query time with an OR operand between them. The big advantage of this approach is that you can compute the exact position of each matching synonym with a post-processing task (see next section for details). The disadvantage is that it requires more CPU as we have to compute the OR between several inverted lists.\nLastly, A hybrid approach is a mix of both approaches where all synonyms are searched at query time but some expensive computations\u00a0are precomputed in one inverted list. Most of the time the expensive computation that is indexed is the search of consecutive words (phrase query). For example, the \"New York\" synonym would be indexed as one inverted list. The complexity of this implementation is to make it work with prefix search as we will describe below.\nThe implementation of synonyms in the Algolia engine is fully done at search time as we want to have an exact computation of the proximity (see next chapter for more details). We plan to implement a hybrid approach in the future where phrase queries will be pre-indexed in order to improve performance and have the best of both worlds.\nHandling multi-words synonyms is hard\nEven if our implementation is currently fully done at search time, we are still forced\u00a0to rewrite the matched word position in order to have an exact computation of the relevance. We have explained in details this process in part 4 of this series in section 2.3.2.\nWithout this processing, having a synonym \"NY\" = \"New York\" would lead to different proximity for the query \"New York Subway\" on those two records:\n\n \nThe complexity is that \"New York\" is identified as position 1 and 2 in the first record, followed by \"Subway\" at position 3, whereas the second record contains \"NY\" at position 0, followed by \"subway\" at position 1. Rewriting this will ensure the proximity measure between the matched word will be identical for both records. This process essentially considers that \"NYC\" was composed of the two original words","record_index":3},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 \"new\" and \"york\" and will increase all following positions.\nThe following diagram shows the transformation done on the position of words to have an exactly identical proximity computation for records with a different number of words in the synonym (Note: words are indexed starting at position 0. For example, the first record is indexed with \"why\" at position 0, \"new\" at position 1, etc.)\n\nReducing noise in results\nSynonyms are just one of the various alternative expressions that are added on the queries. We also add typo tolerance, concatenation & split of query terms, transliteration and lemmatization. All those alternative expressions are described in the part 3 of this series.\nIn order to avoid adding noise, we do not apply the other alternative expressions on synonyms. This is the big reason why a search for \"New York\" won't give the same number of hits than \"NY\" when you have a synonym \"NY\" = \"New York\", records containing \"NY\" and \"New York\" will be retrieved in both cases but the record containing \"NewYork\" for example will be only returned for the \"New York\" query via the concatenation expression.\nThe following diagram explains the way we add alternative expressions, all steps are computed at the same time and are not re-entrant. In other words, we do not use the output of one alternative expression to feed the input of another step.\n\nPrefix match on synonyms\nHandling synonyms correctly in an instant search use case is pretty complex. Most engines fall short in this respect and have a bad user experience because of this processing. For example, let's say you have 5 records containing \"NY\" and 5 records containing \"New York\" and have defined the synonym \"NY\" = \"New York\":\n\nthe query \"n\" will match all words starting with \"n\", so records containing \"NY\" and \"New York\" will be matched: the 10 records are found\nthe query \"ne\", \"new\", \"new y\", \"new yo\", \"new yor\" will only match the 5 records containing \"New York\" as the synonym is not triggered\nThe synonym","record_index":4},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 will only be triggered when the query contains \"New York\", returning the 10 records\n\nIn order to avoid this flickering of search results, we consider that the last word of an expression is allowed to match as a prefix. So the synonym will be triggered as soon as the query contains \"new y\" (of course the synonym will also be triggered for the query \"new yo\", \"new yor\" and \"new york\"). This approach reduces a lot the problem of flickering as the synonym is added as soon as the last word is started and deliver an intuitive behavior:\n\nIt would be very complex to understand that the query \"ne\" match \"NY\" because we match a prefix of the expression \"New York\" (would be the case if we allow to match the complete synonym as a prefix)\nIt is pretty easy to understand that the query \"new y\" is matching as a prefix of \"New York\"\n\nThis is why, in the Algolia engine, the final word of a synonym expression can be matched as a prefix. Of course, we continue to add the synonym when the expression is only a part of the query, so the query \"New York subway\" will still match the synonym \"New York\" = \"NY\" and records containing \"NY\" and \"subway\" will match. This recognition of expressions at any position of the search query is not performed by most\u00a0engines.\nDifference between original terms and synonyms\nMost users want to consider synonyms exactly identical to the search terms but this is not always the case, as we described before there is several degrees of synonyms and some synonyms are closer than others.\nWe have several tunings available in the engine to let you configure the way to define the expressions that should be considered as identical or not:\nalternativesAsExact is an index setting that specifies the type of alternatives that should be counted in the \"exact\" criterion of the ranking formula. As we explained in detail in part 4 of this series, it counts the number of terms that exactly match the query. By default, this settings is set to [\"ignorePlurals\",","record_index":5},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 \"monoWordSynonym\"], which means that the singular\/plural alternative and the synonyms containing one word are considered as exact but a synonym containing several words won't be considered as exact as they are often semantically farther. You can consider them as exact by adding \"multiWordsSynonym\" in the array.\nThe synonyms dictionary also contains alternative corrections that are different semantically and should be considered as one or two typos. The spelling can be very far from the original words and so won't be cached by the typo detection, so you can define them manually. For the moment this feature is limited to single-word expressions.\nImplementation details matter!\nThe configuration of synonyms is a key aspect of the search relevance and should be configured in most use cases. We have seen in this post that there are a lot of implementation details in the way synonyms are handled that have a direct consequence on the user experience or the ranking.\nAs with most of the features we introduce in the engine, we have spent a lot of time designing synonyms\u00a0and how they\u00a0should behave. This is why we introduced their support progressively: we started with support of single-word synonyms in May 2014, followed by multi-word synonyms in April 2015 and a dedicated API to handle synonyms in June 2016.\nWe took the time to think about these implementation details and to our knowledge we are the engine that goes the deepest in terms of supporting as-you-type synonyms and multi-word synonym positions to have a perfectly identical ranking. We still have improvements to the way we handle the synonyms in our roadmap - like the hybrid indexing of multi-word synonyms to improve performance.\nIf you have any other ideas of improvements, we\u00a0would be very interested to discuss them with you!\nWe recommend to read the other posts of this series:\n\nPart 1 \u2013 indexing vs. search\nPart 2 \u2013 the indexing challenge of instant search\nPart 3 \u2013 query processing\nPart 4 \u2013 textual","record_index":6},{"post_id":5759,"post_title":"Inside the Algolia Engine Part 6 \u2014 Handling Synonyms the Right Way","post_date":1479692078,"post_date_formatted":"Nov 21st 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-engine-part-6-handling-synonyms-the-right-way\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/11\/synonyms2-360x200.png","time_to_read":12,"content":"\u2026 relevance\nPart 5 \u2013 Highlighting, a Cornerstone of Search UX\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":7},{"post_id":5697,"post_title":"How we tackled internationalization in our Zendesk integration","post_date":1478750256,"post_date_formatted":"Nov 10th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-tackled-internationalization-in-our-zendesk-integration\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/blog_zendesk_intl-360x200.png","time_to_read":8,"content":"Zendesk customers are worldwide, coming from every continent. It\u2019s no surprise that their Help Centers support multiple languages out-of-the-box. When we launched our Algolia for Zendesk integration, it shipped with English support by default, and you could extend it to handle other languages. Today, we\u2019re proud to announce that our integration supports 30 languages.\nAlgolia has always been language-agnostic. You can search in an English, Arabic or Chinese text without touching the parameters; but each integration comes with some specific features. Searching in help articles is a specific use-case, and since we\u2019re providing some front-end features (e.g. an autocompletion menu), we also had some text that needed to be translated ( e.g. \u201c10 results found in 2ms\u201d).\nOur initial release had some flaws that we quickly discovered by talking with our first integration users. We got great feedback from pretty big Zendesk users that needed multiple language support out of the box, like Dashlane, whose Help Center is available in English, French and Spanish.\nTL;DR\nI\u2019ve learned the hard way that there\u2019s no magic bullet. Languages are too different to expect being able to simply replace parts of your text easily. Some things aren\u2019t obvious - for example, how one thousand is written numerically in English (1,000) vs. French (1 000)) - some languages have multiple plural forms and you can\u2019t expect that the sentence construction will be the same in any other language. As soon as you want to have dynamic content in a sentence, you\u2019ll need to use some form of templating logic.\nHow did it work before?\nWhen you call our front-end library, you can pass some parameters. We had exposed a simple  parameter, in which each key held either a string with the desired value for all languages or an object associating a locale with a translation.\n\nWe then decided that we wanted to embed a few languages directly inside the application by default, to ease our users\u2019 life.","record_index":0},{"post_id":5697,"post_title":"How we tackled internationalization in our Zendesk integration","post_date":1478750256,"post_date_formatted":"Nov 10th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-tackled-internationalization-in-our-zendesk-integration\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/blog_zendesk_intl-360x200.png","time_to_read":8,"content":"\u2026 That\u2019s when we\u2019ve learned that the simple solution I had developed wasn\u2019t sufficient at all.\nFlaws of the previous solution\nWe used Gengo to get our sentences translated in a first batch of 5 languages. Satisfied with the results, we ordered a total of 30 translations (most of the ones supported by Zendesk). Some languages had a grammar similar enough to the English one that the integration was straightforward. Others brought up issues with the current solution, at three levels.\nDifferent ways of displaying the same information (dates, numbers)\nWe have some great tools to display the same information in multiple languages.\nYour browser ships with  and . Unfortunately, those methods by default use the user\u2019s localization. ECMA2015 has added the support for a new  parameter, but its support by browser vendors is still too low for us to confidently use it for an integration targeting our customers\u2019 end-users.\nAt that time, we simply ignored the different ways of displaying numbers depending on the language. However, for dates, we used the great  library.\nPlural forms\nThis one is a very basic one, but I didn\u2019t think about cases where a plural form was not needed in English but needed in other languages (and I\u2019m French):\n\n\n\nForm\nEnglish\nFrench\n\n\n\n\nSingular\nFound in\nTrouv\u00e9 en\n\n\nPlural\nFound in\nTrouv\u00e9s en\n\n\n\nSomething that came as a real surprise to me though was that languages also have multiple plural forms:\n\n\n\nAmount\nEnglish\nCzech\n\n\n\n\n1\nresult\nv\u00fdsledek\n\n\n2 and 3\nresults\nv\u00fdsledky\n\n\nmore than 4\nresults\nv\u00fdsledk\u016f\n\n\n\nSentence construction\nSometimes it\u2019s just the sentence construction which is completely different. In some languages, the words positioning can be totally different:\n\n\n\nLanguage\nTranslation\n\n\n\n\nEnglish\nNo results found for \u201cquery\u201d\n\n\nGerman\nKeine Ergebnisse f\u00fcr \u201cquery\u201d gefunden\n\n\nJapanese\n\u201cquery\u201d \u306e\u7d50\u679c\u304c\u898b\u3064\u304b\u308a\u307e\u305b\u3093\u3067\u3057\u305f\u3002\n\n\n\nAnother small difference between languages can simply be on the punctuation side. You","record_index":1},{"post_id":5697,"post_title":"How we tackled internationalization in our Zendesk integration","post_date":1478750256,"post_date_formatted":"Nov 10th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-tackled-internationalization-in-our-zendesk-integration\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/blog_zendesk_intl-360x200.png","time_to_read":8,"content":"\u2026 have different quotes in different languages:\n\n\n\nEnglish\nCzech\n\n\n\n\n\u201cquery\u201d\n\u201equery\u201c\n\n\n\nIt\u2019s with all those cards in hand that we\u2019ve started realizing we needed a better framework.\nWhat we ended up with\nWe kept the same logic for the translation object, but this time added the ability to have logic on top of it.\nThe static standalone sentences are still simple translated strings:\n\nOthers now are functions. Those function are called with access to the other translations () and take the dynamic parts as parameters:\n\nFor each locale (e.g. ), you can either override the root translation by using the  key, which will change the translation for every english-speaking locale (i.e. , ,  and ), or by providing a locale specific translation by using the full locale as a key (e.g. ).\nThe only exception for this is Chinese, where Simplified Chinese and Traditional Chinese are too different to have a common root.\nGoing further\nOnce this was done, we started thinking about how we could improve the relevance in each languages.\nThe first thing we thought about is related to the type of queries the users might send. On a Help Center, it\u2019s not unusual to have users type \u201chow to do \u2026\u201d. The issue with that type of query is the potential noise related to words as common as \u201chow\u201d, \u201cto\u201d, \u201cand\u201d or \u201cmy\u201d, that are called stop words in natural language processing.\nA simple example speaks thousand words. Consider two articles:\n\n\n\nChange your password\nHow to delete your account\n\n\n\n\nJust follow this link to change your password\nClick on \u201cDelete my account\u201d. In case you change your mind, your old login\/password will still work during 14 days.\n\n\n\nFor the query \u201cHow to change my password ?\u201d, without any special handling of those frequent words, the Change your password article would not even show up because of the lack of the words \u201chow\u201d, \u201cto\u201d and \u201cmy\u201d in the article. How to delete your account would instead show up.\nThat\u2019s where stop words","record_index":2},{"post_id":5697,"post_title":"How we tackled internationalization in our Zendesk integration","post_date":1478750256,"post_date_formatted":"Nov 10th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-tackled-internationalization-in-our-zendesk-integration\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/blog_zendesk_intl-360x200.png","time_to_read":8,"content":"\u2026 handling kicks in. Algolia offers an automatic stop words removal feature, with the query parameter . You can either use it with  to remove all the stop words in all languages or limit yourself to a specific language. Activated with the example above, Change your password would come back first in the results list and How to delete your account second.\nWhile this works great for full queries, its behavior can be a bit strange when used with prefix search, because the words are used as part of the query while they\u2019re used as a prefix (the query \u201cho\u201d for instance would match \u201chow\u201d, but the query \u201chow to delete\u201d would only match \u201cdelete\u201d). That\u2019s why we chose another solution.\nAlgolia also provides another query parameter called , which can be used to specify which words aren\u2019t required for a record to match. At each keystroke, we\u2019re now looking at which stop words in the current language the query contains, and add them as optional words to the query. How to delete my account ? would still show up first because it matches more query words, but Change your password would show up in the results list.\nThis solution in the end brings a good balance between no stop words handling and the usage of , in terms of relevance but also user experience. This works especially well because the dataset we\u2019re searching in is usually in the 100s of articles, so there aren\u2019t that much relevant results for a query.\nNext steps\nNow that we have this framework, what could we improve upon?\nThe first thing would be to have a way to use language specific tags. We\u2019re indeed displaying all the tags on the search result page by default, you can hide them with three lines of CSS, but having only localized tags could be an even better solution.\nAnother big improvement we will do will be to have one index per locale. This would allow us to move the whole stop words logic from the front-end to the Algolia index settings instead, which would save some space in the","record_index":3},{"post_id":5697,"post_title":"How we tackled internationalization in our Zendesk integration","post_date":1478750256,"post_date_formatted":"Nov 10th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-tackled-internationalization-in-our-zendesk-integration\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/blog_zendesk_intl-360x200.png","time_to_read":8,"content":"\u2026 front-end library.\nFeel free to contribute, the whole code is open-source, and we\u2019d be happy to look at any feature you\u2019d like!","record_index":4},{"post_id":5598,"post_title":"Bringing Algolia\u2019s search-as-you-type experience to Shopify stores","post_date":1477987980,"post_date_formatted":"Nov 1st 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-algolias-search-as-you-type-experience-to-shopify-stores\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/Artboard-Copy-6-360x200.png","time_to_read":3,"content":"Today, we\u2019re excited to release Algolia for Shopify, our all-in-one app that brings Algolia to your store. We've built Algolia for Shopify with simplicity in mind, which means that we automatically keep your Algolia indices up-to-date with your store, and we've provided shop owners with a beautiful UI that works out of the box.\nOur mission at Algolia is to make great search available to everyone - website & application builders, and also their end-users. We've been working hard on our Shopify integration to make sure it brings everything we've learned with our 2,000+ customers and makes it available to the 300,000+ Shopify store-owners at the click of a button.\n\nHow it works\nFor e-commerce websites, search is mission critical. Indeed, a user will often browse a website first to see what\u2019s available, then search for the products he\/she got interested in and checkout. By easing up this process, our aim is to improve the end-user satisfaction, increasing conversion rates and return rates of our customers.\nWe\u2019ve worked hard to provide you with an easy to use, easy to install application.\nOnce you've downloaded our App from the Shopify Marketplace, you'll create\/login to your Algolia account & we\u2019ll start indexing your products and collections right away. Those indices will be updated in real-time using Shopify webhooks, so you\u2019ll never need to worry about indexing again.\nOn the indexing side, we\u2019re simply creating Algolia indices, which means that you get all of the features Algolia provides by default: typo-tolerance, relevance tweaking, synonyms handling,\u00a0analytics and a rock-solid infrastructure.\nOn the front-end side, you can either go ahead and create your own implementation following one of our guides or use the built-in front-end experiences provided with the app.\nAn auto-completion menu\nThe auto-completion menu will be available on any page of your store, in any search input:\n\u00a0\nAn instant-search page\nYou can also replace your current search","record_index":0},{"post_id":5598,"post_title":"Bringing Algolia\u2019s search-as-you-type experience to Shopify stores","post_date":1477987980,"post_date_formatted":"Nov 1st 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-algolias-search-as-you-type-experience-to-shopify-stores\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/Artboard-Copy-6-360x200.png","time_to_read":3,"content":"\u2026 page with an Algolia powered search engine!\n\nAs with all our integrations, it brings the speed, relevance and customizability of Algolia to your end-users, wherever in the world with our Distributed Search Network.\nBut don't take our word for it, try it live!\n\nGoing further\nWant to learn more? We have a few resources for you!\n\nCommunity page\nEcommerce Shopping Search\nDocumentation\nMarketplace entry\n\nGet started","record_index":1},{"post_id":5709,"post_title":"Searching camelCased parameters in API documentation: how we handled it","post_date":1477016985,"post_date_formatted":"Oct 21st 2016","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/searching-camelcased-parameters-in-api-documentation-how-we-handled-it\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/pasted_image_at_2016_10_05_14_45-360x200.png","time_to_read":9,"content":"As you may\u00a0already know, we love having great\u00a0search on documentations we use daily. That\u2019s the main reason we built\u00a0DocSearch.\nDocSearch is live on hundreds of documentation sites, including our own. We believe DocSearch is the easiest and fastest way to navigate technical documentation. Because of that, we invest a lot of time in making it better and better by improving it on a daily basis.\nImproving your DocSearch\nOnce you setup DocSearch, there are three main areas for improvement:\n\nThe structure\nThe content\nThe search itself (indexing and\/or querying)\n\nThe list is ordered by importance, meaning that if you find a relevance issue on a DocSearch implementation it\u2019s usually due to either the structure or the content. Then, in very few cases, it\u2019s due to the search itself.\nThe camelCase issue\nWe just came across one of those search issues: camelCased words.\nCamel case is the practice of writing compound words or phrases such that each word or abbreviation in the middle of the phrase begins with a capital letter\nIf you are an Algolia user you know that all our api parameters are camel cased: see our list of parameters.\nSearch for parameters is working but we found it far from perfect.\nLet me explain why.\nLet\u2019s take, for example, one of the parameters from our doc: snippetEllipsisText\nIt\u2019s 1 word but you understand 3 different words: \u201csnippet Ellipsis Text\u201d\nLooking at it split up, it makes sense to expect the search engine to be able to return results for the following queries:\n\n\u201csnippetEllipsisText\u201d (original name)\n\u201csnippet Ellipsis Text\u201d (split name)\n\nBut also:\n\n\u201cEllipsis\u201d (middle word only)\n\u201cEllipsisText\u201d (two last words, not split)\n\u201cEllipsisTex\u201d (prefix query of \u201cEllipsisText\u201d)\n\u201cEllipsis Text\u201d (two last words split up)\n\u201cEllipsis snippet\u201d (split up, inverted first and second word)\n\u2026\n\nThere is a few queries where you are not expecting results:\n\n\u201cEllipsisSnippet\u201d (not split inverted first and second","record_index":0},{"post_id":5709,"post_title":"Searching camelCased parameters in API documentation: how we handled it","post_date":1477016985,"post_date_formatted":"Oct 21st 2016","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/searching-camelcased-parameters-in-api-documentation-how-we-handled-it\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/pasted_image_at_2016_10_05_14_45-360x200.png","time_to_read":9,"content":"\u2026 word)\n\u201cTextEllipsis\u201d (not split inverted second and third word)\n...\n\nIn plain words we want to match:\n\nThe exact parameter name (Because people might copy\/paste it from their code to know more)\nAny combination of sub-word of the parameter name split up\nExact parameter name omitting 1 or more starting sub-words\n\nOne of the great features of Algolia is the highlighting. We describe in detail how it works in a previous blog post.\nSo we also expect, when searching for camel case, to have highlighting working correctly, meaning that if I search \u201cellip\u201d I expect to see \u201csnippetEllipsisText\u201d in the result\nFor now we were handling only:\n\n\u201csnippetEllipsisText\u201d (the basic one)\n\u201csnippet Ellipsis Text\u201d because the engine tries to concatenate the query.\n\nThere will be a few search inputs like the one just bellow along the blog for you to try and understand the process. Those inputs will search inside all Algolia parameters (at the time of the writing)\nWorking queries: \"snippetEllipsisText\", \"snippet Ellipsis Text\",\n Not working queries: \"Ellipsis\",\u00a0\u201cEllipsisText\u201d, \"EllipsisTex\", \"Ellipsis Text\", \"Ellipsis snippet\"\n\n\n\nAs you can see from the examples above, that\u2019s 2 out of 7 working, which we can agree is bad.\nWhy we get those results\nUnderstanding why we are handling so few queries out of the box is the key to fixing it properly - let\u2019s dive in. Algolia is doing prefix matches only (more details in this article). It\u2019s one of the reasons Algolia is able to search so fast, but for our camel case use case it\u2019s preventing us from searching in the middle of the word. So we had to find a way around that.\nThe iterative process to fix it\nIndexing the splitted content\nSince we want to be able to search the middle of our camelCaseWords we knew we had to index it as \u201ccamel Case Word\u201d so basically \u201cuncamelizing\u201d the content.\nSo we started to look for existing librairies doing that (in python because the DocSearch scraper is built with python.\nWe","record_index":1},{"post_id":5709,"post_title":"Searching camelCased parameters in API documentation: how we handled it","post_date":1477016985,"post_date_formatted":"Oct 21st 2016","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/searching-camelcased-parameters-in-api-documentation-how-we-handled-it\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/pasted_image_at_2016_10_05_14_45-360x200.png","time_to_read":9,"content":"\u2026 found the stringcase\u00a0library which has a sentencecase function wich does the job of \u201cuncamelizing\u201d but there is two issues with such library:\n\nIt\u2019s working too well :), what I mean by that is it\u2019s going to uncamelize everything, like \u201cAPI client\u201d is going to become \u201cA P I client\u201d, we don\u2019t want that to happen as the brains reads and understand it as \u201cAPI client\u201d not \u201cA P I client\u201d\nA camelCasedWord in the context of a documentation is usually surrounded by text and it\u2019s not allowing us to know which words got uncamelized in the process (more on why we need that information bellow)\n\nSo we had to write our own:\n\nif a letter is preceded by an non-capital alphanumeric character we add a space, fairly simple.\nWith this in place:\n\n\u201csnippet Ellipsis Text\u201d gives the expected results\nwe can now search in the middle of the camelCasedWord\n\nBut:\n\nwe now have a display issue when looking for \u201csnippet Ellipsis Text\u201d\n\u201csnippetEllipsisText\u201d is not returning results anymore\nwe are still not able to have results for \u201cEllipsisText\u201d\nwe can know exactly which word in a sentence was camel cased\n\nWorking queries: \"Ellipsis\",\u00a0\u201cEllipsisText\u201d, \"Ellipsis Text\", \"Ellipsis snippet\",\u00a0\"snippet Ellipsis Text\"\nNot working queries:\u00a0\"snippetEllipsisText\",\u00a0\"EllipsisTex\"\n\n\n\nThat\u2019s 5 out of 7 working, better but still not perfect\nFixing the remaining issues\nThe display issue\nAs mentioned, we now have a display issue. The content we show on the search result for the query \u201csnippet Ellipsis Text\u201d is not the one that you can see in the content and expect in the search result: \u201csnippetEllipsisText\u201d.\nWe came up with a nice trick. We looked for an invisible unicode character: \\u2063 (there are others but this one does the job) to put as a replacement for the space. This make the engine still considering snippetEllipsisText as several words while displaying snippetEllipsisText because the separator is not visible in a browser.\n\n\n\nThe _uncamelize_word","record_index":2},{"post_id":5709,"post_title":"Searching camelCased parameters in API documentation: how we handled it","post_date":1477016985,"post_date_formatted":"Oct 21st 2016","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/searching-camelcased-parameters-in-api-documentation-how-we-handled-it\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/pasted_image_at_2016_10_05_14_45-360x200.png","time_to_read":9,"content":"\u2026 function code now looks like:\n\nLast but not least: the no result issue for \u201csnippetEllipsisText\u201d and \u201cEllipsisT\u201d\nSearching for \u201csnippetEllipsisText\u201d does not bring any result anymore since the index does not contains anymore the word snippetEllipsisText.\nSearching for \u201cEllipsisTex\u201d does not work because the word \u201cEllipsisText\u201d is not indexed, we indexed \u201cEllipsis\u201d and \u201cText\u201d but not \u201cEllipsisText\u201d.\nNote that EllipsisText is returning the expected result because it\u2019s one typo away from \u201cEllipsis Text\u201d, same for \u201cEllipsisT\u201d. It\u2019s better but we would rather have the engine considering it as 0 typo\nFortunately the Algolia engine has a handy synonym feature.\nFirst thing first: \u201csnippetEllipsisText\u201d\nWe can just add a 1 way synonym:\nsnippetEllipsisText => snippet Ellipsis Text\nThen for \u201cEllipsisT\u201d in the end what we want is to have another 1 way synonym:\nEllipsisText => Ellipsis Text\nBut we need this to be generic. If we summarize we want to:\n\ncreate a synonym for the complete name,\nremove the first sub-word and creating a new synonym\niterate until only 1 sub-word remains.\n\nThe following schema should help you understand:\nLet\u2019s consider \u201csnippetEllipsisText\u201d as \u201cA B C\u201d, we are going to create the following 1 way synonyms:\nABC => A B C\nBC => B C\nC => C we actually don\u2019t need this one as it\u2019s already handled by the initial splitting\nYou can have a look at the final code here.\nFinal result:\n\n\n\nHandling camel case seemed like an easy thing, but after having to handle it I can fairly say it\u2019s not that simple after all, because it implies a lot of edge cases.\u00a0The work we did here\u00a0is improving a lot the search for parameters in our doc, and the search for all already live DocSearch implementations.\nOne area where DocSearch doesn\u2019t shine yet is searching in generated api documentation from code like JavaDoc where camel case is omnipresent. This work is is a big step forward it making it","record_index":3},{"post_id":5709,"post_title":"Searching camelCased parameters in API documentation: how we handled it","post_date":1477016985,"post_date_formatted":"Oct 21st 2016","author_name":"Maxime Locqueville","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/3e048a212b69b9885f14b8c628198c8e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/searching-camelcased-parameters-in-api-documentation-how-we-handled-it\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/pasted_image_at_2016_10_05_14_45-360x200.png","time_to_read":9,"content":"\u2026 available.","record_index":4},{"post_id":5587,"post_title":"Inside the Algolia Engine Part 5 \u2013 Highlighting, a Cornerstone of Search UX","post_date":1475500709,"post_date_formatted":"Oct 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-5-highlighting-a-cornerstone-to-search-ux\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/highlight-360x200.png","time_to_read":9,"content":"Visually highlighting search results is a must-have component of a great search experience. This is even truer when you start to do advanced processing on the query (synonyms, typo tolerance, concatenation\/split of query words, etc.), like we presented in the third installment of this series.\nA search result that is considered as weird by a user without highlighting can become smart just by explaining the processing done to retrieve it and by making it easy for the user to check if this is exactly the content they were looking for. In this article, we\u2019ll show you in detail how we have implemented our highlighting to make sure it always provides a great user experience.\nOn this Google query, the first hit shows the standard highlighting done by Google. We removed the highlighting manually on the second hit - the result is much more difficult to understand.\nDifferent approaches\nHighlighting tends to be a subject that appears easy at first glance; however, the reality is much more complex, namely because it is a different process entirely than that of matching & ranking your objects. There are three main ways to implement highlighting:\n\n1. Take the query and the text to highlight and imitate the job of the search engine. This approach is partial as you don't have access to the processing done by the search engine like the extraction of synonyms. Usually this means that you just highlight the query terms, which will be misleading for users as they will not see why a record was found.\n2. Apply the query interpreter on the query to extract all possible extensions like synonyms and use that information to highlight a text. This approach will give you a good visual result as you will have all the alternatives. But you will have to test a lot of expressions that do not match against your text. There is a lot of waste of performance here.\n3. Apply the query in the search engine as usual but keep the matched terms for each result. This list of matched terms will be used","record_index":0},{"post_id":5587,"post_title":"Inside the Algolia Engine Part 5 \u2013 Highlighting, a Cornerstone of Search UX","post_date":1475500709,"post_date_formatted":"Oct 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-5-highlighting-a-cornerstone-to-search-ux\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/highlight-360x200.png","time_to_read":9,"content":"\u2026 by the highlighter to process a record. This approach offers the best of two worlds: you have exactly the expected highlight whereas the highlighter remains fast and only focuses on the expression that is in the record.\n\nThe big advantage of the last two approaches is that your highlighter will be easier to implement. You don\u2019t have to worry about alternatives like synonyms or typo-tolerance as it will already be resolved by your query interpreter. In other words, if your record matches with a typo, you will have the word with a typo as an input of the highlighter.\nIn the Algolia engine, we have used the third approach since day one. It was actually one of the many reasons to redevelop the Algolia engine from scratch. We had already developed several highlighters in the past and we knew from experience the third approach would be the best; however, we had to to keep all matched terms for each record, which needs to be done in a very efficient way in order to not create a bottleneck in term of CPU or RAM.\nDifferent expressions we highlight\nThere are four types of expression that the highlighter can highlight:\n\n1. A word: in this case, we need to find all tokens in the text to highlight that are identical to this word (with an accent and case-insensitive comparison).\n2. A prefix: in this case, we need to find all tokens in the text to highlight that start with this prefix (again with an accent and case-insensitive comparison). Usually, this word corresponds to the last query term that is matched as a prefix; however, it can also contain a typo (as we support typo tolerance on prefixes).\n3. A phrase: in this case, we need to find a sequence of words in a specific order in the record (also with an accent and case-insensitive comparison).\n4. A prefix phrase: identical as a phrase, except that the last word of the phrase can be matched via a prefix.\n\nAll those expressions come from the search engine and are an input for the highlighter, for example the user query","record_index":1},{"post_id":5587,"post_title":"Inside the Algolia Engine Part 5 \u2013 Highlighting, a Cornerstone of Search UX","post_date":1475500709,"post_date_formatted":"Oct 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-5-highlighting-a-cornerstone-to-search-ux\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/highlight-360x200.png","time_to_read":9,"content":"\u2026 \"searchengine\" contains only one term but will also add the alternative phrase \"search engine\" which is the result of our split of query tokens processing (described in the third article of this series).\nExplaining the result using query terms\nThe highlighter is not just the process that adds some tags around matching expressions, it plays a bigger role in the user experience. You have potentially dozens of attributes in your objects used for search, displaying all of them would give too much information to the user. You only have to show the relevant one to explain the result.\nFor example, if you are typing the query \"Twilio IPO\" on your favorite news site, you will have several objects that will match. Some with both terms in the title like this one:\n\nAnd some with only one term in the title like this one:\n\nOn the first one, the highlighter will give you the information that all query terms were found in the title attribute (via the `matchLevel=full`), which allows you to consider a specific display of this article in the UI as only the title is required to explain the result.\nHere is the highlighter information on the title of the first article:\n\nOn the second article, the highlighter will give you the information that the title attribute is partially matching the query (\"matchLevel=partial\").\n\nThe highlighter gives you all information needed to explain the query, you can scan all attributes in the highlighter and only select the ones that \"explain\" one query term that no other one explains. Most of the time, you don\u2019t have enough room to show every title and its content, in this case the highlighter will help you to show the content only when it's relevant to explain the result. This approach of explaining search results plays a big role in user engagement and improvement of your ROI on search.\nAn example of a search query where several attributes are required to explain the result: the movie title & the actor name.\nComputing a snippet\nWhen the text of an","record_index":2},{"post_id":5587,"post_title":"Inside the Algolia Engine Part 5 \u2013 Highlighting, a Cornerstone of Search UX","post_date":1475500709,"post_date_formatted":"Oct 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-5-highlighting-a-cornerstone-to-search-ux\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/highlight-360x200.png","time_to_read":9,"content":"\u2026 attribute contains more than a few words like the content of a news article, you want to summarize it by keeping only the important sections. The result of this process is called a snippet and there are a lot of different approaches, so every snippet computation is different.\nIn Algolia, the snippet computation relies on the highlighting.\nThe first step of the process is to flag each token of the text to snippet with the corresponding query term that matches it. Then the second step is to find the window of N tokens that maximise the number of different query terms matched. You can have several windows of N tokens that contains the same number of highlighted terms, in this case we prefer to leave some text before and after the matching terms to give some context to the user.\nAlgolia lets you customize the number of words in a snippet, as this parameter depends on the UI you are building. In the example below, we will use 10 words for the \u00a0description of the two articles:\n\n\nThe two snippets actually return the 10 first words of the content as there is no 10 words window that contains both terms.\nYou can note that we do not return matchedWords attribute in the snippet as the result is partial. You need to use the highlighter to fully explain a result but you can, of course, request to have both the highlighted version and the snippet version.\nHow the engine identifies matching words efficiently\nOur highlighter is exhaustive while having very good performance, a big part of the hard work is actually done in the search engine itself in the identification and storage of all matched words for an object.\nDuring the query processing, we compute all expressions to find and create a description object for each of them that contains the expression to highlight and the link to the original query token. At the end of the query processing, we have a vector with all those alternative expressions.\nThen, when we create the boolean query that will be applied to the index, we keep","record_index":3},{"post_id":5587,"post_title":"Inside the Algolia Engine Part 5 \u2013 Highlighting, a Cornerstone of Search UX","post_date":1475500709,"post_date_formatted":"Oct 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-5-highlighting-a-cornerstone-to-search-ux\/","categories":["Guides","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/10\/highlight-360x200.png","time_to_read":9,"content":"\u2026 the link to the original expression. For example if you have the query \"Hotel NY\" with a synonym \"NY\" = \"New York\", the expression vector would be:\n\nWe would execute the following query:\n\nWhen a record matches, we know which part of the expression matched (list of integers). For example:\n\n\n\nA record containing \"hotel\" and \"NY\" will have a vector containing [0, 1]\nA record containing \"hotel\" and \"new york\" will have a vector containing [0, 2]\nA record containing \"hotel\" and \"NY\" and \"new york\" will have a vector containing [0, 1, 2]\n\n\n\nThis example is very simple as we have a very small number of expressions to match. In reality, we usually have hundreds of expressions to try because of typo tolerance and it becomes critical to identify only the one found in the record.\nWe finally keep this vector of integers for each result to be able to produce the list of terms to highlight and the list of matched words.\nWhy search without highlighting is bad\nAs soon as you have advanced query interpretation,highlighting becomes essential to a good \u00a0user experience. Having a great experience is more than highlighting the most important attribute, it is searching and displaying all attributes that are important for the user to understand why the result was displayed. This is key to help the user quickly decide which result they\u00a0will choose first. Without this aspect, you will leave your user disappointed as they will inevitably choose a bad result.\nI hope this explanation has underscored why highlighting is such a complex and important topic of any search engine!\nWe recommend to read the other posts of this series:\n\nPart 1 \u2013 indexing vs. search\nPart 2 \u2013 the indexing challenge of instant search\nPart 3 \u2013 query processing\nPart 4 \u2013 textual relevance\nPart 6 \u2013 Handling Synonyms the Right Way\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":4},{"post_id":5574,"post_title":"Unveiling Algolia\u2019s WordPress search plugin - blog search at the speed of thought","post_date":1475014866,"post_date_formatted":"Sep 27th 2016","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/unveiling-algolias-wordpress-search-plugin-blog-search-at-the-speed-of-thought\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/wordpress-1-360x200.png","time_to_read":5,"content":"Today we are very excited to announce our Search by Algolia plugin for WordPress which will bring relevancy, instant search and autocomplete capabilities to almost 30% of the internet which is powered by WordPress.\nWe believe that every website or blog, irrespective of size, deserves good search so that users can easily navigate and find relevant content. \nWith our new plugin, every WordPress user will be able to leverage Algolia in just a few clicks, and replace the default WordPress site search with a blazing fast, highly relevant search engine.\nToday\u2019s default WordPress search and most other search plugins out there are based on simple SQL queries, which aren\u2019t high on the relevancy quotient, adds unnecessary load to your hosting environment, and slows down your site. They weren\u2019t designed to handle complex user needs, such as typo tolerance, and they weren\u2019t designed to scale with your content (in fact, they do just the opposite). We think the Search by Algolia plugin will solve these problems.\nEasy Set-up and Automatic Updates\nSince Algolia is a completely hosted service, you initially need to send the data. The plugin handles this automatically for you and lets you index the following content types by default:\n\nPost-Types: Including Posts, Pages and custom registered post types,\nTaxonomies: Including Categories, Tags and any custom registered taxonomy,\nUsers: All the contributors of the website.\n\nEvery time you update Posts, Taxonomy terms and Users, we automatically synchronize the data for you. \nNo matter how many blog posts, categories, tags, or users you have, the plugin\u2019s built-in queue mechanism allows it to handle any amount of data.\nAlgolia makes it easy for you to customize ranking and configure indices in order to achieve optimal relevancy. When you use the Search by Algolia plugin for WordPress, we take care of everything for you, so you get the best search experience out of Algolia without any of the setup hassle.\nOut-of-the-box","record_index":0},{"post_id":5574,"post_title":"Unveiling Algolia\u2019s WordPress search plugin - blog search at the speed of thought","post_date":1475014866,"post_date_formatted":"Sep 27th 2016","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/unveiling-algolias-wordpress-search-plugin-blog-search-at-the-speed-of-thought\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/wordpress-1-360x200.png","time_to_read":5,"content":"\u2026 Autocomplete and Drop-down menus\nMost WordPress themes display a search bar on every page of the your website by default. By enabling autocomplete, we add instant results at each keystroke in every one of your search bars.\nHere is what it looks like on SaaStr\nOur plugin provides you with a beautiful default layout for your dropdown results, and the templates can be completely customized, of course.\nBringing instant search results page to your WordPress\nOur plugin provides you with a fully featured instant-search results page. You simply have to enable the instant-search feature in the plugin and it will replace the default search page with a search page that includes filters and displays results as-you-type!\nThe newly provided search page will allow users to filter the content of your website based on content types, categories, tags or even authors.\nOf course the default appearance and behavior can be fully customized with just a little knowledge of CSS and JavaScript. If you're not comfortable customizing your WordPress UI, we\u2019re confident you\u2019ll be happy with the default design!\nOur Plugin is designed with the WordPress developer community in mind\nHere is what it looks like on ForEntrepreneurs.com\nThe whole plugin has been built with extensibility in mind and can be used as the foundation for every Algolia search experience on WordPress websites.\nHere are a few highlights:\n\nEasy to override templates (without requiring editing of the actual plugin files),\nOver 30 WordPress action & filters to customize and extend functionality,\nBuilt-in logging system to help understand everything that happens when building your own search experience,\nEvery feature can be enabled\/disabled so that you can leverage the features that matter for your project,\nLast but not least, the plugin is thoroughly documented here.\n\nThe plugin is open sourced on GitHub. We invite anyone to submit feature requests or bug reports by opening issues on the repository:","record_index":1},{"post_id":5574,"post_title":"Unveiling Algolia\u2019s WordPress search plugin - blog search at the speed of thought","post_date":1475014866,"post_date_formatted":"Sep 27th 2016","author_name":"Raymond Rutjes","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6897886507394280b20f02b64cb37c73?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/unveiling-algolias-wordpress-search-plugin-blog-search-at-the-speed-of-thought\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/wordpress-1-360x200.png","time_to_read":5,"content":"\u2026 https:\/\/github.com\/algolia\/algoliasearch-wordpress\nWe haven\u2019t forgotten about you, WooCommerce users!\nIf you are a WooCommerce user, we\u2019ve got you covered. We\u2019re about to release an extension to this plugin, so stay tuned for the Search by Algolia for WooCommerce plugin by subscribing here (We will only send you 1 email to inform you of the release) \nWe wanted to mention beta testers and developers without whom this plugin wouldn\u2019t be out yet!\n\nGaadiKey: https:\/\/blog.gaadikey.com\nFinanzdiskurs: http:\/\/finanzdiskurs.de\/\nWooNinjas: https:\/\/wooninjas.com\/\nMagnolia: https:\/\/magnoliamarket.com\/\n\n\nTom Witkowski: https:\/\/github.com\/Gummibeer\nRahil Wazir: https:\/\/github.com\/rahilwazir\nAndrei Serban: http:\/\/andreiserban.com","record_index":2},{"post_id":5546,"post_title":"Why we're returning to Mind The Product for the second time this year","post_date":1473677850,"post_date_formatted":"Sep 12th 2016","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/why-were-returning-to-mind-the-product-for-the-second-time-this-year-2\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/indexing-part2-copy-8-1-360x200.png","time_to_read":3,"content":"Earlier this year we participated in our first Mind the Product event in San Francisco as an exhibitor, and we\u2019re excited to be returning this month to their London edition. We first discovered Mind the Product when we were looking at ways to get better acquainted with the product manager community, and we\u2019ve been utterly blown away by our experience.\n\nThe event itself brings together some of the greatest minds behind product innovation today - as you can see in the recap of their San Francisco event above (in addition to some nice Algolia love at the end of the video), the event brings together an increasingly important (& large) section of the technology workforce, a workforce that we\u2019ve been increasingly working with as we roll out Algolia to larger & larger companies.\nFor sponsors, the Mind the Product team is very committed to making sure that we get the most out of our experience, without compromising on the overall experience of the event. The speakers are hand-picked, and there\u2019s plenty of time to attend sessions and meet with fellow attendees.\nAt our first edition, we made 100+ meaningful connections - entities as big as the Staples & Wells Fargo as well as burgeoning companies. Product managers are increasingly becoming champions inside their teams for new technologies, approaching product development from the user\u2019s perspective. We\u2019re committed to great Search UX, and product managers know how important that is.\nOne of the things we love about Mind the Product is that, as an exhibitor, we feel so special, because there are only a handful of select partners who exhibit. While we had originally planned on balancing team members at our stand and team members moving around the event to see what else was going on, we ultimately ended up with so many people on our stand that we had to keep all four of us on the stand to field questions and make new friends.\nFor this edition, we\u2019re sending four people again - over 1,000 attendees came","record_index":0},{"post_id":5546,"post_title":"Why we're returning to Mind The Product for the second time this year","post_date":1473677850,"post_date_formatted":"Sep 12th 2016","author_name":"Liam Boogar","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46a5c344feb37b87dcb143854f461c35?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/why-were-returning-to-mind-the-product-for-the-second-time-this-year-2\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/09\/indexing-part2-copy-8-1-360x200.png","time_to_read":3,"content":"\u2026 to San Francisco, and the London event already sold out as well - our goal will be to have 100 meaningful conversations. In addition, we\u2019ll be making the most of our time in London by speaking at a number of other meetups in London, like the Elastic London User Group & UX Connect at Google Campus London.\n\nA few tips to optimize your experience\nWhen we talk about Algolia, we often dive right into the technical aspects of search - the advantages & disadvantages of various implementations & options - however, we figured out quickly at MindTheProduct that it was much more important to focus on the impact that better Search UX has on your core business - conversion, retention, user experience - as we were mostly speaking with product manager & VPs of product, who have a much more high-level look at the evolution of their product.\nFor PMs and VPs of Product, Algolia often serves as a partner in helping them get their engineering team excited about using Algolia. Understanding how product managers fit inside of development teams is key to becoming a partner, and ultimately in helping them improve their product.\nA final tip is to give attendees something they can put in their pocket (or wear) and remember you by. Edible goodies are good for attracting crowds, but if you want to stay in their mind, give them something branded that they can use for weeks to come - mugs, bottle openers, even a backpack - and you\u2019ll stay with them through at least breakfast (for the mug, of course!).\nIf you\u2019ll be attending MindTheProduct or will be in London and want to connect with us, you can find us on our stand or ping us @Algolia and we can talk about Search & the future of Product Development!","record_index":1},{"post_id":5487,"post_title":"Search Party recap: 60+ attendees, 10 projects, 1 \"magnificent\" photo booth","post_date":1472133793,"post_date_formatted":"Aug 25th 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-party-recap-60-attendees-10-projects-1-magnificent-photo-booth\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/search-party-360x200.png","time_to_read":4,"content":"Several weeks ago\u00a0we welcomed a group of Algolia developers, customers and partners into our SF office to\u00a0have a conversation\u00a0about search. It was also an opportunity to say \"Hey, thanks for working with us!\"\n\nFireside Chat\nThe evening kicked off with\u00a0a fireside chat between\u00a0Algolia co-founders Nicolas Dessaigne and Julien Lemoine. Some fun facts came out, like\u00a0Nicolas and Julien having a combined 27 years of experience working on search.\u00a0The conversation touched on\u00a0company history, culture, and a\u00a0few technical topics\u00a0including\u00a0our new premium SLA and the assistance we provide to\u00a0customers for\u00a0relevance tuning.\n\nDuring the Q&A someone asked \"How many records is Algolia storing today across all applications?\" \u201cWe don\u2019t know\u201d answered\u00a0Julien, the candid response\u00a0getting a nice\u00a0chuckle from the crowd. He added\u00a0\u201cToday we count overall operations but not records, but maybe someday we will count those too.\u201d\nCommunity Projects\nAfter the fireside chat I dismissed\u00a0the crowd for a break\u2014to get up, stretch their legs and grab a drink\u00a0from our (deserted) island-themed bar. This was actually a mistake :\/ It took several minutes to get people to quiet down and return to\u00a0their seats. Lesson learned for next time!\nOnce things settled down I gave a presentation called 10 Community Projects in 15 Minutes.\u00a0At Algolia we believe search is a conversation, a dialogue that helps you\u00a0learn what your users want. Each of these\u00a010 projects helps facilitate that conversation in some way:\n\nAlgolia Places\nDocSearch\ninstantsearch.js\nAlgolia JS Helper\nJekyll plugin\nZendesk plugin\nMagento plugin\nWordPress plugin\nOval Quotes\nSearchstone\n\nWhat do these projects have in common? They either\u00a0use Algolia or help you use Algolia, and they're\u00a0all looking for feedback, beta testers or\u00a0contributors.\nPhoto Booth\nWe handed out brand new\u00a0community-themed stickers and t-shirts\u2014and then there was the photo booth \ud83d\ude42 The booth was\u00a0stocked with every search-related","record_index":0},{"post_id":5487,"post_title":"Search Party recap: 60+ attendees, 10 projects, 1 \"magnificent\" photo booth","post_date":1472133793,"post_date_formatted":"Aug 25th 2016","author_name":"Josh Dzielak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/52fce504cc701577cb6ca94528b977e6?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-party-recap-60-attendees-10-projects-1-magnificent-photo-booth\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/search-party-360x200.png","time_to_read":4,"content":"\u2026 knickknack\u00a0we could get our hands on, as well as a few heavy-duty magnifying glasses\u00a0I found at\u00a0Pottery Barn.\nWho needs\u00a0Snapchat when you have old school photo filters:\nThat's Yonas from StackShare trying to hide back there\nWhat are *you* looking at?\n \n\nLast\u00a0Call\nPhotos of the event are available on our Facebook page in two different albums: Search Party #1 and Search Party #1 Photo Booth. If you couldn't make it this time don't worry, just stay tuned to hear about the next one.\nBig thanks to everyone who came out and helped make it a very special evening. All of us at Algolia really enjoyed hanging out and hearing from you. Until the next #searchparty! ??","record_index":1},{"post_id":5464,"post_title":"Bringing Advanced Search to Magento 2","post_date":1471851883,"post_date_formatted":"Aug 22nd 2016","author_name":"Jan Petr","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/63ca90bb47b1383bd24caf2cc102ceb4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-autocomplete-synonyms-instant-results-to-magento-2\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Pasted-image-at-2016_08_22-16_49-360x200.png","time_to_read":5,"content":"When it comes to eCommerce, the goal of search is to read the customer\u2019s mind. When we released Algolia for Magento 1.0 last year, that\u2019s exactly what we\u00a0wanted to do.\u00a0Algolia for Magento 1.0 has been\u00a0starred over 100 times, and thanks to our amazing community, we learned a lot about how we can improve the shopping experience on Magento.\nToday we\u2019re releasing our search extension for Magento\u00a0to the growing Magento 2 community, bringing the same benefits of Algolia search to Magento 2 stores.\nWhat we learned from Magento 1.0\nSince\u00a0releasing Magento 1.0 last year, our Magento community has grown to over 190\u00a0customers making\u00a06 million queries per day. We\u2019ve also seen a lot of interest from our very active Magento community on how we can make our extension even better. \nThrough a series of feature (and pull) requests, we were able to refine and iterate on Magento 1.0 to make it the best way to improve search in your store:\n\nWe cleaned up some code and tried to make the extension as developer-friendly as possible\n\n\nWe worked on our extension\u2019s ability to synchronize with Algolia in real-time\n\n\nWe implemented the most common Algolia features directly onto the Magento dashboard so that customers could manage their Algolia settings in the same familiar environment they're used to. For instance our Synonyms feature is available directly on the Magento dashboard.\n\nWe also received a lot of requests for an extension for Magento 2. And we thought, why not? Today, we\u2019d like to introduce you to the all new Magento 2 extension.\nUpgrading the Magento Search Engine\nOur Algolia extension\u00a0works with\u00a0Magento 2 just as well as\u00a0Magento 1.0 - \u00a0it replaces Magento\u2019s default search engine,\u00a0which\u00a0uses simple SQL queries, but lacks\u00a0basic features like\u00a0autocomplete, fuzzy search or partial matches. We\u00a0make it easier for shoppers\u00a0to find what they are looking for.\nOur plugin also scales to any size of store, so you can have instant results even with very large","record_index":0},{"post_id":5464,"post_title":"Bringing Advanced Search to Magento 2","post_date":1471851883,"post_date_formatted":"Aug 22nd 2016","author_name":"Jan Petr","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/63ca90bb47b1383bd24caf2cc102ceb4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-autocomplete-synonyms-instant-results-to-magento-2\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Pasted-image-at-2016_08_22-16_49-360x200.png","time_to_read":5,"content":"\u2026 inventories. Some of our customers are searching millions of SKUs.\nMagento 2 is completely synchronized with Magento 1.0 with regard to features, so if you\u2019re looking to migrate your store onto the newer platform, the transition will be seamless.\nHere\u2019s a look at some of the features we\u2019ve incorporated onto our Magento advanced search extension:\nOut-of-the-box autocomplete\nAutocomplete or \"find-as-you-type\" search experience always makes the search more intuitive and goes a long way in shortening the distance between the end user and relevant content on the website. \nHow could we\u00a0NOT include auto-complete for Magento 2?\nClick the GIF to check out our live Magento 2 demo to see our extension in action!\nInstant-search results page\nWouldn\u2019t it be awesome if search results were automatically updated for your customers with every subsequent letter of their search query? That's exactly what instant search\u00a0gives you.\nInstant search not only refines search results with each keystroke, but also updates filtering options such as price range, stock availability, colors, sizes and many others. With every letter they\u00a0type, shoppers get constantly refined results.\n\nSynonyms\nHave you ever tried looking for a TV but come up with absolutely nothing because it was called \u201cTelevision\u201d, not TV? \nWe know the feeling. \nThat\u2019s why we integrated\u00a0Algolia\u2019s Synonyms feature directly into the extension - you\u00a0can manage your synonyms right from your store\u2019s admin interface!\n\nIndexing your Products\nIndexing your products is as easy as 1, 2...3. After you install the extension and synchronize your indices for the first time, you won\u2019t have to worry about synchronizing data ever again. The extension uses Magento\u2019s hooks and every time you add \/ update \/ delete a product in your Magento dashboard it will automatically be synchronized with the indices.\nInstalling\u00a0the Magento 2 extension takes under 10 minutes with a just few lines of code - or you can upgrade your store","record_index":1},{"post_id":5464,"post_title":"Bringing Advanced Search to Magento 2","post_date":1471851883,"post_date_formatted":"Aug 22nd 2016","author_name":"Jan Petr","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/63ca90bb47b1383bd24caf2cc102ceb4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-autocomplete-synonyms-instant-results-to-magento-2\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Pasted-image-at-2016_08_22-16_49-360x200.png","time_to_read":5,"content":"\u2026 search directly through the Magento Marketplace.\nJoin the Algolia + Magento community\nWe\u2019re on a constant quest to improve the quality of our products here at Algolia, and like last time, we\u2019d love your feedback and support to take the Magento 2 search experience to the next level.\n\"The Balance Internet team love using and recommending Algolia Search-As-You-Type extension on all our Magento 1 and Magento 2 projects. The extension is easy to install, configure and customise for our clients. The search functionality delivers a fantastic user experience that is lightning fast and relevant. Our customers love being able to include CMS pages, categories and products in their Magento site search. In the past 7 years, I haven't seen another Magento search extension \/ platform come close to Algolia from a performance, functionality and price perspective.\"\n-- Kieran Smith, Solutions Architect @ Balance Internet - Australia's most experienced Magento Partner\nIf you are a Magento store owner, developer, system integrator... or just someone interested in our Magento extension, we want to hear from you! You can also try a demo with our extension here.\nDrop us a line at magento@algolia.com to become a member of the our Magento community\u00a0and\u00a0get exclusive priority access to trainings, newsletters and even workshops focused on our latest Magento related news and updates.\nThe extension is completely open-source and on GitHub. Feel free to fork the extension, send a pull request or open an issue. Any tweaks you suggest are highly welcome. Let's make the Magento world more navigable for shopaholics everywhere!","record_index":2},{"post_id":5415,"post_title":"For SLAs, there\u2019s no such thing as 100% Uptime - only 100% Transparency","post_date":1470793213,"post_date_formatted":"Aug 10th 2016","author_name":"Adam Surak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/0d23f348c8416b65e5acf54cd52edf57?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/for-slas-theres-no-such-thing-as-100-uptime-only-100-transparency\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-04-at-10.39.41-360x200.png","time_to_read":7,"content":"With the advent of cloud computing and its ubiquity over the past 10 years, the SaaS model has, brick-by-brick, revolutionized the way businesses all over the world operate - from processing payments to processing paychecks in HR, from measuring marketing ROI to boosting sales efficiency, the modern enterprise\u2019s go-to reflex when looking at improving how it operates is to turn to SaaS.\nWhile SaaS is undeniably an improvement over the former software licensing model, which came with bulky infrastructure, on-site maintenance, and less global scalability, SaaS is a global light switch that can instantly turn your business on and deliver it to literally billions of customers. But light-switches go both ways, and, as any enterprise that has ever suffered an outage knows, SaaS can cost big-time.\nJust recently, Amazon\u2019s own search went down for a few hours - even the most conservative estimates suggest that the outage cost them tens of millions in revenue. Outages like these are inevitable; however, the strongest SaaS providers today invest heavily both in strong uptime and having a strong SLA, so that customers know that, in the event that an outage impacts their business, that they have insurance.\nToday, I\u2019d like to talk a little bit about our brand new SLA policy, and how it came to be.\nWhat\u2019s an SLA?\nBefore getting started, just to make sure we\u2019re all on the same page, let\u2019s get our vocabulary straight. \nA Service Level Agreement (SLA) is essentially an insurance policy in case the service you paid for doesn\u2019t operate as advertised. In general, the more critical a service is to a company\u2019s operations, the stronger the SLA will need to be to satisfy customer worries. Likewise, a stronger SLA is a stronger commitment from a service provider to ensure flawless service.\nAs a SaaS player and a customer of several external services for both our search platform and our operations, as a company we\u2019ve seen (more than) our fair share of SLAs. As we evolved and","record_index":0},{"post_id":5415,"post_title":"For SLAs, there\u2019s no such thing as 100% Uptime - only 100% Transparency","post_date":1470793213,"post_date_formatted":"Aug 10th 2016","author_name":"Adam Surak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/0d23f348c8416b65e5acf54cd52edf57?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/for-slas-theres-no-such-thing-as-100-uptime-only-100-transparency\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-04-at-10.39.41-360x200.png","time_to_read":7,"content":"\u2026 improved our own SLA over the past two years - providing an increasingly strong & transparent promise to our customers - we began combing through SLAs with a fine-toothed comb. It turns out, not all SLAs were created equal.\nBusting the myth of the 100% SLA\nIn recent years it\u2019s become fashionable for companies to include 100% uptime guarantees in their SLA - and, in some cases, even more than 100%, despite its mathematical impossibility.\nNow, don\u2019t get us wrong - all service providers have an obligation to put 100% of their effort into keeping their service running like a well-oiled machine; however, the detection of an outage itself can sometimes even be impossible\u2026 until it\u2019s too late, of course.\nBeing a SaaS provider on the internet implies dozens of dependencies on intermediary devices and networks, which themselves have downtime. When you promise 100% uptime, every millisecond of downtime counts - however, what if you can\u2019t detect the outage? How can one tell if the issue comes from your connection dropping data, the service provider, or any of the dozens of intermediaries in between?\nTo resolve this issue, SLAs define a minimum outage necessary in order to be triggered. The market standard is typically 1 minute; however, 1 minute of downtime per month means 99.9977% uptime - so what exactly is 100% uptime then?\nLet\u2019s take a look at an example of a \u201c100% SLA\u201d from a SaaS provider on the market today:\n\u201ccredit of 5% of the fees paid for the month in which we fail to provide the stated level of service for every 0.05% of such a month, during which you are unable to transmit information up to an aggregate of 50% of the monthly service billing\u201d\nThis SLA only kicks in after 0.05% downtime in a month, which is a little over 20 minutes of downtime - and yet, that same SLA claims 100% uptime guaranteed.\nHow we define our Enterprise SLAs\nWhen we set out to design our SLA, we had three goals:\n\nMake it Simple - you only ever look at an SLA before","record_index":1},{"post_id":5415,"post_title":"For SLAs, there\u2019s no such thing as 100% Uptime - only 100% Transparency","post_date":1470793213,"post_date_formatted":"Aug 10th 2016","author_name":"Adam Surak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/0d23f348c8416b65e5acf54cd52edf57?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/for-slas-theres-no-such-thing-as-100-uptime-only-100-transparency\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-04-at-10.39.41-360x200.png","time_to_read":7,"content":"\u2026 purchasing and when there is downtime, and, in both instances, simplicity is key. \n\nMake it Transparent - no one wants unexpected surprises, especially if there is downtime. The easiest way to lose a customer is to let them down.\nTrust our platform - we trust the the system we built, and we want an SLA that speaks to that trust.\nAt Algolia, we currently have two different setups for our customers:\n\nEnterprise: we replicate your search on at least three different machines hosted by two different providers in different datacenters and autonomous systems\nPremium: we replicate your search on at least three different machines hosted by three different providers in three different data centers with three autonomous systems using at least two different Tier1 upstream providers.\n\n(Did we mention how inevitable dependency on intermediary services are?)\nThese setups are not different just on paper but they\u2019re also different in terms of infrastructure and come with two different SLAs:\n\nEnterprise: 99.99% uptime, each minute of downtime would make you eligible for 100 minutes of refund, up to a cumulative value of 100% of the monthly service billing.\nPremium: 99.999% uptime, each minute of downtime would make you eligible for \u00a01,000 minutes of refund, up to a cumulative value of 600% of the monthly service billing over a year.\n\n \n\n \nOur outage detection starts at 30 seconds (0.001% of a month) instead of 1 minute. This is so granular that it can\u2019t be measured with traditional monitoring architecture, so we built our own monitoring network that continuously monitors our API infrastructure, that gives us a fairly unique ability to detect downtime this fast.\nHere\u2019s what our refund policy looks like in practice:\n \n\n \n\n\n\nSearch down time\nTotal refund of the monthly service bill\n\n\nEnterprise SLA\nPremium SLA\n\n\n30 seconds\n0%\n1%\n\n\n1 minute\n0%\n2.3%\n\n\n5 minutes\n1%\n11.6%\n\n\n30 minutes\n7%\n70%\n\n\n45 minutes\n10%\n100%\n\n\n1 hour\n13.8%\n138%\n\n\n2 hours\n27.7%\n277%\n\n\n4","record_index":2},{"post_id":5415,"post_title":"For SLAs, there\u2019s no such thing as 100% Uptime - only 100% Transparency","post_date":1470793213,"post_date_formatted":"Aug 10th 2016","author_name":"Adam Surak","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/0d23f348c8416b65e5acf54cd52edf57?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/for-slas-theres-no-such-thing-as-100-uptime-only-100-transparency\/","categories":["Infrastructure","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-04-at-10.39.41-360x200.png","time_to_read":7,"content":"\u2026 hours\n55.5%\n555%\n\n\n8 hours\n100%\n600%\n\n\n\nAs you can see, with our Premium SLA, if our service is down 45 minutes, we refund you 100% of your monthly bill - it doesn\u2019t get much simpler than that. \nSLA\u2019s are more than just insurance\nMost people don\u2019t really see SLA as much more than a form of SaaS insurance - at Algolia, we see it as something much greater, a way to remind our customers of our reliability. We back our Premium SLA with our reinforced infrastructure and our goal is to make sure we provide the best service in the market - we don\u2019t want downtime any more than you do, and we put our money where our mouth is. We incentivize ourselves to do everything possible to ensure that the probability of an outage is as close to zero as possible! \nIt has been a\u00a0year since we introduced our three provider set-up, and, with it, we\u2019ve been able to placate the worries of even the most cautious of customers. \nOur setup has been extensively tested - with outages of entire datacenters and networks - and we\u2019ve still been able to maintain 100% uptime.\nTo the best of our knowledge, our Premium SLA is unique to the market - in terms of simplicity, transparency & refund guarantee - \u00a0and we\u2019d love to tell you more about it if you have any questions, or would like to see how your current SLA stands up against ours!","record_index":3},{"post_id":5418,"post_title":"Behind the Scenes: Algolia Places","post_date":1470360617,"post_date_formatted":"Aug 5th 2016","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/behind-the-scenes-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-05-at-10.31.14-360x200.png","time_to_read":10,"content":"All of us have at some point or the other been stuck in the middle of nowhere, desperately trying to locate an address, with our \u201cfat fingers\u201d for company; or abandoned a purchase in our virtual shopping baskets\u00a0because it was too much trouble to fill out\u00a0our entire address. \nAlgolia being Algolia, we wondered how we could reduce instances of this in our own way and Algolia Places was born.\nIn case you haven\u2019t heard, Algolia Places allows you to create beautiful address auto-completion menus with just a few lines of code; it can also be personalized and customized as per your use-case, not to mention enriched with additional sources of data. Today, we\u2019d like to share the story of how we built Algolia Places with you. \nHow did we do it?\nStep 1: Data Collection\nTo build Algolia Places we relied on the open-source datasets provided by OpenStreetMap & GeoNames. These datasets are actually very different from each other and we chose them for precisely this reason.\n\nThe OpenStreetMap dataset contains map data: it basically constitutes of the geo representation (polygons, lines & points) of about 200 million geographical features.\nThe GeoNames dataset contains geographical names of about 9 million features: it\u2019s a \u00a0regular list in the TSV format and contains names of every single city\/country\/place associated with some meta-data (population, zip codes, ...).\n\nStep 2: The Indices\n\n\nThe city & country index\n\n\nIn order to build the city and\u00a0country search experience, we exclusively used the GeoNames dataset - it\u2019s a pretty exhaustive list and is quite simple to parse.\nFor every single city & country name on the TSV files, we created an Algolia record. These records not only include the variations and translated names of countries & cities (if available), but also some meta-data such as associated postcodes, populations, location and various tags & booleans we use for the ranking\/filtering capabilities of our API.\nThe result? ~4 million","record_index":0},{"post_id":5418,"post_title":"Behind the Scenes: Algolia Places","post_date":1470360617,"post_date_formatted":"Aug 5th 2016","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/behind-the-scenes-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-05-at-10.31.14-360x200.png","time_to_read":10,"content":"\u2026 records.\nThe address indices\nTo build the address search experience (including countries, cities, streets & notable places), we used both OpenStreetMap and GeoNames. \nThe OpenStreetMap initiative is wonderful but the underlying data is not always on point.\u00a0Why?\n\nIt may not always be exhaustive (especially for non-famous places)\nIt may sometimes include erroneous values (from our experience, postcodes used to be wrong)\nIt may contain duplicates\nIt may not always follow the same conventions across countries\nIt may lack some internationalisation\/translations\n\nTo convert the OSM map data into Algolia records, we imported the whole OSM planet (40GB compressed) inside a PostgreSQL+PostGIS engine. The resulting database was a whopping 600GB!\nWe then used Nominatim to export this huge database to an XML format, rebuilding the hierarchy of places. This actually brought an interesting problem to light: the raw map data of OSM doesn\u2019t have any hierarchy, for instance you don\u2019t have an obvious hierarchy\/link between San Francisco, California and the United States. Nominatim was used to rebuild that hierarchy in the exported format so another tool could easily process it.\nThe resulting XML file weighs 150GB and looks somewhat like:\n\nWe also built a SAX parser to process the file and rebuild the features hierarchy by following the \u201cparent_id\u201d and \u201cparent_place_id\u201d XML attributes.\nIt turned out to be a tiny bit more complex than expected\n\nA lot of features were associated with duplicates and inconsistency:\nSome cities have both \u201cboundary\/administrative\u201d and \u201cplaces\/city\u201d features, some have just one of them\nA street is composed of several segments, so you might see several features representing the same street\nThere are some street names which are common to several cities, so we needed to have multiple records within Algolia\n\n\u2026..some hierarchy couldn\u2019t be resolved\n\n\nSome features did not have any parents\nSome streets were attached to their countries","record_index":1},{"post_id":5418,"post_title":"Behind the Scenes: Algolia Places","post_date":1470360617,"post_date_formatted":"Aug 5th 2016","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/behind-the-scenes-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-05-at-10.31.14-360x200.png","time_to_read":10,"content":"\u2026 but not cities\nSome cities are also states, so the hierarchy was very confusing\n\n\nand some features also lacked some metadata.\n\n\nThey weren\u2019t associated with the corresponding population\nSome counties didn\u2019t have postcodes\nSome cities didn\u2019t have translations\n\n\nDeduplication is generally super easy, but when you need to deduplicate 150GB of data, it leads to all sorts of new problems - in our case, we never seemed to have enough RAM! We also wanted to make sure that parsing was fast enough in order to avoid waiting days to build the Algolia records ... after all, milliseconds matter!\nSo we tried to leverage the previously built GeoNames index as much as possible to fix the missing data we could hit with OSM; but since these 2 datasets didn\u2019t share the same IDs, it was obviously way more complex to aggregate.\nFor performance reasons (more on that under \u201cSearch strategy\u201d), we decided to build multiple indices from those records:\n\n1 index for the whole planet (~20M records)\n1 index per country (~6M records for the US, ~1.5M records for France, \u2026.)\n\nThat\u2019s about 60GB of indices.\nJust so you have an idea, the overall parsing + record generation + indexing, takes ~12 hours at a time, today.\n The record schema\nHere\u2019s what our final record schema looks like:\n\n \nStep 3: Index configuration\nWe constructed the underlying Algolia indices with the following configuration:\n\nSearch works with all localized names, but considers the default name as the most important\nThe names are considered as more important than the city, the postcode, the county, the administrative area, the suburb, the village and even the country.\n\n\n\nWe make sure to rank countries above cities and cities above streets, ensuring that we get the highest populated places first.\nIn case the population is not set, we fall back on the OSM\u2019s importance field; which reflects the importance of the underlying place.\n\n\nStep 4: The search strategy\nWe built a custom API endpoint to query the indices","record_index":2},{"post_id":5418,"post_title":"Behind the Scenes: Algolia Places","post_date":1470360617,"post_date_formatted":"Aug 5th 2016","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/behind-the-scenes-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-05-at-10.31.14-360x200.png","time_to_read":10,"content":"\u2026 and implemented a custom querying strategy:\nIf the query is performed from your backend (and therefore doesn't specify any  query parameter) or your source IP address isn't geo-localized, the results will come from all around the world because we target the \u201cplanet\u201d index.\nOr, if the query is performed from the end-user browser or device (and hence specifies an  query parameter) or has a source IP address that is geo-localized), the results will be composed of:\n\nNearby places\n\nPlaces around you (<10km): this is a query using our  feature,\nPlaces in your country: this is a query targeting the specific country index,\n\n\nPopular places all around the world using a query targeting the \u201cplanet\u201d index.\n\n\nSpecifying a country query parameter will override this behavior, restricting the results to a subset of the specified countries only.\nNumerical tokens in the query string are considered as optional words to make sure we always find the address even if the postcode & house number are wrong.\nWe also defined a list of stopwords and tagged every stopword of the query string as optional.\nWe queried the underlying indices multiple times to ensure that:\nPopular places will always be retrieved first.\nNearby places will always be better than places in your country, which are better than world-wide results.\nIf both a city and an address match the query, the city will be retrieved first.\nIf the query doesn't retrieve any results, we fallback on a degraded query strategy where all words are considered optional and we only target cities.\n\nThe Result\nWe\u2019ve had a great time building Algolia Places and the results today make all those coffee-fueled sleepless nights absolutely worth it. Here\u2019s a quick look at them:\n\nWe\u2019re hosting the Algolia Places infrastructure on 1 Algolia Cluster + 9 DSN replicas\n\n3 main servers in Germany\n4 DSN servers in France\n4 DSN servers in US\n1 DSN server in Singapore\n\n\n\n\nOur Algolia Places infrastructure is currently processing 60 queries","record_index":3},{"post_id":5418,"post_title":"Behind the Scenes: Algolia Places","post_date":1470360617,"post_date_formatted":"Aug 5th 2016","author_name":"Sylvain Utard","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/15705b7092e32cd80c32b0510ba59673?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/behind-the-scenes-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/08\/Screen-Shot-2016-08-05-at-10.31.14-360x200.png","time_to_read":10,"content":"\u2026 per second and provides answers in 20-30ms on an average.\n\n\nOur infrastructure is still at <3% of its capacity.\n\n\nWe\u2019ve reached 3200 stars on the GitHub repository of the frontend JavaScript library and a bunch of positive feedback.\n\nWant to use it?\nGo for it, it could be FREE!\nWe\u2019re providing a non-authenticated free plan limited to 1,000 queries a day and we increase the free plan usage to 100,000 a month if you signup.\nTry Algolia Places now!","record_index":4},{"post_id":5393,"post_title":"Algolia (and Burgers) at PolyConf","post_date":1469680747,"post_date_formatted":"Jul 28th 2016","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-and-burgers-at-polyconf\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/Pasted-image-at-2016_07_21-15_58-1-360x200.png","time_to_read":6,"content":"Have you ever had a\u00a0conversation with your friends which goes something like this?\nYou: Let\u2019s go grab some burgers for dinner tonight!\nFriend 1: Not burgers, anything else is fine.\nFriend 2: Thai?\nYou: Meh\u00a0\nUs too.\nFood is something we\u2019re very passionate about here at Algolia (along with coding). In fact, there\u2019s an animated discussion about where we should go for lunch every single day - pretty much like the conversation described above.\nSo, in typical fashion we decided to build an app, aptly named Burger, that can help tackle this problem. And then took it to PolyConf 2016, held in Poznan, Poland.\nWhat Did We do at PolyConf?\nPolyConf, now in its eleventh edition, is a tech conference for programmers interested in a polyglot approach to software development. This year\u2019s edition was held in Poznan and we had the chance to enjoy three intense days full of awesome talks about programming languages and technology. We also had the opportunity to host a workshop (more on that later)!\nThis edition was particularly focused on functional programming languages such as Clojure, Haskell, Oden, Crystal and Erlang \/ Elixir with a sweet spot for immutable data structures and a look at JavaScript\u2019s interoperability with ClojureScript and Elm - both heavy sources of inspiration for Redux (In case you\u00a0missed it we wrote a detailed blog post about Redux here).\nBack to the workshop. With Algolia being a polyglot programming company itself (we have over 15 API clients!), it wasn't very hard for us\u00a0to work with\u00a0a topic which could\u00a0interest\u00a0programmers with a large variety of skillsets. We decided to host a workshop which would teach participants to build a full-stack application using different technologies that we use daily.\nIntroducing Project Burger\nAfter a couple of discussions we decided to build Burger, an app which can be used to find lunch venues with people who have similar culinary tastes - a sort of meetup\u00a0app\u00a0for meals if you will; we wanted to mainly","record_index":0},{"post_id":5393,"post_title":"Algolia (and Burgers) at PolyConf","post_date":1469680747,"post_date_formatted":"Jul 28th 2016","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-and-burgers-at-polyconf\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/Pasted-image-at-2016_07_21-15_58-1-360x200.png","time_to_read":6,"content":"\u2026 demonstrate how to integrate different technologies and languages to build a conceptual -yet plausible- app from the ground up.\nHere\u2019s a small demo of the App:\n\nWe implemented the app by using React, Redux and a REST API Service by using Scala, a JVM functional \/ Object Oriented language; we also indexed all the venues in Poznan by using FourSquare\u2019s API in order to make sure it had some Algolia magic.\nThat\u00a0way we had all the ingredients for our workshop: a little bit of Back-end, with some Front-end, topped off with an overview of how to integrate these technologies with a 3rd party API such as Algolia. Et Voila! Burger was born.\nIf you\u2019re curious you can find the complete app source code on GitHub and some slides we used for the workshop introduction.\n\nWhat We Learnt Along the Way\nIt was our very first time organizing a 3-hour workshop and we learnt quite a bit along the way. Here\u2019s a brief recap:\n\n Keep the scope small\n\nWe didn't quite realise while building Burger that it would probably be a bit hard to build an entire app from scratch in 3 hours. We ended up almost running out of time and just about\u00a0managed to cover all the topics we needed\u00a0to in order to have a functional application at the end. This is also because we tried to cover too many topics without accounting for enough time for the participants to apply them.\nIf given the chance to redo it, we\u2019d probably narrow the scope: go for choosing either of the two - front-end or back-end.\n\n Focus on concepts and material\n\nWe took for granted that most of the workshop attendants were familiar with React and JavaScript Development in general; in reality we had some people interested in Scala and some others in the Redux portion, and so, consequently, some of the participants\u00a0didn't get\u00a0a chance to explore their favorite part in depth. \nIf we could go back we\u2019d probably spend more time preparing material and exercises to keep participants\u00a0engaged\u00a0and involved until the end without focusing too","record_index":1},{"post_id":5393,"post_title":"Algolia (and Burgers) at PolyConf","post_date":1469680747,"post_date_formatted":"Jul 28th 2016","author_name":"Gianluca Bargelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/74b7711f3f1936c9bed8cc949e7e1c26?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolia-and-burgers-at-polyconf\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/Pasted-image-at-2016_07_21-15_58-1-360x200.png","time_to_read":6,"content":"\u2026 much on the implementation details.\nIntegration examples are great\nIntegrating a third party API into an application is something that every developer knows by heart; it\u2019s a great example of how leveraging existing libraries can help achieve speed and productivity in such a short amount of time.\nWe wanted to demonstrate\u00a0something similar in our workshop by wrapping our Algolia JavaScript client in a React component and use it for our app, saving the time and effort (out of the scope of the workshop) needed to implement an\u00a0autocomplete component from scratch.\nWhat Else?\nPolyConf was a great experience for us and we were really lucky to find ourselves amidst coders with very varied skill-sets - in fact, after our workshop, we attended several sessions on functional programming and we found ourselves newly enriched with knowledge of Clojure and Elm - we actually hacked a possible first draft of an Elm Algolia Client!\nApart from the conference, we also got a bit of\u00a0time to do some sightseeing and of course ate our fill of Burgers:\n\nHow could we not?\u00a0\nOur Favorite Part\nOrganizing a successful workshop is no easy job but we believe we've learned some important lessons for the next round; we also hope that some of the attendees will try to apply the\u00a0ideas we exchanged in their work or in their personal projects (and let us know, please!).\nAs developers, we absolutely loved PolyConf: interesting talks, nice food and a lot of ideas coming from different coder communities. It was a pleasant experience and we\u2019d definitely like to thank the organizing crew, particularly Zaiste and Martyna. Thank you for making it happen! See you next year!","record_index":2},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"No matter how\u00a0well crafted a search engine may be, if the query it is given is incomplete or inaccurate, the search results will always seem a little off.\nE-commerce often falls prey to this because large catalogs and innumerable filters complicate curation. Since relevancy is at the very top of our priority list, let us introduce a little experiment of ours, designed especially to enhance the search experience when it comes to image-driven content.\nThe Problem?\nThe goal we set for our computer vision experiment was to detect the color of the dominant object of a picture. Searching by color is a common occurence\u00a0when it comes to e-commerce websites, and while the nature of the object is usually clear in the description, its color is often more of a problem. As an example, let us consider an article containing several colors in its description (for the sake of the example: white and blue), but having a picture clearly showcasing a blue shirt. While searching for a white shirt this article may show up and may even rank very high, because the relevant color - white,\u00a0is present in the description. However correct this may be, having a number of images showing a blue object while searching for a white one does not give the best impression of relevance to the end user. The idea is certainly not to remove these results (because they are still correct), but to boost the ranking of the ones which more closely represent objects of the searched color.\nSome colorful\u00a0context...\nWe are obviously not the first to attempt to address this problem, but we\u2019ve taken a slightly different approach from the current solutions in the market. We can divide the applications trying to solve this problem into two different groups that provide different experiences:\n\nCommercial applications like vue.ai\nOpen source scripts like josip\/node-colour-extractor or lokesh\/color-thief\n\nThe open-source scripts listed are going for a different and simpler problem. They extract the palette composed of","record_index":0},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 the principal colors of an image. While they do the job well, they can\u2019t be applied out of the box to fashion images because we are not interested in the colors in the entire image. Using them directly would result in the color of the background being detected as the main color of the given image. Moreover they do not provide a word for the extracted colors, and indexing RGB values would not improve the search experience.\nVue.ai showcases some impressive results and provides many more features than our experiment. We wanted to provide a quick, easy way for users to enrich their records a bit and improve their ranking, especially if they have to deal with poor descriptions in their records. This script doesn\u2019t need you to register or anything: all you have to do is download, install dependencies, enjoy.\nOur approach \nChosen method\nThe problem we try to address is a relatively complex one and can be solved by a number of different ways. Most state-of-the-art computer vision frameworks these days adopt the Deep Learning path by using techniques such as Convolutional Neural Network to classify images. This is a path we decided not to take. Neural networks offer astounding results, but to do so they also need large datasets in addition to\u00a0dedicated hardware which shortens the computation time. We started this project as an experiment and we began with simpler methods to see where those would take us. We always knew that we would eventually open-source this project allowing anyone to run it and add information to their records: deep learning frameworks can be somewhat hard to setup and run; they also require a great deal of computing time. The sequence of algorithms we chose was able to process images reasonably fast on a machine with a classic CPU and can be installed as simply as any other python package.\nPre-Processing\nBecause we only cared about the color of the dominant object in the picture, a high level of detail was likely to confuse the algorithms. Moreover,","record_index":1},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 smaller images meant less computing. After performing tests on several images we observed that a size of 100x100 was a good compromise.\nWe really had the use-case of a fashion e-commerce website in mind while designing this tool, and it led to the idea of an additional step to improve our color detection tool: cropping.\nThe object we are interested in is highly likely to be centered within the image, cropping allows us to reduce the size of the background image. Although cropping does yield better results overall, keeping less than 90% of the original image doesn't work\u00a0well\u00a0with background removal when considering\u00a0a heterogeneous data set. This is due to the main object, sometimes touching the corners of the picture as a result of which it gets considered as \u00a0the \u201cbackground\u201d for subsequent algorithms.\nResizing and cropping (original on the left)\nExcluding background\nBe it plain white or complex, the background isn\u2019t something we want to interfere with, while detecting the color of the main object of a picture. Separating the \u201cforeground\u201d and the \u201cbackground\u201d is not an easy task unless you know that it follows a simple convention (i.e. plain white), or exactly where the targeted object is placed in the picture. We make none of these assumptions to allow for broader use cases, and combine several simple computer vision algorithms to handle reasonably complex backgrounds.\nThe first step can be viewed as a thresholding over the image. We take the pixel values at the four corners, identify all pixels that are close in terms of color-shade and consider them to be a part of the background. We use the \u00a0metric to determine the distance between two colors. This step is particularly useful on more complex backgrounds because it doesn\u2019t use information such as edges, but can\u2019t discard gradient or shadows efficiently.\nGlobal thresholding struggling with shadows\nThe second step uses flood filling and edge detection to remove shadows and gradients. The","record_index":2},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 technique is largely inspired by this article on the Lyst engineering blog. The main goal is to remove the background- generally indicated by the absence of sharp edges. To achieve this, several steps of smoothing are used to maximize the efficiency of the Scharr filter\u00a0used. Contrary to the first step, this step doesn\u2019t behave well on complex backgrounds that contain edges themselves, or in cases where the object and the background don\u2019t have sharp edges separating them.\nShadows are handled by edge detection and flooding\nBy using both of these filters we try to get the best of both worlds. However, this\u00a0is not always achievable, sometimes one of the two steps does not\u00a0accurately separate the background and the foreground and ends up removing way too many pixels. When this happens, the result of the step is simply ignored.\nExcluding skin\nA lot of pictures include models and if the targeted object is a swimsuit, the number of pixels representing skin can be greater than the number of pixels we are actually interested in. For cases like this, we implemented a really simple method to discard additional pixels.\nDetecting skin is an open problem and the possible solutions range from testing if each pixel color can be considered as skin to training a complex artificial intelligence model. We went for the lower end of the spectrum of complexity: a simple filter using only the color of the pixels to classify them.\nThere are a lot of skin filters out there, and each has their pros and cons. We chose one based on its false-positive rate. Indeed most of the filters only consider the color of the pixel - in fact it is very common (unfortunately) to see orange-tinted clothes entirely labeled as skin. Choosing a low false-positive rate reduces this risk but doesn\u2019t remove it, and because all data sets don\u2019t contain skin, we made this step completely optional in the final script.\nFor now, a single filter has been implemented which tries to be as versatile as possible,","record_index":3},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 however we definitely plan to implement several narrower ones to be able to target certain skin types more efficiently.\nDetecting skin pixels\nClustering pixels\nWe have no interest in the multiple shades of colors that images may contain, but only for the canonical, definitive names. To achieve this we use a clustering algorithm to group similar pixels together.\nClustering a gray T-shirt (from left to right: original, background, skin, clusters)\nAll pictures are not equal in terms of number of colors, and even if we want to group together different shades of the same color, we don\u2019t want to say mix blue and green pixels. Because of this property, we can\u2019t build the same number of clusters regardless of the image. We try to detect the best number of clusters, using the \u201cjump\u201d method for which a simple description can be found here on Wikipedia. This technique is relatively easy to understand. The error in a particular case of clustering can be measured using the following formula where \u00a0is the center of the cluster that  belongs to:\n      \nThis formula says that the greater the distance from a particular point to its attributed cluster\u2019s center, the higher the error. This value will always decrease when clusters are added, until it reaches the point where \u00a0which gives an error of 0. However, this error won\u2019t decrease at the same rate every time a cluster is added. At first, each added cluster will decrease the error by a huge amount, and at one point, the gain will become much smaller. The goal is to find the moment when adding a cluster is way less significant.\nThe algorithm uses a slightly more complex computation of error, but the idea behind it is the same. This way we adapt the number of clusters to have colors close to one another grouped together but still enough clusters not to group distinctly different colors on multi-colored items.\nRainbow clusters (from left to right: original, background, clusters)\nAttributing color names\nLast but","record_index":4},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 not the least we have to return readable names in a given language, say English, in the results\u00a0and not just RGB values. This problem is harder to solve than it looks: the color space is huge (~16 million\u00a0possible values), and different colors have very heterogeneous sizes. Defining each possible color range by hand is very tedious and yields poor results in a lot of cases. We turned to a K-Nearest-Neighbors algorithm to give color names to RGB values, thanks to the XKCD Color Survey. The XKCD survey consists of 200,000 RGB values labeled with 27 different color names (e.g. black, green, teal, etc.) that we use to train a scikit-learn KNeighborsClassifier.\nThis method is still far from perfect; for example, it is not able to handle shades of grey. We are working with the RGB system, and the grey colors form a plan in this colorspace. Because of this topology, there are never enough neighbors (using a euclidean distance) to categorize a color as grey. To solve this we added an additional step dedicated to these colors. By computing the \u00a0distance between RGB values and their projection on the grey plan, it is possible to tell how \u201cclose\u201d the color actually is to grey. We then apply an arbitrary threshold to classify the color as grey, or continue with the classifier.\nThe overall process still has drawbacks. For example it doesn\u2019t provide enough insights to be able to put a color in multiple categories (sometimes it is hard to tell if a color is green or blue for instance, and we may like to label it both green and blue).\nThe final result\nToday this computer vision experiment is available here on github as a stand-alone python script that anyone can run to enrich their records containing an image URL with the extracted color tags. Both a library for more advanced usage and a CLI are available for use as well.\nThis is as simple as it gets using the CLI:\n\nWe ditched the idea of a dedicated search engine optimized for images for\u00a0to a simple reason: Algolia","record_index":5},{"post_id":5299,"post_title":"How We Tackled Color Identification","post_date":1469413826,"post_date_formatted":"Jul 25th 2016","author_name":"L\u00e9o Ercolanelli","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/7953f90c086b96d7b0bb85f14592ce11?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-handled-color-identification\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-5-360x200.png","time_to_read":12,"content":"\u2026 already has a state of the art search engine, and not taking advantage of it would be a pity!\nA self-serve script fulfills another criterion close to our hearts: configurability. Our script uses sane defaults, but all steps described above can be tweaked to yield better results on a specific catalog. These tweaks range from making certain steps more or less aggressive to deactivating them entirely. We also provide a way to support any language: the dataset used by the script to give color names to images can be changed by the user, thus making room for non-english color names.\nOur script doesn\u2019t use any kind of machine learning yet and we\u2019re aware that results could be greatly improved by going down this road. For this first release our tool features ~84% accuracy. While this is still low, our tool isn\u2019t designed to be used on its own, but in conjunction with your textual searches by boosting images whose colors match what is searched for.","record_index":6},{"post_id":5333,"post_title":"How Algolia Is Powering Up ProdPad's Help Center","post_date":1469165427,"post_date_formatted":"Jul 22nd 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-is-powering-up-prodpads-help-center\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-4-360x200.png","time_to_read":6,"content":"Most Saas\u00a0companies today are still figuring\u00a0out how to get customers to move towards self-service, especially when it comes to support. But one company has figured out how to make it easy to find all the answers.\nCustomers at ProdPad are consistently amazed at how lightning fast their customer support is, especially when they learn that the company is still a startup.\nTheir secret sauce? A blazing fast help center search, powered by Algolia.\nAndrea Saez, Head of Customer Success at ProdPad, tells us how ProdPad is using Algolia to provide consistently fast support to a growing customer base.\nWhat does ProdPad do?\nProdPad provides product management software that helps product teams collect ideas, identify priorities, and build flexible product roadmaps.\nHow are we handling a fast growing customer base without an army of customer support agents?\nWith the help of our incredibly fast knowledge-base powered by Algolia.\nSince we started using Algolia on our Zendesk Help Center, we\u2019ve seen self-service become the preferred method of support for our customers.\nWhy?\nBecause they\u2019re now finding answers on their own instantly, without having to go through the entire process of contacting our support team. \u00a0Even though our response times are surprisingly quick and our live support brings a 100% satisfaction rating, our customers prefer to help themselves.\nSelf-service is the best way for us to give our customers what they want - and Algolia is helping us make this a reality.\nHere\u2019s how Algolia is helping us support over 500+ customers without breaking a sweat.\nOur Secret Sauce\nYou don\u2019t know how much time you\u2019ve been wasting until you use a search bar that just lights up with exactly the right article, even before you\u2019ve finished typing.\nOur Help Center has always performed well - we worked hard to create a well-organized support center with useful guides and self-help resources. Usually, as a user, you might need to click around for a bit, try a few different","record_index":0},{"post_id":5333,"post_title":"How Algolia Is Powering Up ProdPad's Help Center","post_date":1469165427,"post_date_formatted":"Jul 22nd 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-is-powering-up-prodpads-help-center\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-4-360x200.png","time_to_read":6,"content":"\u2026 queries and multiple searches but you\u2019d always find the right article.\nBut with our Algolia search, that same content is now instantly discoverable.\nThe impact \u00a0was immediate - we saw a dramatic drop in live support requests around the basic and recurring questions that used to make up a significant part of our support queue.\nWe\u2019ve been able to create an experience our users absolutely love. They\u2019re now finding support articles in just milliseconds.\nOur new users notice it the most because they\u2019re usually digging around and setting up different areas of ProdPad during their magically extending free trial - and our search helps them find docs and instructions in less than a hot second!\n\nFast, helpful support is something we\u2019ve become known for, and now Algolia is helping raise the bar again.\nWe have more control over our support experience\nConsumers generally try to help themselves before reaching out to live support. Today we\u2019re able to monitor what they\u2019re searching for so we can improve our overall product experience. \nWith our support traffic now divided between our knowledge-base and incoming support tickets, we\u2019re constantly studying our data to understand how we can clarify and simplify information for our customers.\nOur goal is to reduce as much friction as we can during our customer\u2019s support journey.\nWe\u2019re using Algolia and Zendesk analytics together to dig for answers to questions like:\n\nHow many times do they need to contact support to fix a problem? \nWhich searches result in tickets being opened?\nHow long is it taking our customers to get the answers they\u2019re searching for?\n\nFor example, we found that some users were searching for \u201cgoogle docs\u201d and weren\u2019t finding any results that contained that term. \nThey were looking for more information about our Google Apps integration, but couldn\u2019t find it because we hadn\u2019t included that specific keyword. It was an easy fix. I simply went in and added \u201cgoogle docs\u201d to that","record_index":1},{"post_id":5333,"post_title":"How Algolia Is Powering Up ProdPad's Help Center","post_date":1469165427,"post_date_formatted":"Jul 22nd 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-is-powering-up-prodpads-help-center\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-4-360x200.png","time_to_read":6,"content":"\u2026 article to ensure that from then on, the right article would pop up straightaway.\nA quick change like this saves time and provides more bandwidth to answer individual support tickets. It also saves our customers from the hassle of opening up a support ticket for minor questions.\nAs we continue to make these improvements from our end, we\u2019re making the support experience better for both our customers and ourselves.\nWe\u2019re learning exactly what they\u2019re trying to achieve\nWe can track queries in our Algolia analytics, so we know the kind of questions our customers are keying in and how they\u2019re thinking.\nTake a look at our top queries:\n\nThis tells us what people are looking for, how many results come out of that search, and how many times users click on a given result.\nAs product managers, this kind of data is a treasure chest for us. What could we be explaining better? What are customers trying to do within ProdPad? Where are they getting stuck?\nWe can use it to understand what areas of our product are pain points for our clients. Conversely, we can use it to see which areas they\u2019re most interested in. We can and do spot search trends which help us identify what our customers are trying to achieve.\nThis has not only helped us make improvements to the app itself, but the way we communicate our features and functionality across touchpoints, from our in-app messaging to our blog.\nThe Takeaway\nAs the overall number of support tickets in our queue drops, we\u2019re seeing a greater percentage of customers reach out for help with setting up custom and increasingly sophisticated processes in ProdPad. \nThis is an excellent sign for us. It means they\u2019re becoming increasingly proficient in ProdPad and are finding it worth their time and effort to invest in our more advanced functionality.\nThat\u2019s how we\u2019ve become the product management tool that everyone loves.\nSmall team, big impact - Algolia has made a big difference.\n ","record_index":2},{"post_id":5278,"post_title":"DocSearch: 150 sites, 75k searches per day","post_date":1468542531,"post_date_formatted":"Jul 15th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/docsearch-150-sites-75k-searches-per-day\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-3-360x200.png","time_to_read":4,"content":"In December 2015 we released DocSearch, an easy way to make your software or API documentation searchable.\nFast forward 6 months later and DocSearch is powering the documentation behind 150 projects including React, Symfony, Play framework and Meteor.\nWith just a little configuration, DocSearch can automatically crawl most documentation websites and then provide a nice autocomplete search box to add to documentation pages.\nToday we have some exciting news for the DocSearch project:\n\nAn improved design is here with the release of DocSearch.js v2\nWe are open sourcing the full DocSearch crawling code!\n\nA better default design\nWith several iterations, we upgraded the DocSearch default design incorporating all the feedback we received from the community. The new design is available today with the release of DocSearch v2. \nHere's what\u00a0it looks like on the\u00a0Middeman website:\n\nFrom 0 to 75k searches a day\nDocSearch is now powering 150 documentation searches. Most of them are in English, but we also have sites in Turkish, Chinese, French, Japanese, etc.\nIn total we\u2019ve indexed around 1.5M records and are performing 75k searches per day and growing. An interesting trend we\u2019ve noticed is that most of these searches are made during weekdays. Who ever said developers worked on the weekend?\n\n\nThe last bar, starts on the 26th of June, that\u2019s why it\u2019s smaller.\nThe DocSearch open source stack\nDocSearch is composed of three different projects. as of today they will all be open source :\n\nThe front end of DocSearch (which was already open source): https:\/\/github.com\/algolia\/docsearch\nThe scraper which browses & indexes web pages: https:\/\/github.com\/algolia\/docsearch-scraper\nThe configurations for the scraper: https:\/\/github.com\/algolia\/docsearch-configs\n\nWant to try DocSearch on your documentation? Just follow these steps:\u00a0https:\/\/github.com\/algolia\/docsearch-scraper\nThe scraper is a collection of submodules, each in its own directory:\n\ncli: A command line tool to manage","record_index":0},{"post_id":5278,"post_title":"DocSearch: 150 sites, 75k searches per day","post_date":1468542531,"post_date_formatted":"Jul 15th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/docsearch-150-sites-75k-searches-per-day\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-3-360x200.png","time_to_read":4,"content":"\u2026 DocSearch. Run `.\/docsearch` and follow the steps\ndeployer: Tool used by Algolia to deploy the configuration in our Mesos infrastructure\ndoctor: A monitoring & repairing tool to check if the indices built by the scraper are in good shape\nplayground: An HTML page to easily test DocSearch indices\nscraper: The core of the scraper. It reads the configuration file, fetches the web pages and indexes them with Algolia.\n\nFuture of DocSearch\nWe want DocSearch to be easy to integrate and customize for everyone. To do that we are building a visual configuration tool that will help you auto-generate the DocSearch configuration and the styling of the autocomplete menu.\nRequest an awesome documentation search\nWe\u2019re always looking for websites that are missing a great documentation search and are happy to help create it. We\u2019ve seen DocSearch improve the search experience of many types of documentation and we think it could improve yours too.\nRequest your DocSearch now!","record_index":1},{"post_id":5252,"post_title":"How we reduced boilerplate and handled asynchronous actions with Redux","post_date":1468296873,"post_date_formatted":"Jul 12th 2016","author_name":"Alexandre Meunier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e29b5b29d63034859dee0965fa82b5e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-reduced-boilerplate-and-handled-asynchronous-actions-with-redux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-2-360x200.png","time_to_read":7,"content":"At Algolia, we do a lot of front-end JavaScript, because for us UX and UI are key components of a great search \u2013 we want the search experience to be perfect across the entire stack.\nRecently, we\u2019ve been playing quite a lot with React\u00a0in our projects, especially in Instantsearch.js, an open source library for building complex search UI widgets. So far\u00a0our experience has been very positive; it was so good that we eventually decided to introduce React onto our dashboard, along with Redux to handle shared state across components.\nWe're really happy with the benefits that both have brought to our stack; in particular, we found that Redux brought quite a few cool things to the table:\n\nTestability: Because view components essentially hold no logic, it is very easy to test them \u2013 just provide them with data and spy functions, and check that it renders the way you want;\nSimplicity: We noticed that using Redux\u00a0led us\u00a0to write very clean and elegant code: no obscure\u00a0features or leaky abstractions means that\u00a0we always know what's going on and virtually never run into cryptic, framework specific errors like some of AngularJS's errors;\nDeveloper experience:\u00a0There are great developers tools (i.e. check out Redux Devtools, it's awesome!) out there which allows you to easily inspect Actions and State mutations; on top of that you can also rely on very cool features such as hot reload and, even better, time travel: fixing bugs it's just a matter of rewinding \/ forwarding a set of actions, find the state change which caused the error, fix it and finally replay the actions.\n\nIf you are not familiar with React or Redux as yet, there are many\u00a0great resources available online for both.\nReact and Redux give you great powers, but also great responsibilities ! You are free to handle many things exactly the way you want. However, to really harness their combined potential, we've found that setting up and enforcing conventions is crucial.\nOrganising your single page","record_index":0},{"post_id":5252,"post_title":"How we reduced boilerplate and handled asynchronous actions with Redux","post_date":1468296873,"post_date_formatted":"Jul 12th 2016","author_name":"Alexandre Meunier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e29b5b29d63034859dee0965fa82b5e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-reduced-boilerplate-and-handled-asynchronous-actions-with-redux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-2-360x200.png","time_to_read":7,"content":"\u2026 application\nWe're using the following fractal directory structure. Each page is an almost self contained app, asynchronously loading its own subpages.\n\nThere\u2019s nothing particularly controversial here, with the exception of one convention we\u2019ve chosen to impose: To collocate Redux action creators and reducers in the same files, following the Ducks proposal. Our \u201cactions\u201d files look something like this:\n\nThis allows us to use default imports for the purpose of creating the combined reducer when creating the store, while still being able to used named imports in containers to import just the actions that are needed.\nReducing boilerplate in ...reducers\nWe found that whenever we were writing reducers and action creators, we were often writing duplicate action creators and reducers, doing little more than updating a subset of the reducer\u2019s state, for instance:\n\nStrictly speaking, action creators are not required since components can directly dispatch actions by themselves. However, we found that working with the more abstract action creators, allowed us to write more modular components, which would literally need no knowledge of how the reducers work or which action types are used. Instead, we simply need to pass them data and (wrapped) action creators.\nTherefore, we looked into how we could simplify the reducer side of things, instead of the action creators. After a few iterations, we ended up with redux-updeep, a strongly opinionated createReducer implementation which assumes that the majority of the actions will result in deep updates of the reducer\u2019s state. It allowed us to write the following code:\n\nHow does it work ? As mentioned before, it handles\u00a0all unspecified action types in the same way, using\u00a0updeep to immutably merge the action\u2019s payload into the reducer\u2019s current state. While it is still possible to explicitly specify action types and reducers, we still haven't felt the need to do so!\nGranted, when using redux-updeep, it becomes more","record_index":1},{"post_id":5252,"post_title":"How we reduced boilerplate and handled asynchronous actions with Redux","post_date":1468296873,"post_date_formatted":"Jul 12th 2016","author_name":"Alexandre Meunier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e29b5b29d63034859dee0965fa82b5e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-reduced-boilerplate-and-handled-asynchronous-actions-with-redux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-2-360x200.png","time_to_read":7,"content":"\u2026 complicated to compose reducers, which is idiomatic Redux. However, we've made it such that it is very easy to write factories that allow us to parameterize the action type's namespaces as well as the path at which the update is merged:\n\nThen, it becomes possible to use the initial action creator deeper into a reducer state:\n\nWe're pretty happy with our current set up, so we thought that we would share it with the world. In fact, we've just open sourced it ! Find it on GitHub.\nHandling asynchronous actions \u2013 and reducing more boilerplate\nUsing thunks ?\nBy default, Redux does not care how you handle asynchronous actions. The redux-thunk middleware is available but enforces no rules: it simply enables you to dispatch multiple actions from a single action creator. Using it, you can for instance easily dispatch loading\/success\/error actions in response to a Promise:\n\nThis is a great start, but the code still relies on a lot of boilerplate. If you have as many simultaneous asynchronous actions as we do, and want to handle them all in the same way (e.g. how to keep track of the pending state, how to handle errors), it rapidly becomes a tedious task.\nLess boilerplate, more magic!\nThere is a lot of middlewares in the Redux ecosystem designed to handle asynchronous actions and\/or promises, but we couldn\u2019t find one that would handle a few conventions that we had initially defined for handling asynchronous actions and asynchronous data in the components:\n\nReducer states can have several keys that will ultimately be assigned a value (let\u2019s call those \u201ceventual values\u201d);\nWe want to be able to enquire whether a particular value is ready or not, without resorting to isXXX boolean flags;\nWe wanted to be able to dispatch eventual values like normal values in action creators.\n\nSo we decided to create one, the redux-magic-async-middleware, and we've just open sourced it! It is optimised to work with redux-updeep, and enables dispatching promises in payloads like","record_index":2},{"post_id":5252,"post_title":"How we reduced boilerplate and handled asynchronous actions with Redux","post_date":1468296873,"post_date_formatted":"Jul 12th 2016","author_name":"Alexandre Meunier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/5e29b5b29d63034859dee0965fa82b5e?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-reduced-boilerplate-and-handled-asynchronous-actions-with-redux\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-2-360x200.png","time_to_read":7,"content":"\u2026 synchronous\u00a0values:\n\nThe middleware will automatically extract the promises from the payload, and replaces them with eventual values. When a promise is resolved, it will trigger a success action which will update the resolved data into the reducer states, thus resolving the eventual value.\nIn combination with this, we have created the eventual-values library, very much inspired by another eventual values library. This library is extremely simple, but allows us to encapsulate and abstract the behaviour that we desired around asynchronous values. It allows to you write code like this:\n\nWhat's next ?\nWe're still experimenting with those concepts and tools, and are gradually introducing flow typing in our codebase.\u00a0Watch this space for more updates on how we use React, Redux and JavaScript in general!","record_index":3},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"The way we search has changed a lot in the past decade. The original function of the Enter key was to begin a search, today it\u2019s used to select a result that\u2019s already been displayed. The type of information that people search for has changed too.\nToday\u2019s search engines are used for much more than just web sites and documents\u2014they\u2019re also used to find specific items like people, places and products.\nThe Algolia engine was designed with both of these trends in mind. Algolia\u2019s sweet spot is providing instant, relevant access to millions of small, structured records (and now from anywhere in the world). In this blog post, we\u2019ll talk about how the Algolia engine computes textual relevance and why it works well for exactly these kinds of records. By textual relevance, we mean the letter and word-based aspect of ranking query results. Textual relevance does not include business relevance, which is used to provide a custom ranking based on non-textual information like price, number of followers or number of sales.\nIn a previous post we discussed how our ranking formula works, specifically how it uses a tie-breaking algorithm to rank each component of textual relevance one-by-one. In this post, we\u2019ll look at each of the five individual components it uses in detail: number of typos, number of words, proximity of words, importance of attributes and exactness.\n1. The five components of textual relevance\nEach of these components address a unique relevance challenge while also playing their part in the global relevance calculation.\n\n1.1 Number of typos\nThe number of typos criterion is the number of typos in the query when compared with text in the record. This criterion allows you to rank results containing typos below results with the correct spelling. For example, in a search for \"Geox CEO\", you would prefer to have the \"Geox\" result (a typo-free match) displayed before \"Gox\" (a 1-letter-off typo). This example has a twist\u2014\u201dGox\u201d is not actually a typo in","record_index":0},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 the usual sense (it\u2019s a correctly spelled proper noun) but it\u2019s still possible that a user could have mistyped \u201cGeox\u201d when searching for it.\n\n\n\n1.2 Number of words\nThe number of words criterion is the number of query words found in the record. This criterion applies when the query is performed with all words optional (a.k.a. an \"OR query\"), a technique commonly used to avoid displaying no results. Here\u2019s a simple example. If you search for \"supreme court apple\" you would prefer to have the first record before the second one as it contains all 3 of the words in the query.\n\n\n\n1.3 Proximity of words\nThe proximity of words criterion is the distance between the matched words in the result. It\u2019s used to rank records where the query words are close to each other before records where the words are farther apart. In the example below, you can see two different results for the query \"Michael Jackson\". In the first record, the proximity criterion is 1 - the terms \u2018Michael\u2019 and \u2018Jackson\u2019 are right next to each other. In the second record, the proximity criterion is 7 because there are 6 words between the matching terms. A lower value is better, i.e. records with a lower proximity value will be ranked about records with a higher value.\n\n\n\n1.4 Importance of attributes\nThe importance of attributes criterion identifies the most important matching attribute(s) of the record. Since records can contain more than one attribute, it\u2019s not uncommon for one query to match multiple attributes of the same record or different attributes of different records. However, a match of one attribute might be more important than another when it comes to ranking, and that\u2019s precisely why we have the attribute criterion. An example is shown below. Given a query for \u201cNetflix\u201d in a collection of articles, you\u2019d want the title attribute match to carry more weight than the description attribute, hence the title-matching record is displayed first.\n\n\n\n1.5 Exactness\nThe exactness","record_index":1},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 criterion is the number of words that are matched exactly. This criterion is very important for building quality instant (search-as-you-type) experiences because it\u2019s important to display full-word matches before prefix matches. See the example below. If you search for \"Prince\", you want the exact word match to appear before the match where \u201cPrince\u201d is being considered as a prefix of \u201cPrincess\u201d.\n\n\n2. How the criteria are computed\nWe\u2019ve just seen what each criterion does and how it can be used to influence the ranking of search results. Next, let\u2019s look at how the five criteria are computed and what we\u2019ve done to make their computation efficient.\n\n2.1 Computing the number of typos\nIn our previous post about Query Processing, we explained how alternative terms are added to queries. For each alternative term, we also have an associated number of typo criteria (which corresponds to the edit-distance). This value of the overall typo ranking criterion is the sum of all typos used to find the match. For example, if the record is found via 2 words having 1 typo each, the typo criterion value will be 2.\nIt\u2019s a very good practice to have this criterion at the top of the ranking formula. However, there is one exception: typosquatting. If you operate a service with a lot of user-generated content, you might already be familiar with this problem. Let\u2019s take Twitter as an example. If you search for \"BarakObama\" on Twitter (note the typo), you would want the official @BarackObama account to appear before the typosquatting account @BarakObama. However, today it does not (as of June 2016). Typosquatting is a complex and difficult challenge.\nOne way that we recommend to handle typosquatting is to add an isPopular boolean attribute to the record of popular accounts and then to put it in the ranking formula before the typo criterion. On Twitter, isPopular could be set to true if the account is verified. This approach ensures that the official account with a typo","record_index":2},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 will be displayed before any typosquatting accounts, no need to hide any results. For this approach to work well, the number of records with isPopular set to true should be a relatively small percentage of the dataset (less than 1%).\n2.1.1 Typo tolerance tuning\nThe Algolia engine provides several different ways to tune typo tolerance for specific words and attributes:\n\nminWordSizefor1Typo and minWordSizefor2Typos: Specify the minimum number of characters in a query word before it can be considered as a 1 or 2 typo match.\nallowTyposOnNumericTokens: Specify if typo tolerance is active on numeric tokens (numbers) present in the query.\ndisableTypoToleranceOnWords: Specify a list of words for which typo tolerance will be disabled.\ndisableTypoToleranceOnAttributes: Specify a list of attributes for which typo tolerance will be disabled.\nAlternative corrections: Specify a set of alternative corrections with the associated number of typos.\nadvancedSyntax: Add quotes around query words to disable typo tolerance on them.\n\n\n2.2 Computing the number of words\nComputing this criterion is relatively easy, it\u2019s just the number of words from the query that are present in the record. Each word might be matched exactly or matched by any alternative form (a typo, a prefix, a synonym, etc.).\nThe position of the words criterion in the ranking is important and will impact results. For example, the query \"iphone case catalist\" with all terms as optional will have two different ways to match the record \"Catalyst Waterproof Case for iPhone 6\/6S Orange\":\n\nIf words is before typo in the ranking formula: The ranking will be set to words=3 and typo=1 as matching more words is important to the ranking (even if Catalyst is found via a typo).\nIf typo is before words in the ranking formula: The ranking will be set to words=2 and typo=0 as less typos are more important to the ranking (in this case it\u2019s better to not match the Catalyst term and therefore not introduce a typo).\n\n\n2.2.1 The","record_index":3},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 removeWordsIfNoResults parameter\nBecause of situations like this, we usually recommend against performing queries with all terms optional. Instead, we recommend using a specific query parameter that we\u2019ve added to the API to help with these types of situations. The removeWordsIfNoResults parameter instructs the engine to first test the query with all terms as mandatory, then progressively retry it with more and more words as optional until there are results. This parameter gives us the best of both worlds (predictable behavior and always showing results) and also does not pollute faceting results. When all terms are optional there is a large increase in the number of matched records, as only one matching word is required to add the record to a facet. For example, in a query for \u201ciphone black case\u201d, any black item will be matched which will have a big impact on the quality of the faceting.\n\n2.3 Computing the proximity of words\nThe proximity criterion\u2019s value is the sum of the proximity of all pairs of words that are next to each other in the query. Computing the proximity of \"iphone release date\" is done like this: proximity(\"iphone\", \"release\") + proximity(\"release\", \"date\").\nThe engine stores word positions at indexing time in order to efficiently compute proximity at query time. We define the distance between two words A and B as the shortest distance between any A and B in the record (A and B may appear more than once). We increase the distance by 1 if the match with the shortest distance is not in the queried order.\nThe computed proximity between two words is capped at 8. If the words \"apple\" and \"iphone\" are separated by a distance of 10 or 100 words, the distance will still be 8. The goal is to avoid creating different proximity values for words that are not used in remotely the same context. In practice, 8 words is a sizeable distance between any 2 words, even when it\u2019s mostly common words between them. Capping the maximum proximity has two other","record_index":4},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 advantages:\n\nIt helps our tie-breaking algorithm. A maximum distance of 8 means that we don't know if a distance between words of 10 is better than 100, and usually we don\u2019t want to (it\u2019s just noise). Capping the proximity at a small number prevents introducing false positives into the relevance calculation. More importantly, it lets the other criteria in the ranking formula take over when proximity values are not meaningfully differentiated.\nIt enables a lot of optimizations on the storage and computation side, as we will see.\n\n\n2.3.1 How word positions are stored\nAs we index a document, we assign an integer position to each occurrence of each word. For example the string \"Fast and Furious\" would be indexed as:\n\n\"Fast\" at position 0\n\"and\" at position 1\n\"Furious\" at position 2\n\nAdditionally, we assign positions to some word prefixes as described in our blog post about indexing prefixes.\nThe position is also used to efficiently store associated information:\n\nWe use a single value to store the position of each word in the text and which attribute it came from. We keep the position of the first 1000 words inside each attribute and associate an number to each attribute (the first attribute will have positions 0-999, the second attribute will have positions 1000-1999, etc.)\nFor multi-valued string attributes, we need to ensure that the last word of a string isn't considered to be next to the first word of the next string. To do that, we increment the first position of each string by 8 each time we move from one string to another. For example if a record contains the following \u201cActors\u201d attribute, we will store the word \"Diesel\" with a position of 1 and the word \"Paul\" with a position of 10 (+8 compared to the value in the same string). This will generate a proximity of 8 if you type \"Diesel Paul\", which is the highest value for proximity.\n\n\n\n2.3.2 Bonus challenge: multi-word alternatives\nComputing a proximity distance can be very complex when you support","record_index":5},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 alternative word matching, e.g. synonyms. If you have a synonym that defines \"NYC\" = \"New York City\" = \"New York\" = \"NY\", you would like to consider a record that contains \"NYC\" as strictly identical to a record that contains \"New York\". We solve this tricky problem by rewriting all word positions at query time.\nTime for an example. Let's say that you have two different articles that you want to be searchable via a query for the \"New York subway\". You would like the proximity distance calculation to be the same between them. Here are the article titles:\n\n\"Why New York Subway Lines Are Missing Countdown Clocks\" \n\"NYC subway math\"\n\nWhen this query is run, the first record will be matched via three words (\"new\" at position 1, \"york\" at position 2 and \"subway\" at position 3) and the second record will be matched via two words (\"nyc\" at position 0 and \"subway\" at position 1).\nIn order to avoid this inconsistency, we rewrite positions using a multi-word expansion. For the \"NYC subway\u201d query, we apply the following transformation:\n\n\"Nyc\" at position 0 became:\n\nVirtually \"new\" at position 0\nVirtually \"york\" at position 1\n\n\n\"subway\" at position 1 became:\n\n\"Subway\" at position 2 because we introduced \"york\" word at position 1.\n\n\n\nAfter the transformation, we have exactly the same proximity measure between the two records. This rewriting is entirely cumulative. For the \"New York\" query, if a record contains \"NYC\" at positions 5 and 15, it means that all original positions greater than 15 will be increased by 2 and all original positions between 6 and 15 will be increased by 1.\n\n2.3.3 Attribute-aware proximity\nWhen computing the proximity score, we don\u2019t want to be misled when different parts of the search expression are found in different attributes of the record.\nSay you\u2019re looking for \"soup of the day\" and you have a record with \"soup\", \"of the\" and \"the day\" separately in three different attributes. In this case we do not want to compute a total distance of 10","record_index":6},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 (distance(soup, of)=8 + distance(of, the)=1 + distance(the, day)=1) because it would mean \u201cof the day\u201d are close together, which is not the case. Instead, we want a total distance of 17 (distance(soup, of)=8 + distance(of, the)=1 + distance(the, day)=8). The distance of 8 between \"the\" and \"day\" tells us that \"day\" was found in a different block than \"of the\". We use a dynamic programing algorithm to do this computation efficiently with a cost of O(N) (N being the number of matched positions).\n2.3.4 Proximity computation tuning\nThe minProximity index setting allows to change the minimum proximity to something that fits your data better. If you set minProximity=2, it would mean that the query \"New York City Subway\" will have the same proximity score whether the record contains \"New York City Subway\" or \"New York City's newest subway\". The distance would normally be 1+1+1 for the first record and 1+1+2 for the second, but setting minProximity=2 means that all pairs of words with a distance less than or equal to minProximity are counted for 1. In this example, both records are given a proximity criterion score of 3.\n\n2.4 Computing the most important attributes\nAs a reminder, the attribute criterion lets us say that a query match in some attributes is more important than others.\nEach attribute is indexed as ordered or unordered (this is configured in the index settings). Indexing an attribute as ordered means that the position of the best matched word inside the attribute is taken into account during ranking. So earlier words will matter more. Indexing as unordered means that the position within the attribute is not taken into account. Word position is not important.\nAn example\u2014if you index the title attribute as ordered and description attribute as unordered, the query \u201cNetflix\u201d will return the following 2 records in the order displayed.\n\n\n2.4.1 When proximity is before attribute in the ranking\nThere are two strategies to compute best-matched attributes and","record_index":7},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 they depend on the relative position of proximity and attribute criteria in the ranking formula. Let's illustrate them by using the query \"VMware CEO\" to retrieve the following record:\n\n\nIf proximity is before attribute in the ranking formula, we first find the location with the best proximity in the record and we then compute the best attribute on this location. On the record below, the query \u201cVMware CEO\u201d will have an attribute criterion of 1000 as the best proximity is found in the description attribute instead of the title attribute.\nIf attribute is before proximity, the best-matched attribute will be decided by the position of the best-matched word in the record. In this case, the query \u201cVMware CEO\u201d will have an attribute criterion of 0 as the best word is matched in the title attribute.\n\nWe recommend keeping the proximity criterion before attribute criterion. Proximity usually leads to a better identification of the best-matched attribute.\n\n2.5 Computation of the exactness criterion\nThe exactness criterion represents the number of words that match entirely, not one as a prefix of another. This criterion works well when the query contains several words but the one-word case requires a modified approach. In an instant search use case, if a user has typed \u201cwas\u201d, there is a greater chance that he or she might be in the middle of typing a search for \u201cWashington\u201d and not just stopping at the verb \u201cwas\u201d. Displaying \u201cwas\u201d before \u201cWashington\u201d in the results at that moment would not be ideal.\nWe have implemented three different ways to compute the exactness criterion for a single word query, each specified via the exactOnSingleWordQuery parameter:\n\nexactOnSingleWordQuery=none: The exact criteria is set to 0 for all one-word queries.\nexactOnSingleWordQuery=attribute: All records that have an attribute string exactly equal to the query will have exact=1. This option is the default as it works well for databases that contain names, brands,","record_index":8},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 companies, etc.\nexactOnSingleWordQuery=word: All records will have exact set to 1 if there is a word that matches exactly and is not in our stop word dictionary (see the removeStopWords option for more information). For the query \u201cprince\u201d, a record with \u201cprince\u201d will have exact=1 and a record with \u201cprincess\u201d will have exact=0. For a query \u201cwas\u201d, both records containing \u201cwas\u201d and \u201cWashington\u201d will have exact=0 because \u201cwas\u201d is in the stop word dictionary.\n\nThere is unfortunately no magic option that works well with all use cases, which is why we introduced the exactOnSingleWordQuery parameter. We recommend using exactOnSingleWordQuery=attribute if you have a database that contains lots of one-word entries like brand names, account names, movies, etc. We recommend using exactOnSingleWordQuery=word when you have content like titles that have a low probability of matching an entry using only one word.\n\n2.6 Designed for small records\nThe five components of textual relevance we\u2019ve discussed are uniquely designed for small, structured records. A few of them require some fairly complex processing, like the query-time rewriting of word positions for multi-word alternatives, but we think that having an approximation-free, exact value computation is worth it.\nIf you don\u2019t automatically have small, structured records\u2014i.e. if you\u2019re starting with documents or long blocks of text\u2014we recommended splitting them into multiple, shorter records. One record per paragraph is a good rule of thumb. This is the approach used by our DocSearch project, which crawls web pages containing technical documentation and can generate many records per page. DocSearch is now powering the search of 150+ open source projects, so there are lots of places you can try it out.\nWe\u2019re constantly seeking more ways to customize relevance for advanced use cases. Some of the parameters presented in this post have been added to the API recently and several new parameters are","record_index":9},{"post_id":5220,"post_title":"Inside the Algolia Engine Part 4 \u2014 Textual Relevance","post_date":1467869787,"post_date_formatted":"Jul 7th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-enginepart-4-textual-relevance\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/indexing-part2-copy-360x200.png","time_to_read":19,"content":"\u2026 currently being beta tested.\nA few weeks ago we were given a challenge: to index millions of articles of a popular news website and provide relevant results about the artist Prince\u2019s death, even for queries with typos like \"Prinse\". The query \u201cPrinse\u201d matches article with \"Prince\" in the title (typo=1, words=1, proximity=0, attribute=0, exact=0) but the results were being displayed below other articles about \"Sharon Prinsen\" (typo=0, words=1, proximity=0, attribute=1000, exact=0) because it matches a prefix without typo. To solve this issue, we introduced a new option to consider all prefix matches as a typo: now all articles with \u201cPrince\u201d in the title will be served first (typo=1, words=1, proximity=0, attribute=0, exact=0) and articles mentioning \"Sharon Prinsen\" will come after (typo=1, words=1, proximity=0, attribut=1000, exact=0). With this improvement, we were able to address the relevance problem of prefixes while maintaining a high quality instant search experience.\nRelevance is our passion and we love to be challenged. Feel free to contact us if you have a complex use case with relevance issues, we\u2019d be happy to help!\nWe recommend to read the other posts of this series:\n\nPart 1 \u2013 indexing vs. search\nPart 2 \u2013 the indexing challenge of instant search\nPart 3 \u2013 query processing\nPart 5 \u2013 Highlighting, a Cornerstone of Search UX\nPart 6 \u2013 Handling Synonyms the Right Way\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":10},{"post_id":5189,"post_title":"Hello, Hi, Hey from our brand new Synonyms API!","post_date":1466150055,"post_date_formatted":"Jun 17th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/brand-new-synonyms-api\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/synonyms-360x200.png","time_to_read":4,"content":"At Algolia we\u2019re big on choice. We appreciate that different people search for the same things using different queries and keywords. Therefore\u00a0incorporating synonyms has played a crucial part in making our search \u201cintelligent\u201d.\nOver the last two years, we\u2019ve refined and reworked our synonyms feature to make it one of the most comprehensive in the market.\u00a0It now\u00a0supports almost all types of expressions: multi-word expressions, abbreviations, expressions containing symbols and more. It even supports advanced features such as placeholders. Today, Algolia is excited to announce the release of a dedicated API for you to manage synonyms!\nWhy a new API?\nWe\u2019ve made significant improvements to our synonyms feature since its inception, but we also know that the configuration methods for it need work. Up until today, you had to configure synonyms via an index setting. This approach worked perfectly well if you needed to add just a few synonyms but it became harder to manage with regard to\u00a0speed if a lot of synonyms needed to be on the list. There were three main drawbacks:\n\nAdding more synonyms had become a challenge (you could not use the dashboard to configure a large set of synonyms but had to write a few lines of code to push all of them as a new index setting)\n\n\nLong lists of (thousands of) synonyms impacted the performance of the getSettings API call and the performance of search to a certain extent (settings are read when opening an index)\n\n\nConfiguring the same synonyms on replica\u00a0indices was a time-consuming process and left something to be desired for\n\nWhat will I get with this\u00a0API?\nWe\u2019re primarily developers and fully comprehend the importance of backward compatibility. You can still use your previous methods of configuration via an index setting and decide when you want to use this new API. \u00a0In other words: there are no breaking changes! Your production code will continue to work as-is with support from the old API, which is still very much in","record_index":0},{"post_id":5189,"post_title":"Hello, Hi, Hey from our brand new Synonyms API!","post_date":1466150055,"post_date_formatted":"Jun 17th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/brand-new-synonyms-api\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/synonyms-360x200.png","time_to_read":4,"content":"\u2026 existence and receives TLC on a regular basis. \nWe\u2019ve merely introduced a small change in our new API clients: the getSettings method does not return synonyms and you need to use our new specialized search Synonyms method without any query, in order to retrieve all of them.\nThis new API is already supported by all of our API clients and you can start using it today. The documentation is available on this page. The major improvements you can benefit from are:\n\nEase of use: one API endpoint for 99% of the use cases; you can just upload your synonyms in one big batch via the batch API call, using the parameter, replaceExistingSynonyms boolean, set to true.\n\n\nScalability: this new endpoint supports millions of synonyms without impacting dashboard and search performance\n\n\nEnhanced Dashboard: we built a new interface in the dashboard to search and edit synonym sets\n\n\nEase of Reproduction: you can use the forwardToReplicas boolean parameter to forward the synonyms to all replica\u00a0indices (no more painful replication of synonyms!)\n\nAvailable now!\nThis new feature has already been deployed and is available to all of our users. We hope you love this new API and have as much fun with it as we did while making it. \u00a0As always, we eagerly await your feedback, which we value deeply and and use\u00a0to help improve the product!\nHappy coding!\n ","record_index":1},{"post_id":5169,"post_title":"Introducing Algolia Places","post_date":1465972035,"post_date_formatted":"Jun 15th 2016","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/introducing-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/places-featured-360x200.png","time_to_read":4,"content":"Today we\u2019re excited to release Algolia Places, a fast and easy way to turn any HTML input into an address autocomplete.\nSee it live \u2022 Read the docs\nOne of our core missions at Algolia is to improve search-related user experiences. Address forms were high on our list because they're difficult to build and often\u00a0represent\u00a0a\u00a0critical step in user flows like sign up and check out. Our goal with Algolia Places is to provide something that\u2019s both easy for developers to integrate and enjoyable for visitors to use. It\u2019s also highly customizable and can be enriched with additional data sources.\n\nAlgolia Places is packaged as a simple, lightweight JavaScript library and is available on npm and jsDelivr. You can find comprehensive documentation and live examples on the Algolia Places website.\nBeautiful by default\nAt Algolia we try to build tools with sensible design defaults. The Algolia Places JavaScript library displays a modern and clean dropdown menu from the moment you install it. A few lines of CSS are all you need to customize it to the exact look and feel of your site. All of the typo-tolerance & highlighting capabilities of the Algolia search API are already built in.\n\nEasy to use & easy to extend\nHere\u2019s a snippet of code you can use to add Algolia Places to any <input> element on your site. No prior setup or API key required.\nView the code on Gist.\nIn some cases, you\u2019ll want to customize the type of data that is returned and how it is presented. Algolia Places exposes additional configuration options that allow you to:\n\nRestrict search to specific countries.\nRestrict search to specific types of places (e.g. country, city, address).\nSearch for and retrieve localized place names.\n\nCombine places with\u00a0search results\nAlgolia Places is compatible with the autocomplete.js library. You can easily build interfaces that combine address search results with other results from autocomplete.js data sources, including Algolia indices. Here\u2019s an","record_index":0},{"post_id":5169,"post_title":"Introducing Algolia Places","post_date":1465972035,"post_date_formatted":"Jun 15th 2016","author_name":"Vincent Voyer","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/d27b84b2f345cee4af097edb753d42b5?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/introducing-algolia-places\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/places-featured-360x200.png","time_to_read":4,"content":"\u2026 example. The Cities section contains results from Places and the Vacation Rentals section contains results from a conventional Algolia index.\nAlgolia Places is also compatible with instantsearch.js, a library for building full-page search-as-you-type experiences. See the advanced examples to learn more.\nTry it now\nAlgolia Places is free up to 1,000 requests per day. This should be enough to get familiar with the service and deploy smaller applications to production.\nContact us at places@algolia.com if you\u2019d like to use Algolia Places on a larger scale. We\u2019re happy to discuss increased quotas or chat with you about custom plans.\nLike many of our projects, Algolia Places is open source. The Github repository has everything you need to open issues, have conversations and submit pull requests. Contributions from beginners and experts are all welcome.\nWhat\u2019s next?\nWe\u2019re continuing to enrich the underlying data and add more features. We\u2019re working on a detailed blog post about how we built Algolia Places with OpenStreetMap\u00a0and\u00a0GeoNames.\nIn the meantime we\u2019re excited to get your feedback and hear about your experiences!\nTry it now","record_index":1},{"post_id":5150,"post_title":"How Algolia Increased TradePub.com\u2019s Search Conversions by 70%","post_date":1465809494,"post_date_formatted":"Jun 13th 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-increased-tradepub-coms-search-conversions-by-70\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-27-Copy-2-360x200.png","time_to_read":4,"content":"Content is the hottest thing on the internet today. It\u2019s a medium which is simple yet flexible in terms of how it can be used \u2013 for marketing, sales, information discovery, and even, at times, to look cool at parties! It\u2019s also a medium which is heavily dependent on search functionality. In fact, that\u2019s what brought TradePub.com to Algolia.\u00a0\nDavid Fortino, SVP, Audience & Product at NetLine Corporation, recently reached out to share TradePub.com's\u00a0Algolia\u00a0story with us. Here\u2019s what he had to say.\n\nWhat does TradePub.com do?\nLiving in an on-demand world is an amazing thing. Satisfying that demand though is a different ball-game altogether. TradePub.com is one such player that satisfies the demand in the \u201ccontent\u201d sphere by curating resources on behalf of some of the world's largest and most influential companies. The TradePub.com research library is one of the top destinations for professionals to access free research, white papers, reports, case studies, magazines, and eBooks.\nTradePub.com is owned by NetLine Corporation, a top B2B\u00a0content syndication lead generation platform, reaching 125 million business professionals across 300 industry sectors.\nHow We\u00a0Do it\nAs a top research destination for professionals, TradePub.com is built on sophisticated recommendation algorithms that dynamically match content with users throughout their onsite experience as well as beyond the on-site experience via 1-to-1 personalized email newsletters. However, one feature that had not received much TLC or investment in terms of time, was Search. At NetLine, our Product Team was focused on content discovery and recommendation modules on TradePub.com, and didn\u2019t have much bandwidth to optimize the onsite search feature...but then again, how important was search if TradePub.com\u2019s content recommendation technology perfectly did what it was supposed\u00a0to?\nWhy Search?\nWe\u00a0knew that TradePub.com\u2019s search functionality needed work.\u00a0The accuracy and relevance of","record_index":0},{"post_id":5150,"post_title":"How Algolia Increased TradePub.com\u2019s Search Conversions by 70%","post_date":1465809494,"post_date_formatted":"Jun 13th 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-increased-tradepub-coms-search-conversions-by-70\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-27-Copy-2-360x200.png","time_to_read":4,"content":"\u2026 search results constantly received a poor rating from internal parties; however, search only accounted for < 2% of monthly page views \u2014 therefore, upgrading the search functionality was difficult to prioritize on the project roadmap.\nWhile we\u00a0had a hard time directly attributing search to a significant percent of page views and revenue, we did realize that it\u00a0contributed to the onsite experience of thousands of TradePub.com users daily. Therefore, a correlation could be made that thousands of users were receiving a poor user experience, and we were definitely not satisfied with this statistic.\nHistorically, our\u00a0search feature was internally built and supported by NetLine\u2019s Engineering Team. To improve search results and implement additional features such as auto-complete, the tool would require a complete rebuild.\nAfterall, In-House Tool Rebuild = Time + Resources\nWhat Sold Us\u00a0on Algolia\nWe recognized that\u00a0though search was not a top priority, it was\u00a0still an important feature which added to\u00a0the TradePub.com UX. With this in mind,\u00a0we started looking\u00a0for 3rd\u00a0party solutions that could bring TradePub.com up to industry standards quickly, without pulling significant time and resources away from high priority items.\nNetLine had several key features on the \u201crequired\u201d list when interviewing Search providers: fast results, responsive design\/display, auto-complete and data analytics. As such, an exploratory process commenced that focused on the evaluation of various search technologies in the market. It was through this process that we discovered Algolia. While there was a wide assortment of customizable features reviewed throughout the evaluation process, the following key components moved Algolia to the top of the list:\n\nAuto-Complete Drop Down:\u00a0Users could\u00a0avoid the Search Results Page and preview immediate results.\n\n\nInstant Results:\u00a0Users that reached the Search Results Page received dynamically generated result sets as they type search","record_index":1},{"post_id":5150,"post_title":"How Algolia Increased TradePub.com\u2019s Search Conversions by 70%","post_date":1465809494,"post_date_formatted":"Jun 13th 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-increased-tradepub-coms-search-conversions-by-70\/","categories":["Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-27-Copy-2-360x200.png","time_to_read":4,"content":"\u2026 queries.\u00a0Try it!\n\n\nFaceting:\u00a0User facing search refinement options were available, for example: by content topics.\n\n\nFully Responsive: Users could benefit from the fact that it\u00a0worked perfectly across all devices.\n\n\nHuman Centric\u00a0Error Handling:\u00a0Automatically handled typing mistakes, concatenation, and split (\"front end\" vs \"front-end\") queries.\n\n\nAnalytics:\u00a0Easy dashboard reporting complete with:\n\nTop Searches\nTop Searches to yield zero results\nTop refinements\nGeo\n\n\n\n\nOnce the t\u2019s were crossed and the i\u2019s dotted by my colleagues on our cross-functional Product Development team,\u00a0we\u00a0moved quickly to begin the implementation phase. Search was implemented between other top priority projects; but from product discovery to completion, the process was fully implemented within two months.\nThe Bottom Line:\nFollowing the implementation phase, we\u00a0captured and published two months of search data. Initial results comparing the new Algolia Search Feature with 6 months of data from the original in-house search tool are as follows:\n\nAverage Monthly Search Queries increased more than 200%\nAverage Monthly Conversions driven by Search increased more than 70%\nAverage Monthly Revenue driven by Search increased more than 30%\n15% of Search conversions are driven by the Search dropdown menu versus the Search Results Page.\n\nNeedless to say, our team at NetLine is thrilled with the early results \u2013 and the entire TradePub.com audience is reaping the benefits of the new advanced search experience.\n ","record_index":2},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"As a hosted-search engine service, we discuss the relevance aspect of search with our customers and prospects all day long. We now have more than 1500 customers and have seen a large\u00a0variety of real-life search problems. It's interesting to note that more often than not, these problems are in some way connected to the R word. Relevance.\nRelevance is a well understood concept in search engines, but is pretty complex to measure, as a high degree of subjectivity is implicit in the notion of relevance. In fact, we've seen time and again\u00a0that too many people spend too much time\u00a0trying to control their relevance.\nIn this post, we'll share our top 10 tips to help people achieve\u00a0good relevance in their search results.\n1) Structure your data\nIt might seem like we're stating the obvious, but the first step to having good relevance is structuring your data correctly. Having a long string that contains all your information concatenated won\u2019t put\u00a0you on the path to\u00a0good relevance.\nYou should have an object structured with different attributes and different strings to help the engine associate the importance of matches in your preferred\u00a0order. Avoiding string concatenation to list values also ensures that the proximity measure will not be reflected inaccurately because the last word of one value is close to the first word of another value. (Proximity is an important element of the relevance that measures how close the query terms are in the matched document.)\nHere is an example using\u00a0a movie title:\n\nWhich works better than:\n\n2) Handle typo-tolerance\nBefore doing advance tuning, there are a lot of small checks you can do to achieve a\u00a0decent level of textual relevance. For example you should be able to find a record that contains  for the query  and  or find a record containing  with a query . You should also check that you handle abbreviations and are able to find a record containing  with the  query and vice versa.\nTypos are very frequent, especially because of the","record_index":0},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"\u2026 small virtual keywords of mobile devices (and the famous fat-finger effect). If your record contains , you should be able to find it via  or . Users also love as-you-type search experiences, so you should ensure\u00a0that you are able to tolerate typos on the prefix. For example, the code\u00a0should be such that the \u00a0query \u00a0is considered as a prefix of  (because mikca = one typo of micka).\nAll these cases are automatically handled for you by the Algolia engine.\n3) Don\u2019t fix issues locally, think global\nRelevance is not a one day job, and you will discover specific queries with relevance issues over time. You should try not to optimize your relevance for a specific query without having the big picture in mind and having a setup that allows you to efficiently test your modifications on the entire set of\u00a0query logs ensuring you won't degrade the relevance of other queries. A good non-regression test suite is mandatory.\n4) Stop thinking about \"boost\"\nWhen we talk\u00a0to different search teams, we see that they are all used to configuring  and it comes as a reflex for them. By boost they mean integers that they configure in their ranking formula like \"I configured a boost of 5 on my title attribute, 2 on my brand attribute and 1 on my description attribute\" which is kind of code\u00a0for \"The attribute title is 5 times more important than description and brand is twice as important as description\".\nUnfortunately these  are not so great. Changing one from a value X to Y is the most common source of issues we see everyday! No engineer is able to predict what will happen because this boost will be combined with a lot of other factors that make the modification unpredictable (you can see it as one integer mixed with other elements in a big mathematical formula). In other terms, even if you have non-regression tests, a change of boost will just totally change your result set and it will be close to impossible to say if the new configuration is better than the previous one or not.\nThis","record_index":1},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"\u2026 is actually why we designed a different way to handle search relevance with a predictable approach.\n5) Always explain the search results\nSometimes your search can be good but the user can perceive the results as bad because there is no explanation of the match.\nThe best\u2013and most intuitive\u2013way to explain a search result is to visually highlight the matching query terms in the result, but there are also two cases that are often not well handled:\n\n1. When you are searching in multiple attributes but only display one of them statically: in this case it would be better to display the matching attributes. This is, for example, the case if you return \"Elon Musk\" for the query \"Tesla.\" This is great, but it would be even better to add a sub-line with \"CEO: Elon Musk\" to make sure someone who\u00a0doesn't know Elon Musk would understand the result.\n\n\n\n2. When you are matching via a typo: it can sometimes be non-intuitive to see where the typo is, and highlighting the matched term is very useful to help the user quickly understand the match.\n\n6) Think twice before removing stop words\nAs obvious as it seems, before trying to use a recipe you should be sure that your use case is compatible with it. This is the case with stop word removal (removing the most commonly used words in\u00a0a given language like , , , , , \u2026).\nBut those words can be very useful and removing them sometimes hurts the relevance. For example, if you try to search for the \"To Beta or not to Beta\" article on hacker news without stop words, the engine will end up with the query,\u00a0Beta.\nThere are even worse queries, like if you want to search the artist \"The The.\" In this case you would just have a no-results page!\nOf course, there are cases where removing stop words are useful. If you have a query in natural language or if you are trying to search for similar content. But those cases are more of an exception than the norm. Be wary of removing stop words!\n7) There are entries that are complex to find\nThere will","record_index":2},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"\u2026 always be some specific queries that can be complex to handle, this is the case for the TV show called \"V.\" This query is particularly challenging in an instant search use case:\n\nFor the sake of good UX, you should launch the search from the very first character typed (we\u2019ve tested it many times, and have found that any heuristic that launches the query after N letters leads to a poorer UX and conversion rates)\n\n\nYou should have a preference for the exact match over the prefix because there are probably a lot of words that start by  in your data set.\n\nAnother type of corner cases is the usage of symbols, this is the case if you are looking for the band \"!!!.\" We encounter such problems with symbols in almost every use case.\n8) Be careful with language specific techniques\nNatural languages have a lot of variety that can cause your records to not be returned. For example, if you are using a singular word in your query and your record contains the plural word. There is some language specific heuristics that help to address this problem. The most popular are:\n\nStemming: Reduction of a word to the simplest form called a stem. Most of the time by removing the suffix of the word, for example transforming  in . The most popular open source stemmer is Snowball and is based on a set of rules per language.\n\n\nPhonetization: Compute a replacement of the word that represents its pronunciation. Most phonetic algorithms are based or derived of the Soundex algorithm and only work with English language.\n\n\nLemmatization: Reduction of all different inflected forms of a word. This is similar to the stemming approach except it is based on a dictionary developed by linguists, and it usually contains mainly nouns and verbs.\n\nThe major drawback of these approaches is that they only address one language. We see in practice very few cases when there are only words from one language and those techniques can produce noise on proper names such as last name or brand. You can, for example, think","record_index":3},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"\u2026 about a search of people on a social network, where those approaches can introduce bad results.\n9) Use business data in your relevance\nThe first eight tips target the textual relevance, but you should also include business data in order to have good relevance. It can be just a basic metric like the number of page views or something more advanced like the number of times a product was put in a cart.\nIt can even be an advanced metric\u00a0which relates\u00a0to the query like \"the number of times a product was bought when searched with a particular query\".\nFrom our experience, the addition of business data makes a big difference if the textual relevance is good. That said, the business relevance should not bypass the textual relevance or you risk loosing\u00a0all the benefits of the hard work done on relevance! Textual relevance should (almost always) go first and in case the textual relevance doesn\u2019t help to decide whether one hit or the other should go first, then the engine should use the associated business data.\n10) Personalize the ranking for each user\nPersonalization of search is the final touch to get the perfect relevance and is the part that most people don\u2019t really see. Let's take a simple example: if you search for \"milk\" on your favorite grocery store that applied all the previous tips, you will find\u00a0the most popular milk bottle. But if you are a regular user of this store and have already bought a particular bottle of milk several times in the past, you\u2019re likely to expect this one first. This is the ultimate way to make the user love the search result and avoid the perception of a bad relevance. In other words, it's the icing on top of the cake!\nNot an exhaustive list\nWe hope this list of advice will be useful to help you get a better search functionality on your website or app. This list is unfortunately not exhaustive as relevance is a pretty complex domain and there are a lot of specific problems that we do not cover in this list.\nOur team is dedicated to","record_index":4},{"post_id":5131,"post_title":"Algolia's top 10 tips to achieve highly relevant search results","post_date":1464940936,"post_date_formatted":"Jun 3rd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/algolias-top-10-tips-to-achieve-greatly-relevant-search-results\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/Group-51-360x200.png","time_to_read":9,"content":"\u2026 help you have a better relevance, fill free to contact us at contact(at)algolia.com to share your problems and we will be happy to analyse them with you.","record_index":5},{"post_id":5111,"post_title":"Our Post Mortem of the DNS DDoS which took place on Monday May 16th","post_date":1464757226,"post_date_formatted":"Jun 1st 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/dns-ddos-of-monday-may-16th-post-mortem\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/ddos-360x200.png","time_to_read":7,"content":"Milliseconds and transparency matter at Algolia- both inside and outside our company. So we thought we\u2019d share the details and our post mortem of the DNS DDoS incident which took place on Monday, May 16.\nIt all started when we received an alert from our monitoring system - the globally geo-routed endpoint to our API was not available. Despite our fallback plan where we\u2019d designed our API clients to target another endpoint if the geo-routed one is unavailable, the problem still occured. We immediately started investigating the issue because it could slow down the speed of search queries for our end-users which we absolutely could not have.\nIdentifying the Cause\nEarlier that day, we had performed an hour-long maintenance procedure on our monitoring system and verified that it was working correctly on our performance dashboards. When the alert about our main endpoint being unavailable came, we saw a similar drop in the number of\u00a0queries reaching our API and we first suspected a monitoring error. Some local tests showed that the endpoint was responding but was incredibly slow. Our worries were confirmed when we saw that the DNS resolution was actually consuming the majority of the query time. A few minutes later, we received an email from our primary DNS provider informing us of\u00a0an incoming DDoS attack. The root cause had been identified - it was time to update our status page and prepare. \nThe first wave of the DDoS came around 14:00 UTC and lasted till 17:30 UTC. The second one came at 19:30 and lasted till 20:30. At the peak of the attack we lost 25% of the traffic, as can be seen on the following graph.\n\nLearning from our mistakes\nThis was not the first DDoS attack that either Algolia\u00a0or our primary DNS provider has seen. In fact, we\u2019d already put some measures in place for just such an eventuality. Last year, we updated our API clients with what we call the \u201cDNS Fallback\u201d. This allows our API to operate on two independent DNS networks and fallback from","record_index":0},{"post_id":5111,"post_title":"Our Post Mortem of the DNS DDoS which took place on Monday May 16th","post_date":1464757226,"post_date_formatted":"Jun 1st 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/dns-ddos-of-monday-may-16th-post-mortem\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/ddos-360x200.png","time_to_read":7,"content":"\u2026 algolia.net to algolianet.com in case there is a problem.\nOur DNS providers too have DDoS mitigation solutions in place and have a lot of capacity to handle attacks. When forced to explain this new problem, we realised something was not working correctly in our DNS retry strategy. Despite our efforts, we noticed that\u00a025% of our requests dropped. We immediately suspected two sources: usage of outdated API clients (without the DNS fallback) or buggy handling of DNS timeout in some of them.\nEven when DDoS mitigation is triggered quickly, it takes minutes to get rid of the simplest attacks. This is long enough to affect our users\u2019 search availability. That's why we're tuning the timeouts of all the requests in our API clients itself in order to bring the impact close to zero.\nThe Good, The Bad and The Ugly\nAlthough we had introduced the DNS fallback a year ago, we still see usage of very old versions of our API clients. During this year, we tried to eradicate the usage of our old clients by sending notices to the impacted users and introducing a warning in our dashboard. Unfortunately we did not manage to remove all instances of old client usage - there were probably a couple of components missing in our messages since we\u2019d not discovered a good enough incentive to get people to upgrade an API client that worked just fine, as\u00a0there hadn\u2019t been any outages. On the bright side though, when most people using old clients (without fallback support) came to us, we asked them to upgrade their API clients which resolved the issue instantly.\nBut we also discovered during this attack, that we trusted our fallback implementation a bit too much. We started to test all API clients\u2019 implementation by replicating the conditions of the DDoS attack. For these\u00a0tests, we created a new testing domain algolia.biz. This domain timeouts all the requests due to non-responding name-servers.\nWe officially support 15 API clients and here is the overview of what did (or did not)","record_index":1},{"post_id":5111,"post_title":"Our Post Mortem of the DNS DDoS which took place on Monday May 16th","post_date":1464757226,"post_date_formatted":"Jun 1st 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/dns-ddos-of-monday-may-16th-post-mortem\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/ddos-360x200.png","time_to_read":7,"content":"\u2026 work.\nThe Good\nRuby (also Rails), Python (also Django), Go, Swift, C#, and Android API clients passed the new test with flying colours.\nThe Bad\nThe bad one turned out to be the JavaScript API client. In the recent versions we introduced a bug that seems to have disabled the DNS Fallback. That bug was triggered when the DNS browser timeout or the DNS system timeout was triggered before our internal API client timeout.\nFortunately, this didn\u2019t occur for every browser or OS. When a browser fails at resolving a DNS server on time, there\u2019s no timeout exception raised by the browser but rather an error on the underlying XMLHttpRequest object. Internally, we use XMLHttpRequest errors to decide to use JSONP in case XMLHttpRequests are blocked. A recent fix on that JSONP fallback introduced a bug when facing a DNS resolution error.\nWe advised our clients to use the last confirmed working version if they were experiencing issues: version 3.13.1.\nThe good news is that we\u2019ve now reworked the fallbacks and the latest client is working perfectly today.\nThe Ugly\nJava and Scala clients using the standard Oracle JVM had unexpected results with our new algolia.biz testing domain. While the new test worked locally, it kept failing on Travis CI which we use for testing all of our libraries. After carefully tracing the application calls, we discovered 2 things:\n\nThere is no way to set a timeout on DNS resolution, but there is a workaround: http:\/\/bugs.java.com\/bugdatabase\/view_bug.do?bug_id=6450279. The JVM uses the timeout of the underlying OS. That explains the difference between our workstations and Travis CI.\nThe JVM has a DNS resolution cache that is enabled by default (more information on networkaddress.cache.ttl )\n\nAlthough the second one was not an issue during the DDoS, depending on the OS the first one could have been.\nThis implies that some work needs to be done for both the Java and the Scala API clients. For the Java client, we will do it in the upcoming v2. For the","record_index":2},{"post_id":5111,"post_title":"Our Post Mortem of the DNS DDoS which took place on Monday May 16th","post_date":1464757226,"post_date_formatted":"Jun 1st 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/dns-ddos-of-monday-may-16th-post-mortem\/","categories":["Infrastructure"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/06\/ddos-360x200.png","time_to_read":7,"content":"\u2026 Scala\u00a0client, we need to upgrade the underlying HTTP client, which will take some time as we need to change the underlying architecture of the client.\nFor PHP, \u00a0the situation is even trickier. We are using the CURL library to perform all the requests. Unfortunately, the CURLOPT_TIMEOUT and CURLOPT_CONNECTTIMEOUT options do not include DNS resolution time and PHP uses the timeout of the OS. Luckily, if you have the \u201cares\u201d extension installed it sets a CURL_TIMEOUT_RESOLVE that handles DNS timeout.\nIn Closing...\nWhen we implemented the DNS fallback strategy earlier last year, we were confident it was the very last required piece of code to implement the high-availability of our API. Testing such a DNS fallback strategy is complex and it turns out that not having the ability to perfectly reproduce all the conditions of the attack - be it the OS configuration or weird behavior of the underlying HTTP library you don\u2019t understand-\u00a0was more of a handicap than we thought.\nToday we have a dedicated domain name and robust tests to ensure that our fallback code is working in order to alleviate this problem in the future.\nAnd finally, if you are an Algolia customer, please ensure that you are using the latest version of our API clients in order to avoid such impact in the future.","record_index":3},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"Search engines and query processing are not recent in Computer Science: this field known as Information Retrieval has a pretty vast set of state of the art practices. Today most search engines on the market come with a large set of features that developers can use to create their query processing pipeline, but this task is far more difficult than it seems and most people never manage to achieve a good result. In this post, we will cover the classical approaches and how we are handling it in the Algolia engine.\nIntroduction to query processing\nTo be able to find results, a search engine has to be able to understand deeply what was asked. That\u2019s the role of the query processing: to analyse the query, and eventually transform it to make it easier to process by the search engine.\nAnd to do so, a search engine will process the query in two big steps, both of which can be very challenging.\n\n1)\u00a0Detect words in the query: this process is called tokenization and identifies what is a word by answering questions like: \n2)\u00a0Search for alternatives on this set of words: the goal of this step is to be less rigid by adding alternative corrections or synonyms, to avoid missing results that don\u2019t contain exactly what was searched, but something equivalent. It decreases the need to reformulate the query to find what you want.\n\nWhy is tokenization complex?\nThe process of detecting words involves a set of rules to determine what is a word. It is applied both on the query and when indexing the words, to ensure both have the same format of words, which makes them easier to match.\nThis process seems simple at first \u2013 at least for alphabet-based languages \u2013 but as often, complexity creeps in when you consider the edge cases. And you soon realize that it\u2019s way harder than it seemed, with no easy solutions.\nTo give you an idea of the complexity of tokenization, here is a non-exhaustive list of tokenization issues:\n\nSymbols: You cannot simply ignore symbols. C, C++ and C# are","record_index":0},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 different words. And there are expressions that are composed of symbols only like the \u201c!!!\u201d band!\nCompound words: some languages like Dutch and German can aggregate words to form one expression. For example in Dutch, the word  means  and is composed of 3 words: arbeid (labour) + ongeschiktheid (inaptitude) + verzekering (insurance). Of course you would like to have a document containing this long expression match on the  (insurance) query\nAgglutination: some languages like Japanese, Chinese and Korean don\u2019t \u00a0have separators between words (i.e. spaces). Those languages require a statistical approach that helps to detect words in the sequence of ideograms, the search engine also needs to be reliable to an error of word recognition. As with compound words, you want to be able to search for a document with just one of the words.\n Acronyms with Punctuation: you want to find a record containing U.S.A with the USA query and vice versa\nCommon words containing periods: sometimes the period can have an importance like for domain names (.com, .net, .org), titles (mr., dr.) or abbreviations (st.)\nHyphenated words: the hyphen is not an easy case as it can be sometimes considered as a separator like in  or  but sometimes it is important like in \nApostrophes: they can be used for clitic contractions (we're \u21d2 we are), as genitive markers (Carl's haircut) or as quotative markers\nCamelCase: pretty common practice of writing compound words or phrases such that each word or abbreviation begins with a capital letter. You probably want to find your  tag via the  query\nOthers: HTML entities, dates, time, measures, phone numbers, etc.\n\nTo know what to search for, the engine needs to know which words compose the query. But all of these exceptions make the process of detecting words pretty challenging. Not an easy problem to solve.\n\nWhy is searching for alternatives also complex?\nOnce the query is tokenized, you have a set of words that will be searched in your documents. We will","record_index":1},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 now enrich the query by searching for alternative words: by expanding the set of words we\u2019ll be able to find matches that are close to the query but not exactly identical.\nThis process is only performed on the query and not when we index the record. (In a future post, we\u2019ll explain why doing so on indexing is a bad idea.).\nIf you do not apply this process, you\u2019ll have two problems:\n\n1) You will miss some results if there is a small difference between the words in your records and the query (for example a singular versus plural). You can also have typos or errors in your query that cause those records to be unmatched.\n2) The words in the record can be different than the query and lead to a different segmentation. For example, a record that contains 'hispeed' will produce one token whereas the query 'hi speed' will produce two tokens. This difference is pretty big as we cannot simply search for records containing the query terms anymore.\n\nThat\u2019s when it becomes complex. As you can see in this last example, searching for alternatives can potentially change the number of words in the query, which is always a bit of a mess to manage!\nWe can distinguish four different types of alternatives that can be added to expand the query:\n\n1)\u00a0An alternative word on a query word: your query contains  and your record contains .\n2)\u00a0A multi-word alternative on a query word: your query contains  and your record contains  or if your query contains  and your record contains .\n3)\u00a0A word expression as alternative of several query words: your query contains  and you search for a record containing  or if your query contains  and want to search for a record containing \n4)\u00a0A multi-words expression as alternative of several query words: this is the most complex one to handle, this is the case if you add a synonym set containing . In this case, if your query is , it will be extended in:\n\n\n\nThe first type is properly handled in any search engine as it is just an OR between several terms","record_index":2},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 but very few engines handle natively the last three cases which often require a lot of custom code.\nAlgolia\u2019s way of tokenizing\nOur approach of tokenization was always to keep it as simple as possible. In the end, most challenges of tokenization can be solved via an alternative added at query time! That said, there are some cases which require a special processing at indexing time to be perfect:\n\n1) Some abbreviation & apostrophes: If you have  in your record, it is very unlikely that you want to have it match for the query . Which means you just want to have the token  to avoid noise that could break the relevance. It is the same for the  expression, in this case you just want to have one token. There are of course exceptions, for example  (French expression meaning ) needs to be found as  and we do not want a token \"lhotel\". All those cases are handled automatically by our engine without having to configure anything!\n2) Agglutinative languages: For languages such as Chinese, Japanese and Korean we use a word dictionary with frequencies to split a sequence of ideograms in several tokens. We also detect the change of alphabet as a new token (this is useful, for example, in Japanese to produce a new token when we move from Hiragana to Kanji). All this processing is done automatically and we improve this segmentation all the time based on our user feedback.\n3) Symbols (&, +, _ ...): This part is the only one that is configurable since it depends on the use case and cannot be packaged in a generic way. We have a configuration setting to let you specify the symbols which are important for you. For example we used this approach on a website that helps students. They have to index mathematica formulas and make the difference between  and . In this case we have indexed all mathematical symbols (addition, subtraction, multiplication, \u2026)\n4) Compounded words and camel case: A part of this processing has to be done at indexing if your record contains a compounded","record_index":3},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 word like  and you want to be able to retrieve this record via the  query. That\u2019s the only step that we do not have today in production but which is on our roadmap. (You can expect to see this in the future!)\n\nAt first sight, the process of tokenization can be perceived as simple. But as you have seen it is pretty complex in practice and requires quite a lot of work. We constantly work on improving it.\nWe only address a few of the tokenization challenges described at the beginning of this article. We actually solve most of them via alternatives added after tokenization. The next section describes those alternatives in details.\n\nAlgolia\u2019s way of searching for alternatives\nWhen we launched the engine in 2013, we had only typo tolerance as an alternative. But since then, we have improved the ability of the engine to find your records even with complex mistakes. \nToday the engine expands a query via six different types of alternatives:\n1) Typo tolerance on words: after tokenization, we look for potential approximations of each query word in the index dictionary (it contains all words used in this index). In practice, we compute a Damerau-Levenshtein edit-distance between the query word and each word of the dictionary and accept the alternative if the number of typos is acceptable.\nThis Damerau-Levenshtein edit distance represents the number of letters inserted, deleted, substituted or transposed to change the query word to the dictionary word. For example the distance between  and  is 1 as there is one transposition between  and  (transposition corresponds to a common typing mistake, two letter are inverted in the word).\nIn practice, we simplify the process. By looking at the maximum distance that is acceptable for each word, we apply a recursive scan on the dictionary (represented as a radix tree). Then we prune the branches: optimizing the scan by re-evaluating on the fly what doesn\u2019t need to be scanned. \nBy default we accept a maximum distance of 2 if the query","record_index":4},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 word contains at least 8 letters and a maximum distance of 1 if the query word contains at least 4 letters (although those values are configurable).\nLast but not least, the typo-tolerance function is able to correct a prefix, which is important in an instant search scenario. For example  will be considered as one typo of  which is the prefix of  (In this case the dictionary does not contains the word  but only the word )\nIn our engine, typos are tolerated on words and prefix of words natively, without having to configure extra processes like ngram computation. This native approach has also the big advantage of being very efficient at indexing and producing an exact computation of relevance. You can learn more about the way we store our dictionary and prefixes in part 2 of this series.\n2) Concatenation: Typo tolerance on a tokenized query is already a challenge for performance but this is unfortunately not enough: the query tokenization can be different than the one performed at indexing time. This is the case if the query contains  and the record contains  and you would like to have an alternative with the concatenation of query terms! We automatically compute a concatenation for all pairs of terms plus a general concatenation of all the query terms. Which means, for example, that the query  will generate the query:\n\nIn order to avoid polluting the results by doing an approximation on an approximation, we do not apply typo tolerance on those new alternatives; they need to be identical in the dictionary. In this query the typo tolerance is only applied on each words of the  part.\n3) Split of query tokens: Concatenation is a very good added value of the engine but it unfortunately cannot catch all the cases. The word can be concatenated in the query and split in the record, and in this case you have no other choice than splitting the query term in several words. You have this issue if the query contains the hash tag #searchengine and your record contains . The Algolia","record_index":5},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 engine will automatically try to split all your query terms in two words using the dictionary. In this case we know the frequency of the \"searchengine\" term and the frequency of all possible splits. We look at all the possible ways to split the words and select the one that has the best score for . In this case we take the best score between: , , \u00a0\u2026 , , \u2026 , . It is obvious that the maximum will be obtained for the split in  +  and we will extend the query term  with a phrase query between  and . (Phrase query means that the terms  need to be just after  in the record to match.)\n4) Transliteration: All of the approaches we have described before work well for alphabet-based languages but can\u2019t be applied on ideogram-based languages such as Chinese, Japanese and Korean. For these languages we use an extension file of the unicode standard called  that contains links between ideograms. For example those two lines contains a mapping between a simplified Chinese and traditional Chinese character:\nU+673A \u00a0kTraditionalVariant \u00a0\u00a0\u00a0\u00a0U+6A5F\nU+6A5F \u00a0kSimplifiedVariant \u00a0\u00a0\u00a0\u00a0\u00a0U+673A\nThis unicode file also contains a Z-variant of the same ideogram (share the same etymology but have slightly different appearances), for example:\nU+3588 kZVariant U+439B\nU+439B kZVariant U+3588\nThe unicode format goes even further by containing ideogram with overlapping meanings, for example:\nU+349A kSemanticVariant U+7A69<kFenn,kMatthews\nU+349A kSpecializedSemanticVariant U+6587<kFenn\nU+6587 kSpecializedSemanticVariant U+349A<kFenn U+7A69<kFenn\nU+7A69 kSemanticVariant U+349A<kMatthews\nThe engine will consider all those transliteration as one or two typo for ideogram-based languages. You will be able to search for simplified Chinese using a traditional Chinese query and vice versa!\n5) Lemmatisation: Typo tolerance is pretty efficient in tolerating most differences between singular and plural forms. There are of course exceptions like  or . But the most important thing is","record_index":6},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 that you probably don't want to consider those small differences as a typo. This is why we have packaged a dictionary that contains singular\/plurals form of words in 88 languages. This dictionary is called a lemmatizer and is exposed with the ignorePlural setting. You can simply add singular\/plural alternatives without having to care about building such a resource.\n6) Synonyms: The last type of alternative added in a query are synonyms. We have a pretty powerful synonym feature that handles all types of synonyms:\n\nAdd an alternative on a query term (the alternative can be a single word or a multi-word expression)\nAdd an alternative on a multi-term query expression. The alternative can be a multi-word expression and can contain any number of words\nAdd a placeholder token in your text that matches a list of words. You can, for example, add <streetnumber> in your record and configure it to match any numeric expression from 1 to 2000. Of course the record will not be found on the  query, but  will be found for the query .\n\nAll those synonyms can be handled with a correct highlighting ( will be correctly highlighted for the query , despite the typo) with a correct handling of proximity between expressions. This part is pretty complex and not correctly handled by most engines, we will explain how we handle it in a future post!\nWe do not package synonyms with the engine. The reason is that synonyms are tightly linked to a domain and most of the time you just need a few of them to increase a lot the quality of your search. For example, I saw the synonym  =  on several e-commerce website. This synonym makes sense as we see users typing  to search for t-shirt, but it can be dangerous if you have sports articles ( or ).\nThe main issue behind that is the polysemy of natural languages and the various specificities of each website. Our take is to handle automatically all the potential typos that few search engines handle automatically (like t-shirt = tshirt = t shirt) and","record_index":7},{"post_id":5067,"post_title":"Inside the Algolia Engine Part 3 \u2014 Query Processing","post_date":1464090822,"post_date_formatted":"May 24th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-3-query-processing\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/inside-part-3-360x200.png","time_to_read":16,"content":"\u2026 use synonyms as a domain specific tuning you can perform to refine it when needed.\nBuilding a good query processing takes time!\nWe have built all those query processing steps over the last four years and of course new ones will come! Our first users only had typo tolerance at the beginning but they have seen the benefits of each improvement! Better: it did not have\u00a0any negative impact on their relevance due to our way of handling ranking. We also managed to bring all those improvements without sacrificing performance! Before each added alternative, we have implemented a new optimization in the engine to counterbalance the cost of our alternatives!\nWe recommend to read the other posts of this series:\n\nPart 1 \u2013 indexing vs. search\nPart 2 \u2013 the indexing challenge of instant search\nPart 4 \u2013 Textual Relevance\nPart 5 \u2013 Highlighting, a Cornerstone of Search UX\nPart 6 \u2013 Handling Synonyms the Right Way\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":8},{"post_id":5031,"post_title":"How we re-invented our office space in Paris","post_date":1463731588,"post_date_formatted":"May 20th 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-re-invented-our-office-space-in-paris\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/office-360x200.png","time_to_read":3,"content":"We take a few things quite seriously at Algolia\u2014search, fried chicken, cookies and change for the better. In fact, at the rate our product and team are growing, we believe that Algolia is a new company every six months.\nBut when people in our Paris office started taking conference calls from the supplies room and kitchen, we realised that our space hadn\u2019t exactly grown with us and it was time to find a new home.\n\nAlgolia on the move\nLike everything else at Algolia, the big move was a collective effort from the get go. Our office manager and CTO spearheaded months of site visits and negotiations, sharing everything from proposed locations to paint colors with the team. After months of searching, we found a space in a vibrant neighborhood in the heart of Paris, just next to the lovely River Seine. At roughly 8600 square feet, the location would allow us to host meetings and workshops as well as accommodate our fast growing team. In fact, once we found out about the wide choice of restaurants to try out during lunch, there was no other address for us but 88 Rue de Rivoli.\nRe-inventing our space...at the watercooler\nThe build-up to moving day was a bit chaotic. \u00a0The primary discussion at the water-cooler was anything related to the office map - lighting, seating to even our highly anticipated, brand new Playstation! We even had a mini \u201cHunger Games\u201d situation at the office, with people laying claim to their desks and favorite spots beforehand.\nAt Algolia, we\u2019re not very particular or rigid about work spaces. People pick their own spots or organize themselves depending on where they\u2019re most comfortable. After much discussion we decided to design the new office keeping this in mind with adjustable standing desks, large \u201cFatboys\u201d and lots of meeting rooms. \nIn order to avoid a situation where people had to resort to taking calls from the kitchen, we also made sure there were lots of little call rooms for privacy. Of course we\u2019re thinking non-stop about","record_index":0},{"post_id":5031,"post_title":"How we re-invented our office space in Paris","post_date":1463731588,"post_date_formatted":"May 20th 2016","author_name":"Pragati Basu","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/021820ea84bbbab7d45e79973578e5ee?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-re-invented-our-office-space-in-paris\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/office-360x200.png","time_to_read":3,"content":"\u2026 newer ways to make our space more comfortable even today.\nWhen nostalgia set in\nOf\u00a0course there was the matter of a zillion questions which needed to be answered. Where were the lights going to go? How many new desks did we need? Who scheduled the office ski trip the same weekend as the move? There were a lot of questions that needed answering. But as the date drew nearer and we confirmed all the logistics, the nostalgia set in.\nThough we had been talking about the move for months, actually packing our things into boxes and getting ready to leave the old office behind wasn\u2019t easy. It was here that Algolia had tripled in size. It was here that Algolia had held its first Thanksgiving. It was here that we had our very first \u201cGarden\u201d\u2014where every member of the Algolia team sat together and had lunch each day. But now we had the opportunity to create new traditions and memories...starting with moving into the new office at 8am on a Saturday morning!\n\nNew traditions in the making...\nOne of the first things we did after moving to the new office\u2014after laying claim to our desks in no particular order (yes, the \u201cHunger Games\u201d had been a futile, but fun exercise), was map out all of the interesting cafes and restaurants we could go to for lunch\u2014after all, milliseconds do matter.\n\nIt\u2019s been just a few weeks in our new office and, to put it simply, we\u2019re loving it. Several team-mates have personal projects ranging from painting the walls to purchasing plants (over and above the 20 odd we already have) to even buying games for our Playstation which occupies pride of place in our new \u2018Garden\u2019.\nSome things of course, are better left unchanged.\n ","record_index":1},{"post_id":4856,"post_title":"Search your knowledge base quickly and easily with Algolia for Zendesk","post_date":1463391451,"post_date_formatted":"May 16th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-your-kb-quickly-easily-algolia-zendesk\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/Pasted-image-at-2016_07_21-15_58-360x200.png","time_to_read":4,"content":"We\u2019re excited to announce the release\u00a0of our new Zendesk integration as a part of our community package. This integration adds as-you-type search to\u00a0your help center in just a few clicks and can be integrated at no cost.\nWhy do you need a help center?\nTo quote Zendesk, you need a help center to:\n1. Increase customer satisfaction by providing better service and meeting the needs of customers who prefer self-service.\n2. Reduce costs and increase efficiency by eliminating repetitive costs so agents can focus on more strategic tasks.\n3. Grow your business community and build deeper connections between your company and customers.\n6 Tips for Building a Thriving Help Center by Zendesk\n\nWhat if we told you that with this release, we can help you with both 1 and 2 on this list?\nThe Algolia Zendesk usecase\nAt Algolia, the whole technical team handles support\u00a0because it gives us more insight into the frequently recurring questions and forces all of us to acknowledge our user's\u00a0pain points.\u00a0By sharing their pain, we end up doing\u00a0everything we can to help them find answers as quickly and easily as possible. Every\u00a0email not written to support because the\u00a0users were able to find\u00a0the answer on their own is a win, for\u00a0us and for them.\nThat\u2019s why our documentation and FAQ are both searchable. We even embedded them in our Need help? widget that's accessible from anywhere on the website.\nWhy is knowledge base search\u00a0important?\nWhen users visit your help center, they rarely want to browse it. Instead, they want to find answers to their question(s), as fast as possible. And a good search is an obvious way to help them achieve this. Zendesk\u2019s own help center has a huge search bar to invite you to search before opening a ticket. Unfortunately, with a \u201ctype-keywords-press-enter\u201d strategy, you often have to try multiple times before finding the answers you're looking for.\nThat\u2019s where Algolia comes into play. By providing as-you-type results, Algolia gives your users","record_index":0},{"post_id":4856,"post_title":"Search your knowledge base quickly and easily with Algolia for Zendesk","post_date":1463391451,"post_date_formatted":"May 16th 2016","author_name":"Matthieu Dumont","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/8103edb127eeac0b51813f96493b14ae?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/search-your-kb-quickly-easily-algolia-zendesk\/","categories":["Community"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/07\/Pasted-image-at-2016_07_21-15_58-360x200.png","time_to_read":4,"content":"\u2026 instant feedback on the keywords they\u2019re currently typing, allowing them to quickly and easily narrow their search\u00a0and find the information they need.\u00a0Reducing friction for your users not only makes them happier but also helps your agents by reducing the number of repetitive questions they have to answer.\nHow to get started\nThe Algolia Zendesk integration is easy to install and\u00a0only requires\u00a0a few clicks and one copy\/paste.\nThe integration automatically exports your Zendesk articles into your Algolia account and keeps them up-to-date. Not a single line of code needed!\nOn the front-end side, it adds:\nAutocomplete to every search field in your help center\n\nInstant capabilities to your search page\n\nBut don\u2019t just take our word for it,\u00a0try it live!\nAnd just in case you missed it the first time\u2014this integration is available for\u00a0free!\nGoing further\nThis integration is actually creating an Algolia index and keeping it up-to-date, which means you can easily add\u00a0an autocompletion menu using this data anywhere in your website\/app without having to manage the indexing.\nFinally, the whole code (crawler, JavaScript library and community website) is open-source under the MIT license. Your contributions are very welcome!\nRelated links:\n\nCommunity page\nDocumentation\nGitHub repository\n\nGet started","record_index":1},{"post_id":5003,"post_title":"Designing a developer friendly API client in Scala","post_date":1462968560,"post_date_formatted":"May 11th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/designing-a-developer-friendly-api-client-in-scala\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/Scala-blog-360x200.png","time_to_read":6,"content":"The Algolia Scala API client is our 14th API client and was released earlier this year. As with every API client we release, we made it as easy to use as possible and want to give you some insight into our building process.\nIs it developer friendly?\nHave you ever struggled to use an external library or even just find some simple getting started documentation? What about a clear enough API that you don\u2019t actually need to read the documentation?\nAs developers, we know how painful it can be to use external libraries, so we wanted to make the Algolia experience as easy as possible. I remember when I used Algolia for the first time; I used the ruby gem and loved the experience. It took me only 10 minutes to index and start searching some data, and this included a bundle install!\nEvery time we develop a new Algolia API client, our main goal is to finish with something that strongly leverages the language's strengths and, at the same time, is easy to use and feels productive from the very first minute.\nRTFM\nFirst, the documentation. It should be easy to read, but it also should be easy to write. As we support 10+ languages, we need the documentation to be clear enough that someone can just jump in, regardless of the language. For now, we use a custom project that generates Markdown files. It\u2019s quite simple\u2014a template, some code in each programming language, some ifs to handle special case. This way, every time we add a feature to our API, we don\u2019t need to completely rewrite the documentation. Furthermore, we use the custom project to generate the README.md on GitHub and the documentation on the website.\nSecond, the code. As with the documentation, it should be pretty straightforward for a developer to use. We want each API client to use the idioms\/syntax of its programming language. But, at the same time, we want to use the same vocabulary across each of our own API clients so a developer can move seamlessly from one to another. For example, we refer to a document","record_index":0},{"post_id":5003,"post_title":"Designing a developer friendly API client in Scala","post_date":1462968560,"post_date_formatted":"May 11th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/designing-a-developer-friendly-api-client-in-scala\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/Scala-blog-360x200.png","time_to_read":6,"content":"\u2026 as an object, so this term should be used in every API client.\nWhat principles are unique to Scala?\nScala is a very powerful language, and because of that developers are used to powerful libraries that can do a lot in just a few lines. Furthermore, with the non-blocking trend, we are used to asynchronous libraries. These two things were the core design principles we had in mind when we developed our Scala API client.\nA few lines of english\nOne great thing about Scala is its flexibility. For example, you can omit parentheses and dots when calling method, with some rules. For example:\n\ncan be written as\n\nIt can also be extended further.\n\ncan be written as\n\nWith this simple functionality, and with some clever choice of words, you can have a fluent API that looks like English sentences. As you might have guessed, we used this a lot to provide a really straightforward API:\n\nThere are some limitation to this. First, it only works with 1 parameter methods. Second, some keywords are reserved in Scala, so you can\u2019t use them as method names (for example: object).\nThe first limitation is not really a limitation as you can always find a way to get around it. If you need 2 or more parameter methods, you can find ways to express it differently in English. Another way is to use `object` for drop-in words. Let\u2019s take this code:\n\nIf you add the parentheses, it becomes:\n\nAs you can see, the methods `synonyms` and `and` take an `object` as parameters. The first one `of` is just for show, and the second one is a real parameter that will be used in the API.\nThe real issue is for 0 parameter methods. For example, to list all the indices of your account:\n\nIt\u2019s possible to write\n\nbut it\u2019s considered unsafe.\nThe second limitation can be worked around with the backtick notation:\n\ndoes not compile, but\n\ndoes.\nFor example, indexing an object:\n\nDon\u2019t bore me with simple stuff\nAnother thing that could help developers is taking care of repeating tasks for them. For example, we decided","record_index":1},{"post_id":5003,"post_title":"Designing a developer friendly API client in Scala","post_date":1462968560,"post_date_formatted":"May 11th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/designing-a-developer-friendly-api-client-in-scala\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/Scala-blog-360x200.png","time_to_read":6,"content":"\u2026 to map as many things to object\/classes as possible. The objects you index are `case class`, type safe and much easier to use. To have as much type safety as possible, a lot of parameters were transformed to case objects. This way you have the compiler working for you. For example:\n\n\nThat\u2019s not much, but it can save a lot of time in the case of a typo. Our search is typo tolerant, not our settings. \ud83d\ude09\nSecond things second: Asynchronous\nIt seems as simple as we should just return `Future[_]`. But there are two things to consider:\nFirst, users might want to specify their own `ExecutionContext`, so we added it as an implicit parameter:\n\nBut when we wanted to implement `browse`, we stumbled upon a limitation.\u00a0This method allows you to get all the objects of an index. It uses a cursor, as in a SQL database.\u00a0The first call returns a list of objects and a cursor, then you use this cursor to call the API again so you can get the next objects, and you repeat that until the cursor is empty.\nOf course the first call to `browse` in Scala would return a `Future[Iterable[_]]`. But we need to call it multiple times to have all the results. So our final method would have returned a `Iterable[Future[Iterable[_]]`. Not nice.\nFurthermore, each `Future` needs the result from the previous one to be able to run (remember the cursor thingy). And it would have been nicer to have, as a result type, something like a `Stream[_]`. It\u2019s achievable with an `Iteratee`, but it would have needed to add a dependency. As we want to keep our API clients as small as possible, we chose not to implement it in this client. For more on `Iteratee`, see here.\nWhat could we improve?\nOf course there are always improvements to be made. For example, we would love to simplify the way batches are made (see: https:\/\/github.com\/algolia\/algoliasearch-client-scala\/issues\/67).\nAs of today, if you want to have a batch, you need to call the method `batch` that takes a list of operations:\n\nSimple enough, but these","record_index":2},{"post_id":5003,"post_title":"Designing a developer friendly API client in Scala","post_date":1462968560,"post_date_formatted":"May 11th 2016","author_name":"R\u00e9my-Christophe Schermesser","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/6da5f25e177a84733a05b55defda9ed4?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/designing-a-developer-friendly-api-client-in-scala\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/Scala-blog-360x200.png","time_to_read":6,"content":"\u2026 pesky ending commas could be removed, and there is a lot of repetition (index into \"indexToBrowse\" `object`). It would be better to do something like this:\n\nThis feature is planned, but we are not sure if this new syntax is needed.\nLong story short, I tried to keep in mind the ease of use while developing the Scala API client. Hopefully you have some better insight into this process now. I would love to hear your feedback and any suggestions you have for improving it. Feel free to leave a comment or open a GitHub issue to contact me directly.","record_index":3},{"post_id":4814,"post_title":"Reducing precision in custom ranking values","post_date":1462460202,"post_date_formatted":"May 5th 2016","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-precision-custom-ranking-values\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/reducing-precision-360x200.png","time_to_read":4,"content":"Search relevance is always top of mind at Algolia. It is one of our differentiating factors, and we are always pushing ourselves to make sure our search engine is as relevant as possible. \nOne of the ways we ensure relevance is with our custom ranking feature, which is a very powerful tool if you know how to use it. One issue you may run into, however, is that custom ranking attributes that span a wide range of values may have too fine a granularity. Think for example of the number of views a photo might have. If you want to take multiple custom ranking attributes into consideration in order to get a good mix of results, you need to reduce the precision of this attribute or the other attributes may never be used.\nTo understand why, it's important to revisit Algolia's tie-breaking algorithm. Every time multiple records have the same score, we bucket them out based on the currently examined ranking factor and then create smaller\u00a0and smaller\u00a0buckets until we have exhausted our ranking factors.\nIf records have gotten through all of the textual relevance factors and are still tied, we take a look at custom ranking factors. Let's say that our custom ranking is set up like\u00a0this:\n1. Photo views (descending), with our most popular photos having millions of views and new photos having 0\n2. Number of likes (descending), with values ranging from 0 to thousands\n3. Date added (descending)\nSince we want the most popular photos to be displayed first, we will achieve this with our\u00a0first factor. But this will, in most cases, be the only factor considered because the values for this attribute are so precise. Think about this\u2014we have six videos tied in textual relevance with the following custom ranking attributes:\n\nIn this case, the photos\u00a0would now be ranked descending, based on number of views (9768, 5341, 1000, 1000, 25, 10). And since only two of them are tied in the same bucket (views equal to 1000), we only examine\u00a0the second custom ranking criteria for those two photos.","record_index":0},{"post_id":4814,"post_title":"Reducing precision in custom ranking values","post_date":1462460202,"post_date_formatted":"May 5th 2016","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-precision-custom-ranking-values\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/reducing-precision-360x200.png","time_to_read":4,"content":"\u2026 And, because the number of likes for those two photos is different, we never actually look\u00a0at the created date at all.\nWhy does this matter?\nIf you just want like count and created date to be in the custom ranking as tie breakers, it doesn't. But it matters a lot if you want your results to display a good mix of well-viewed photos, well-liked photos and new photos.\nBecause of the precision of the number of views attribute, you're not much better off in this case than if you had only\u00a0used this one attribute for your custom ranking.\nWhat do we do about it?\nQuite simply, we need to decrease the range of values by converting continuous\u00a0values into discrete\u00a0ones. We can do this in a few different ways, each with their benefits and drawbacks.\nTiering\nThe first way to do this is to create tiers of these values. What this means is that you take your values and separate them into deciles, quartiles, centiles or any other equal tier that you desire. From there, you send to Algolia the tier each record belongs to. So our record would then look this (with 10 being the highest tier):\n\nThis can be done in-memory or in common databases and is best done with values that don't change often.\nAnother easy way of creating tiers is to reduce the precision of the data itself in isolation of other values. For example, a date could be sent with values by day (20160119) or by hour (2016011922).\nCalculating the logarithm\nAnother\u00a0option is to take the log of the values, rounding down to the nearest integer. Whether it's a natural log, log10 or anything else doesn't matter much, which makes the calculation much simpler.\nThis also creates larger buckets at the high-end, which is valuable because there's a much larger difference between 10 views and 1000 views than there is between 1,000,010 views and 1,001,010 views.\nCustom scoring\nA final option is to create a custom score at indexing time. This isn\u2019t really a great option because you lose a lot of what makes Algolia so\u00a0powerful. We","record_index":1},{"post_id":4814,"post_title":"Reducing precision in custom ranking values","post_date":1462460202,"post_date_formatted":"May 5th 2016","author_name":"Dustin Coates","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/46aa7ea988a4c88c9a7cd846f49f63ec?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/reducing-precision-custom-ranking-values\/","categories":["Guides"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/reducing-precision-360x200.png","time_to_read":4,"content":"\u2026 will go into the pros and cons of this approach in an upcoming blog post.\nWhat's right for you?\nSo what\u2019s the right approach for your situation? It really depends on how often your data changes and how many pieces of data there are. With data that changes very often or with a large set of records, a logarithm might make more sense. For records where values are clumped closely together, perhaps a tiering system would work best. In general, we go with the logarithmic system, but give both a try and see what works best for you!","record_index":2},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"Search engine usage has evolved a lot in the last few years.\u00a0Today\u2019s users expect instant feedback without having to click on the \u201center\u201d button and wait for the page to load to get their results. They also expect the engine to tolerate typing mistakes on the fly without having to validate proposals of query re-phrasing, like the (in)famous \u201cdid you mean?\u201d\nOf course all these changes have pretty big consequences on the engine itself. In this post, we will concentrate on the way we handle indexing at Algolia, why we do it differently in order to maximize indexing speed and search performance and how we natively address these evolutions in search.\nIntroduction to indexing\nThe goal of the indexing process is to store your data in a specific data-structure that is optimized for search. Without indexing, we would need to scan all the documents in a database to detect which ones match a query. This would take forever and be highly inefficient, ultimately resulting in a terrible user experience. This is one of the reasons we continue to focus on optimizing indexing speed as part of our overall efforts to make search more efficient.\nThe main part of the indexing process is to create a data-structure that contains a mapping of each word to the associated list of documents containing this word. One mapping of the word to the list of documents is called an inverted list, with the name coming \u00a0from the inversion of the space: instead of having documents that contains words, we have now words with the list of documents containing each word. The concept is similar to the index you have at the end of a book, except all words are in the index.\nThere are a lot of algorithms to store those inverted lists in an efficient way while keeping good performance. Most of those techniques are pretty old and well described in the very good Managing Gigabytes book.\nThe creation of those inverted lists can be decomposed in two big parts:\n\nFor each document, we extract the list of","record_index":0},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 words and build a hash-table that associates words to documents\nWhen all documents are processed, we compute an on-disk binary data-structure containing the mapping of words to documents. This data-structure is the index we will use to process queries.\n\nHow search-as-you-type is usually implemented\nAs-you-type display of search results was introduced by Google in 2010 as \"Google Instant\" and later referred to as \"Instant Search.\" Users love this experience as it creates a seamless interaction with the data they are looking for! It removes the intermediate steps such as query validation, \u201cdid-you-mean\u201d re-phrasing and decreases the time spent waiting for the results to show up! It also means indexing and search have to be redesigned because you don't simply search for words anymore, but also for partially written words (prefixes) and it needs to be very fast!\nThere are four main different approaches engineers use today to build this type of instant search:\n1) Autocomplete (also called Suggest)\nThis approach is the most commonly used and is based on a static data-structure on the side of the search engine. You can see it as a big dictionary in the form of a Trie (special type of Tree that associates string to an output, all the descendants of a node have a common prefix of the string).\nAdvantages:\n\nVery fast response time (usually few milliseconds)\nScales well, can easily handle thousands of queries per second on one machine\n\nDrawbacks:\n\nThis is not a search engine, you have to search for a sub-string on the pre-defined list of expressions in the dictionary. In terms of user-experience, it often leads to very frustrating \"no results\" or \"bad results\" because of this limitation\nThe relevance is not the same as the search engine itself, it is just a static dictionary. It works well if you propose \"frequent queries,\" but starts to have weird effects if you use it to search on the same records as the fully-featured search engine itself. The relevance will be","record_index":1},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 different, of lesser quality and will lead to a lot of confusion.\n\n2) Index prefixes\nThis approach is simply to build an inverted list for all prefixes of a word instead of just the word itself. For example instead of just having an inverted list for the word \"engine\", you will also have one for \"e\", \"en\", \"eng\", \"engi\" and \"engin\". Those lists will usually contain a specific indicator that allows at query time to make the difference between a prefix and an exact word (both exact and prefix inverted lists will be queried with a preference for exact).\nAdvantages:\n\nFast resolution of prefixes as they are pre-computed\nAbility to keep the relevance of the search-engine for all interaction with the engine, including as-you-type experience (user is never lost between two different ways to rank results)\n\nDrawbacks:\n\nIncrease the time to publish results because the indexing process is far more expensive than without inverted list and consumes far more memory (inverted list are kept in memory)\nIncrease a lot the size of the index, which reduces a lot the likelihood of fitting in memory and so can hurt performance (even on a SSD)\n\n3) ngrams computation\nThis approach is very similar to the indexing of all prefixes but it limits the computation of prefixes by allowing a minimum and a maximum number of letters in the prefix. The main advantage is to reduce the cost compared to the indexing of all prefixes, if you query a prefix that is bigger than the maximum number of letters, the query will be automatically be transformed into a query on several ngrams (queries with fewer letters than the minimum will not trigger the prefix search).\nAdvantages and drawbacks are the same as the indexing of all prefixes; it is just a small improvement to reduce a bit the cost of indexing by adding more computation in the search.\n4) Search prefixes at search time\nThis approach keeps the indexing process pretty light by keeping only one inverted list per word but it requires to do all the","record_index":2},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 processing at query-time. For example if you perform the query 'a', it will mean looking for all words that start by 'a' in a big dictionary and then perform a query with all those words in a big OR operand. This approach is very expensive and usually people that select this approach start the prefix search after a certain number of letters in the prefix to avoid having queries that are too\u00a0expensive.\nAdvantages:\n\nNo indexing overhead for prefix searches\nNo compromises on\u00a0the relevance of the search-engine\n\nDrawbacks:\n\nProduces very complex and slow queries\nOnly works if the number of documents and words is very small, it does not scale\n\nEngineers working on search have no other choice than spending time to test those different approaches and select the one that seems the best for their use cases, unfortunately all of them have serious drawbacks.\nA different approach\nAt the beginning of Algolia, we worked on a different product: an offline search engine SDK for mobile app developers. And we wanted this engine to provide a fast instant search experience on old devices like an iPhone 3G.\nNone of the standard approaches described above would have worked on such a low-end hardware, so we had to design a new way to do the indexing with very little RAM and CPU. All that while maintaining an instant search experience (one of the purpose of indexing being to speed up the search operations).\nHaving already worked on multiple data-structures for search engines prior to Algolia helped me a lot in designing a different way to solve the problem. The first inspiration came from a generational string B-tree I built several years ago. It was pretty similar to what the LevelDB fast key-value store developed by Google is doing (associate an arbitrary size value to an arbitrary size string).\nFirst, I knew that for Algolia I wanted to have a generational data-structure.\nGenerational data-structure\nHaving a generational data-structure makes updating the data a lot faster. Typically,","record_index":3},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 you\u2019ll create two (or more) \u201cgenerations\u201d of your data-structure. A big one containing the \u201cold\u201d data, and a small one containing the \u201cnew\u201d data.\nThen, if you need to add\/update\/delete an object, you\u2019ll update a smaller structure containing only the \u201cnew\u201d information, which is much faster.\nMerge of Generational data-structure\nAt some point (generally when your smallest generation reaches a certain size), you\u2019ll need to merge several generations into one for a better efficiency. This operation is called a merge.\nIn order to have an efficient merge, each generation needs to be sorted, so that you can scan them in parallel while generating the result (merge of the parallel scan is done via a standard heap-merge algorithm).\nRadix tree\nFor Algolia, I decided to represent the words that will be searchable as a radix tree (space-optimized representation of a Trie, below we look at what a radix tree is in practice).\nFirst because a radix tree is sorted, which allowed us to have an efficient merge and to use multiple generations of the tree.\nSecond because it works very well to process the typo-tolerance in a very efficient way (more details on the typo-tolerance will come in another post).\nIt is easy to build a radix tree in memory before writing it on disk, but it consumes a lot of RAM. And I wanted to use the least amount of RAM possible, because:\n\nFor the initial offline search SDK, we simply didn\u2019t have enough RAM on an iPhone 3G.\nFor our current SaaS engine, we want the RAM to be focused on making search fast (the indices are all stored on RAM for search), which means indexing should consume the minimum amount of RAM.\n\nWhat\u2019s great is that you can actually write a radix tree on the fly without having to build the tree in memory if you\u2019re indexing a list of sorted words.\nFor example, let\u2019s consider that the dataset we want to search on contains 5 objects, each having a single word. Each object will be represented by a value from 1 to","record_index":4},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 5.\nNormally, multiple objects could have the same word, so the leaf would have a list of values. But for simplicity\u2019s sake, we\u2019ll take records containing different words.\nDictionary example mapping a word to an integer\nEach node of our radix tree will contain a string.\nIf the node is a leaf, it has a value associated to the corresponding object having this word.\nIf the node is not a leaf, it can have an associated value, but it can also just be an internal node representing a shared prefix.\nThe radix-tree we\u2019re going to build will look like this:\nRadix-tree of the previous dictionary\nNow, let\u2019s build our radix-tree. Seems simple now enough, right? The naive approach is to build the complete data-structure in memory and then to dump it on disk. We do not want to use this approach because we want to keep as much memory as possible on our servers to hosts the indices of our users and provide a fast search! This is why we designed all the indexing to use as little memory as possible while optimizing speed of indexing.\nHere\u2019s the trick: we can build this data-structure on-disk directly from the list of words and with only a few kilobytes of memory. To do that, we\u2019ll flush from the RAM the nodes as soon as possible, by building the tree on the fly. \nThe first step is to order alphanumerically the words of the documents we want to index (which is already done in our example).\nThen, we\u2019re going to build our tree using a depth-first search: since a radix tree is sorted in the alphanumeric order, it\u2019ll be easy to flush nodes to the disk as soon as we don\u2019t need them.\n\n1. Add the word Fast\n\n2. Add the word Faster\n\n3. Add the word Test\n\n4. Flush the branch \u201cfast\u201d to disk\n\n5. Add the word toaster\n\n6. Flush the branch \u201cest\u201d\n\n\n7. Add the word toasting\n\n8. Flush the branch \"er\", then \"ing\", \"oast\" and finally \"t\"\n\nThis way to build a tree is very efficient because it only keeps in memory a very small number of nodes (in the worst case scenario you will","record_index":5},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 have a number of nodes in memory that is equal to the longest key in the tree, in our case we always have fewer than 100 nodes in memory).Now that this is done, let\u2019s focus on the next part: prefix search!\n\nPrefix search\nRemember that we wanted an instant search experience. This means that the engine must be able to retrieve results as-you-type, at each character.\nIf you type \u201cf\u201d, you want to retrieve the results \u201cfast\u201d and \u201cfaster\u201d, so the engine must know that both these words contain the prefix \u201cf\u201d.\nAnd we don\u2019t want to calculate the association prefix-object for all the prefixes, which would take a lot of time and memory, but only when it\u2019s really necessary.\nThe good news is that our radix tree can tell us precisely where we need to store prefixes: all the nodes in the tree which have children.\nIn our example, it is only necessary for the nodes \u201cfast\u201d, \u201ct\u201d and \u201ctoast\u201d.\nRadix-tree with prefix inverted list computed\nYou see what I just did right there? Yes! the prefixes can be computed on the fly too! We just need to remember what was in the last few branches that we flushed.\nFor example, if I just flushed the branch \u201cest\u201d, I\u2019ll remember that the prefix \u201ct\u201d is associated to the object 3 (test). I\u2019ll do the same with \u201ctoaster\u201d (object 4) and \u201ctoasting\u201d (object 5). And when we\u2019re done processing all the words beginning with \u201ct\u201d, I\u2019ll know that this node should contain the objects 3, 4 and 5.\nA few numbers\nThis way to build prefixes has been\u00a0built into our engine since 2012 and we index billions of jobs every day with this algorithm. We have measured the impact on a lot of different indices, the impact on search is pretty obvious as we always have the prefix inverted list when we do a query (and we\u2019ll have a lot of RAM handy), but the impact on indexing time and on the size are less obvious. Here is a small summary of the impacts that are very small compared to the advantages it gives at query","record_index":6},{"post_id":4934,"post_title":"Inside the Algolia Engine Part 2 \u2014 The Indexing Challenge of Instant Search","post_date":1462155441,"post_date_formatted":"May 2nd 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-2-the-indexing-challenge-of-instant-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/05\/indexing-part2-360x200.png","time_to_read":15,"content":"\u2026 time:\n\n\n\n\nMinimum\nAverage\nMaximum\n\n\nImpact of prefixes on index size\n+9%\n+21%\n+43%\n\n\nImpact of prefixes on indexing speed\n+3%\n+12%\n+17%\n\n\n\nThis approach is significantly better than any of the other approaches presented before, it is actually optimal at query time and has a very reasonable impact on indexing performance!\nThis optimization of the prefixed indexing is just one of the numerous optimizations we have built in the Algolia engine. The reason we built it is because of our focus on performance and the fact that our engine was developed specifically to allow for the\u00a0strong constraints of mobiles devices. We also had to think a bit out of the box, mainly by considering the word dictionary and inverted lists as one unique data-structure instead of two independent one.\nI look forward to reading\u00a0your thoughts and comments on this post and continuing to explain how our engine is implemented internally, transparency is part of our DNA! \ud83d\ude42\nWe recommend to read the other posts of this series:\n\nPart 1 \u2013 indexing vs. search\nPart 3 \u2013 query processing\nPart 4 \u2013 Textual Relevance\nPart 5 \u2013 Highlighting, a Cornerstone of Search UX\nPart 6 \u2013 Handling Synonyms the Right Way\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":7},{"post_id":4924,"post_title":"Listen up! Algolia co-founders on the podcast circuit","post_date":1461929488,"post_date_formatted":"Apr 29th 2016","author_name":"Laura Evans","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/81158ee39a22e9a155ae5380da29cd20?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/listen-up-algolia-co-founders-on-the-podcast-circuit\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/04\/podcast-360x200.png","time_to_read":2,"content":"April has been a busy month for the co-founders of Algolia. Nicolas had the opportunity to appear on the 20 Minute VC Podcast and the official SaaStr podcast, and Julien was featured on the Software Engineering Daily podcast.\nNicolas first sat down with host Harry Stebbings for an episode of the 20 Minute VC to chat about Algolia and what it takes to build a successful company while maintaining a strong core culture. He even shares his favorite blog, perhaps a bit predictable, and his favorite book, a little bit less so. \nHe then continued the conversation with Harry on the official SaaStr podcast to discuss what he and Algolia took away from the YC process and some advice for being a successful SaaS company, among several other things.\nMeanwhile, Julien and Jeff Meyerson from Software Engineering daily discussed the unsolved problems in search and building Algolia for the long run. Head on over to SE Daily to hear Julien. While you\u2019re there, check out some of the other content we\u2019ve sponsored, including Bootstrapping a SaaS for Developers with Itai Lahan and Developer Analytics with Calvin French-Owen.\nMany thanks to Harry and Jeff for inviting us to participate in their podcasts. We thoroughly enjoyed the experience!\nLet us know if you have any questions after listening to the podcasts, and definitely give us a shout to share some of your favorite podcasts in the SaaS\/software spaces.","record_index":0},{"post_id":4893,"post_title":"Welcoming Kendall Collins to the Algolia starting lineup","post_date":1461764931,"post_date_formatted":"Apr 27th 2016","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/welcoming-kendall-collins-to-the-algolia-starting-lineup\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/04\/Group-17-360x200.png","time_to_read":2,"content":"We\u2019re excited today to share some news that is very dear to us\u2014Kendall Collins has joined our board of directors!\nWe met Kendall at the end of last year and have been utterly impressed by his understanding of SaaS and his passion for search. We had the chance to build an advisory relationship with him over the last few months and are thrilled to announce that he\u2019s joining Algolia as a full member of our board.\nKendall spent the last 12 years at Salesforce as their global CMO and more recently as the CEO of Salesforce Cloud. In his last role, he was responsible for a number of core product functions, including search and Salesforce\u2019s internal open source implementation. The breadth of his experience at the leading cloud computing company is going to be very valuable to us as we continue to scale\u00a0Algolia at a fast pace. He also recently joined AppDynamics as their CMO and is actually working only two blocks away from our US headquarters!\nI've had the opportunity to chat with Kendall several times over the past few weeks, and he wanted to share a few of his reasons for joining us on our journey:\n\u201cGoogle continues to raise the bar for delightful search for end users, and most companies and apps are failing to keep pace with that user experience. When I first tried Algolia, it was clear to me that their Search Platform and API approach is the future for easily building incredibly fast and relevant search inside any application.\u201d\n\u201cAlgolia has already proven to be trusted by enterprises and loved by developers. Anyone can try it now on Product Hunt, Hacker News, or leading sites and apps such as Periscope, Arcteryx, Medium and Vevo. They have just successfully deployed at one of the top 5 software companies in the world and also at one of the top 5 retailers. And this is just the beginning. It is an honor to work with a team that has such a strong technical vision and an equally caring and compassionate culture.\u201d\nWith now more than 1,400 paying customers,","record_index":0},{"post_id":4893,"post_title":"Welcoming Kendall Collins to the Algolia starting lineup","post_date":1461764931,"post_date_formatted":"Apr 27th 2016","author_name":"Nicolas Dessaigne","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/785489bc2ac2e08ae66648a8936c1101?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/welcoming-kendall-collins-to-the-algolia-starting-lineup\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/04\/Group-17-360x200.png","time_to_read":2,"content":"\u2026 including many enterprise accounts, Algolia is changing the way people interact with content and leading the transformation of the search market to SaaS. Having Kendall join us on that mission is a big boost in our ability to make it happen even faster.\nWelcome to the team, Kendall!","record_index":1},{"post_id":4876,"post_title":"Inside the Algolia Engine Part 1 \u2014 Indexing vs. Search","post_date":1459271652,"post_date_formatted":"Mar 29th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-1-indexing-vs-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/Inside-the-engine-part-1-360x200.png","time_to_read":7,"content":"In previous blog posts, we have discussed the high-level architecture of our search engine and our worldwide distributed infrastructure. Now we would like to dive a little deeper into the Algolia search engine to explain why we implemented it from scratch instead of building upon an existing open-source engine. \nWe have many different reasons for doing so and want to provide ample context for each, so we have split \u201cInside the Algolia engine\u201d into several posts. As you learn more about our search engine, please let us know if there\u2019s anything you would like us to address in future posts.\nIf you have ever worked on a search engine with significant traffic and indexing, you are undoubtedly familiar with the problem of trying to fine-tune your indexing to avoid negatively affecting search performance. Part one of this series will focus on one of the quintessential problems with search engines\u2014the impact of indexing on search queries\u2014and our approach to solving it.\nWhy does indexing impact search performance?\nIndexing impacts search performance because indexing and search share two critical resources\u2014CPU and Disk. More specifically:\n1. Both indexing and search are very CPU intensive and compete for available resources. Imagine if you experience a sudden spike in search queries while simultaneously needing to run a large number of indexing operations. You run a significant risk of not having enough CPU to handle both.\n2.\u00a0Both indexing and search perform a lot of disk I\/Os. Search often performs a large number of read operations on the disk because the data is not always stored in memory, and indexing performs a large number of both read and write operations to the disk. There is also a battle for disk resources, even on high-end SSD drives.\nThe obvious way to solve this problem is to try to reduce or remove the conflicts of access to the shared resources.\nClassical approaches to solving this issue \nThere are a lot of different approaches to dealing with this","record_index":0},{"post_id":4876,"post_title":"Inside the Algolia Engine Part 1 \u2014 Indexing vs. Search","post_date":1459271652,"post_date_formatted":"Mar 29th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-1-indexing-vs-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/Inside-the-engine-part-1-360x200.png","time_to_read":7,"content":"\u2026 issue, and the majority fall into one of the following three categories:\n1. Update the data in batches during a regularly scheduled, controlled period of low traffic. This approach works well when your users are located in a specific country but doesn\u2019t really work if your users are distributed worldwide. More importantly, this approach prevents frequent updating of the index, which means that searches will often be performed on outdated data.\n2. Use different machines for indexing and search. In this approach, a specific set of machines is used for indexing, and the generated files are copied onto the search machines. This is pretty complex to set up but has the added benefit of removing the indexing CPU load from the search machine. Unfortunately, this does not solve the problem of shared I\/O, and your indexing will be bound by the network, especially when a compaction is triggered (an operation performed on a regular basis by the search engine to aggregate all incremental updates, also called optimization). The main drawback to this approach is the substantial impact on indexing speed as a sizable delay is introduced between when an update is made and when that update is actually available to be searched.\n3. Throttle the indexing speed via a priority queue. The goal of this approach is to limit the number of indexing operations in order to minimize the impact indexing has on search performance. Throttling introduces a delay in indexing speed that is difficult to measure, especially when coupled with a priority. The compaction described in the previous paragraph can also worsen the delay by causing a cumulative slow down effect on the indexing. This approach slows down indexing while also making it very difficult to avoid impacting search performance, especially during the compaction phases.\nWhile complex to implement, the second approach of using different machines for indexing and search is a good solution if indexing performance is not crucial to you. The","record_index":1},{"post_id":4876,"post_title":"Inside the Algolia Engine Part 1 \u2014 Indexing vs. Search","post_date":1459271652,"post_date_formatted":"Mar 29th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-1-indexing-vs-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/Inside-the-engine-part-1-360x200.png","time_to_read":7,"content":"\u2026 other two approaches only partially solve the issue as search remains impacted. Realistically, none of these approaches appropriately solves the problem of indexing affecting search performance because either indexing performance, search performance or both end up suffering.\nHow we solved the race for CPU resources\nBy splitting the indexing and search into different application processes! \nAt Algolia, indexing and search are divided into two different application processes with different scheduling priorities. Indexing has a lower CPU priority than search based on a higher nice level (Nice is a tool for modifying CPU priority on Unix-like operating systems). If there is not enough CPU to serve both indexing and search, priority is given to search queries and indexing is slowed down. You can keep your hardware architecture designed to handle both by simply slowing down indexing in the case of a big spike in search queries.\nAs is the case with using different machines for indexing and search, separating them into different application processes introduces some complexity; for example, the publication of new data for search becomes a multi-process commit.\nThis problem is pretty common and can easily be solved with the following sequence:\n\n1. When the indexing process receives new jobs, build a new incremental update of the index.\n2.\u00a0Commit the incremental update to the disk.\n3. Notify all search processes of the data changes to the disk.\n4. Redirect new queries to the new data-structure. Currently processing queries remain served by the old data-structure.\n5. When no more queries are processing on the old data-structure, erase it from the disk.\n\nThis approach solves the problem of needing to share and prioritize CPU resources between indexing and search but is unfortunately something that most search engines on the market today cannot implement because indexing and search are executed in the same process.\nHow we solved the race for disk resources\nThe race for disk","record_index":2},{"post_id":4876,"post_title":"Inside the Algolia Engine Part 1 \u2014 Indexing vs. Search","post_date":1459271652,"post_date_formatted":"Mar 29th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-1-indexing-vs-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/Inside-the-engine-part-1-360x200.png","time_to_read":7,"content":"\u2026 resources is a bit more complex to solve. First, we configured our kernel I\/O scheduler to assign different priorities to read and write operations via the custom expiration timeout settings within the Linux deadline scheduler. (Read operations expire after 100ms, write operations expire after 10s). Those settings gave us a nudge in the right direction, but this is still far from perfect because the indexing process performs a lot of read operations.\nThe best way to address the contention for finite disk resources is to make sure the search process does not perform any disk operations, which means that all the data needs to remain in memory. This may seem obvious, but it is the only way to ensure the speed of your search engine is not impacted by indexing operations. It may also seem a bit crazy in terms of costs (having to buy additional memory), but the allocated memory can actually handle the vast majority of use cases without issue. We of course have some users that want to optimize costs for huge amounts of data, but this makes up a very small percentage of our users (less than 1%) and is addressed on an individual basis.\nDefault to fast and reliable\nEverything at Algolia is designed with speed and reliability in mind\u2014your data is stored in memory and synced on a high-end SSD and at least three different servers for high availability. Our ultimate goal is to remove all of the pains associated with building a great search feature, and solving the dependency between indexing and search was a very important step in getting there!\nWe take a lot of pride in building the best possible product for our customers and hope this post gives you some insight into the inner workings of our engine and how we got where we are today. As always, we would love your feedback. Definitely leave us a comment if you have any questions or ideas for the next blog in the series.\nWe recommend to read the other posts of this series:\n\nPart 2 \u2013 the indexing challenge of instant","record_index":3},{"post_id":4876,"post_title":"Inside the Algolia Engine Part 1 \u2014 Indexing vs. Search","post_date":1459271652,"post_date_formatted":"Mar 29th 2016","author_name":"Julien Lemoine","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/25760a5d4e793e491f26da5db64bb738?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/inside-the-algolia-engine-part-1-indexing-vs-search\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/Inside-the-engine-part-1-360x200.png","time_to_read":7,"content":"\u2026 search\nPart 3 \u2013 query processing\nPart 4 \u2013 Textual Relevance\nPart 5 \u2013 Highlighting, a Cornerstone of Search UX\nPart 6 \u2013 Handling Synonyms the Right Way\nPart 7 \u2013 Better relevance via dedup at query time\nPart 8\u00a0\u2013\u00a0Handling Advanced Search Use Cases","record_index":4},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"When we started Algolia, we wanted to build an offline search SDK for mobile applications. Unlike other search engines on the market, we couldn\u2019t rely on pre-existing search engine software such as Lucene because they weren\u2019t available on mobile.\nWe had to build everything from the ground up within the limitations imposed by smartphones such as low memory and low processing power. Ultimately, we spent a considerable amount of time reinventing the search engine from scratch.\nWhen we eventually decided to pivot to a SaaS solution, we had built a completely new ranking algorithm. And in a typical \u201cconstraints foster creativity\u201d situation, we had made two huge improvements: Speed and Relevance.\nAlgolia is known for its blazing-fast speed. It\u2019s usually the first thing people notice when they start using it and also what makes it so exhilarating to use. But what I consider to be the most impressive and important achievement of Algolia is not actually speed but Relevance.\nAlgolia reinvented how search engines define relevance.\nBefore diving into the specifics behind Algolia\u2019s ranking formula, let\u2019s briefly go over how other search engines work.\nA little history of search engines\nIn the late 90s, the Text Retrieval Conference organized a series of annual workshops financed in part by the US Department of Defense. The goal of these workshops was to determine the best ranking algorithm for searching a dataset consisting of long unstructured documents. The conference resulted in several enhancements to search engine ranking algorithms, the most defining being improvements to TF-IDF, what was then the main component of the most powerful ranking formulas.\nTF-IDF is a statistical weighting scheme based on two elements: TF or Term Frequency (the number of occurrences of a word in a document, which is calculated on a per document basis) and IDF or Inverse Document Frequency (the importance of a query word within the whole corpus of documents).\nBasically, if a query","record_index":0},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"\u2026 word is repeated multiple times within a document, that document will rank higher in the results than a document that only contains the query word once. IDF comes into play to reduce the importance of words that occur a lot in the corpus such as \u201cthe\u201d, \u201cof\u201d, \u201ca\u201d, etc. The more times a query word recurs within the whole corpus, the lower a document containing that query word will be ranked. The length of the document is also taken into account; a short document with query word matches will rank higher than a long document with the same match.\nTF-IDF (along with similar algorithms like BM25) became a central tool to score\/rank documents given a user query, and it was used to search any type of data. Unfortunately, it was not actually designed for that! TD-IDF was built as an answer to TREC\u2019s workshops and was designed specifically to work within long unstructured documents such as PDFs or enterprise documents.\nThe issue with this kind of statistical approach is that it surprisingly only makes sense for a very small number of use-cases, like enterprise document search, where:\n\nYou don\u2019t really know what type of data you\u2019re searching into\nThe content is drastically different from one document to another\nThe documents don\u2019t have any popularity metrics, and you need to rank them solely based on their \u201ctextual relevance\u201d\n\nWhat Algolia was built for\nAlgolia wasn\u2019t built to search long unstructured documents. It was built to power the search of websites and mobile applications. And in 99% of cases, this data is very different.\n\nIt has a defined schema with meaningful attributes that you know beforehand (title, name, list of tags, description\u2026)\nYou know the underlying content, what matters and what doesn\u2019t (e.g. is the title more important than the description?)\nThere are often one or more popularity metrics associated with each object (number of sales\/likes\/views)\nAs the owner, you sometimes have a strategy that impacts the ranking (give a bonus","record_index":1},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"\u2026 to featured objects)\n\nLet\u2019s look at an example\nSay that you\u2019re on IMDB searching for an actor, and you type \u201cBrad\u201d. You don\u2019t care about the Term Frequency (how many times the document contains the word Brad), or the Inverse Document Frequency (how many times the word Brad is present in the list of all actors).\nFortunately, there are other signals that matter:\n\nIs the word Brad in the \u201cname\u201d attribute or in the description?\nIs it the first word or the last word in this attribute?\nIs it spelled exactly B-r-a-d? Does it contain a typo? A suffix?\nHow many followers\/likes\/views does Brad have?\n\nAs you can guess, these questions give us a much better chance at finding Brad Pitt than a statistical approach like TF-IDF.\nThis is the first pillar of Algolia\u2019s revolutionary improvements\u2014the rules taken into account in the Ranking algorithm.\nThe rules in Algolia\u2019s ranking formula\nAlgolia doesn\u2019t rely on any variation of TF-IDF. Instead, it uses six default rules to evaluate the textual relevance of an object for a specific query:\n\n1. Typo: Will rank higher a word that doesn\u2019t contain a typing mistake.\n2. Words: Will rank higher an object matching more of the words typed by the user if the query contains more than one word.\n3. Proximity: Will rank higher words close to each other (George Clooney is better than George word Clooney).\n4. Attribute: Will rank higher a match in a more important attribute (Title, Description).\n5. Words position: Will rank higher words that are at the beginning of an attribute.\n6. Exact: Will rank higher words that match exactly without any suffix (Algolia natively handles prefix search).\n\nMost search engines have somewhat similar implementations of the Words and Attribute rules (some also have an implementation of Proximity). The rest of these rules are unique to Algolia. You can of course disable or customize the rules to fit your specific use-case, and you\u00a0can also use non-textual criteria such as geo-location.\nThese rules","record_index":2},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"\u2026 are a great improvement over what existed before, but the biggest and most important change Algolia brought to the table is something else\u200a\u2014\u200ahow these rules are used to rank the objects.\nHow is the ranking done?\nThe biggest issue with the relevance of most search engines is not TF-IDF (which can actually be disabled in most search engines using it). It\u2019s the way the ranking criteria are used. Other search engines compute a big floating-point score that mixes everything. They have a very complex mathematical formula with a lot of coefficients\/boosts that merges every rule into one score for each document. Then, they use this unique score to rank the search results.\nThe first problem with this approach is that it mixes apples and oranges. For example, if your ranking formula takes into account TF-IDF and the number of likes of the documents:\n\nWhat happens if a document has a very high number of likes? It will override all the other criteria like textual relevance, and this document will always show up in the results, even when the textual relevance score is very bad.\nSimilarly, if an object is the most relevant to the query but doesn\u2019t have any likes, it won\u2019t show up in the results.\nThe second issue with this approach is its complexity. Nobody is able to understand what\u2019s really happening. Even the most skilled search experts need to spend a lot of time crafting the best mathematical formula. Via a trial-and-error process, they\u2019ll find a formula that works well for a list of the 100 most popular queries, but the formula cannot be optimized for the whole catalogue. And as soon as the catalogue (or any metrics used in the formula) changes, they need to go back to the drawing board.\n\nThis exception-driven approach is not maintainable.\nAlgolia\u2019s ranking formula is designed to fix this. And it does so with a different ranking approach via the Tie-Breaking algorithm.\nThe tie-breaking algorithm\nInstead of calculating one big score and then sorting the","record_index":3},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"\u2026 results once, a tie-breaking algorithm calculates N scores and applies N successive sorting. Here\u2019s how it works:\n\nAll the matching records are sorted according to the first criterion.\nIf any records are tied, those records are then sorted according the second criterion.\nIf there are still records that are tied, those are then sorted according to the third criterion.\nAnd so on, until each record in the search results has a distinct position.\n\nThe order of the criteria\u00a0determines their importance. It\u2019s like in a children\u2019s game\u2014if you sort by color and then by shape, you don\u2019t get the same results as\u00a0when you sort by shape then color.\nConfiguring the relevance of Algolia for a specific use-case is as simple as figuring out the set of rules that need to be applied and ordering them by importance.\nOh, and everything is customizable\u2014from the order of the attributes to which attributes are used. So there\u2019s virtually no ranking strategy that you can\u2019t build with this ranking formula.\nLet\u2019s consider an example with the following dataset:\n\nAnd this index configuration:\n\nTo simplify our example, let\u2019s ignore some of the rules and focus only on Typo, Featured and Number of Likes. For the query \u201cJohn,\u201d we\u2019ll get the following results:\n\n1. John Thompson (0 typos; featured; 8 likes)\n2. John Jackson (0 typos; not-featured; 17 likes)\n3. John Paul (0 typos; not-featured; 3 likes)\n4. Jon Black (1 typo; featured; 4 likes)\n5. Jon White (1 typo; not-featured; 9 likes)\n\nIn this example, we:\n\nDecided that Typo was the most important rule. If an object has a typo, it should be ranked lower, no matter what the other rules are.\nGave a bonus to records marked \u201cfeatured\u201d.\nDecided to position our main popularity metric, Number of likes, at the bottom of the formula so it doesn\u2019t override the textual relevance criteria.\n\nA few things to note:\n\nWe didn\u2019t need to use coefficients or boosts of any kind. Nor did we need to craft some complex mathematical formula.","record_index":4},{"post_id":4736,"post_title":"How Algolia tackled the relevance problem of search engines","post_date":1457617462,"post_date_formatted":"Mar 10th 2016","author_name":"Nicolas Baissas","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/dfd620dc267d3f8182cafd9837593294?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-tackled-the-relevance-problem-of-search-engines\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/03\/algorythm-360x200.png","time_to_read":9,"content":"\u2026 But we still managed to take into account business metrics and merge them with our textual relevance.\nThis formula works for every query. It is not designed to work for the top 100 most popular searches, it is designed to work for the whole catalogue.\nWhat happens if 50% of the catalogue changes tomorrow? Nothing. We don\u2019t need to change the formula as long as the structure of the data remains the same.\nIt is very simple to understand why one result is ranked higher than another. And if you can understand the results, fine-tuning the formula will be much easier.\n\nWhere do we go from here?\nAs of writing this, Algolia is live on thousands of websites and mobile applications, and this ranking strategy has consistently proven successful. We have spent a long time fine-tuning every detail and are pleased with the level of performance. We continue\u00a0to make relevance a core focus,\u00a0in particular to introduce new criteria to better enable\u00a0the personalization of results per user.\nWe\u2019re always happy to help you figure out how you could use this algorithm to achieve the ranking strategy that best fits your specific use-case. And we\u2019re eager to know what you think, so feel free to leave us a comment.","record_index":5},{"post_id":4791,"post_title":"Come see Algolia SuperSearch at SaaStr Annual 2016","post_date":1454928550,"post_date_formatted":"Feb 8th 2016","author_name":"Laura Evans","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/81158ee39a22e9a155ae5380da29cd20?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/come-see-algolia-supersearch-at-saastr-annual-2016\/","categories":["Community","News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/02\/indexing-part2-copy-360x200.png","time_to_read":2,"content":"How \u2018bout them Broncos!? The big game might be over, but Super Bowl mania isn\u2019t, at least at Algolia.\nSaaStr Annual 2016 kicks off in San Francisco on Tuesday, and we\u2019re excited to join more than 5,000 other SaaS professionals for three action-packed days of high-quality networking, learning from those who\u2019ve done it and, of course, great food, generously-poured drinks and plenty of fun. We\u2019ll be demoing our new SuperSearch project\u2014every Super Bowl ad ever aired, searchable with Algolia\u2014 for the first time at our booth all day Tuesday in the Mauna Kea Expo Hall\u00a0at booth 61.\nOur CEO Nicolas Dessaigne will also be part of the panel \u201cManaging and Scaling Globally (The Good, The Bad, and The Ugly)\u201d in the Tactical Theater at 3:30pm on Wednesday. \nOther Speakers include: Marketo CEO Phil Fernandez; Netsuite CEO Zach Nelson; New Relic CEO Lew Cirne; Mattermark CEO Danielle Morrill; Salesloft CEO Kyle Porter; Slack\u2019s April Underwood; Hubspot Founder Dharmesh Shah; Bloomberg\u2019s Sarah Frier; Co-Founder of TalkDesk, Christina Fonseca, Tomasz Tunguz of Redpoint Ventures, and over 100 more!\nCheck out the rest of the conference agenda here.\nIn addition to the speakers, panels and expo, there are nightly social events, including our free SaaS to Dev Happy Hour: A conversation on selling to developers featuring a panel with AWS, Algolia and Keen.io. Registration for our event is free, but space is running out, so don\u2019t forget to sign up.\nIf this is your first SaaStr Annual, definitely take a minute to read the guide they\u2019ve prepared for first timers, which includes some great tips on how to make the most of your conference experience.\nHope to see you there, and check back in next week for our conference recap.","record_index":0},{"post_id":4771,"post_title":"Find every commercial ever aired during the Big Game with Algolia SuperSearch","post_date":1454589450,"post_date_formatted":"Feb 4th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/find-every-commercial-ever-aired-during-the-big-game-with-algolia-supersearch\/","categories":["Community","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/02\/indexing-part2-copy-1-360x200.png","time_to_read":3,"content":"Algolia HQ is in San Francisco, so it\u2019s been pretty impossible to escape the Super Bowl mania sweeping the city. (If only they also actually swept the city.)\nOnce we got over our slight annoyance with all the street closures because of the Super Bowl Village they built two blocks from our office (40 miles away from the actual Super Bowl at Levi\u2019s Stadium), we realized that we\u2019re actually pretty excited for this weekend\u2019s game for two reasons:\n1. Neither of the teams playing is the Patriots!\n2.\u00a0THE COMMERCIALS!!!\nIf we\u2019re being completely honest here, Super Bowl party snacks came in at a close third.\nWhen we started talking about commercials from years past and trying to share them with one another, we realized it wasn\u2019t easy to find Super Bowl commercials. This is a huge bummer if you\u2019re anything like me and want to watch every commercial those majestic Clydesdales have ever been in.\nSo we started thinking...and hacking. Why can\u2019t we just make all the Super Bowl ads that ever aired searchable with Algolia? Everyone was onboard almost immediately once we explained that in order to search the data set, we would need to tag the data set...yes, we will pay you to watch YouTube.\nWe won\u2019t admit how much time we\u2019ve now wasted showing each other \u201cthe one with the flying baby\u201d or \u201cthe one with Will Ferrell in that stupid outfit,\u201d but we can tell you that actually building and implementing the search for this demo took us less than a day. This is how we did it:\n1. Scouted the interwebs to find the best archive of existing Super Bowl ads (found on YouTube).\n2. Ensured data searchability and homogeneity\u2014all ads must include Title, Brand, Year and Super Bowl edition.\n3. Pulled custom ranking data for the ads such as number of likes, number of views, etc.\n4. Enriched the existing data set with video content tagging.\n5. Designed and built the UI with instantsearch.js widgets.\nAnd voil\u00e0\u2014SuperSearch.\n\nWe\u2019re going to add this year\u2019s commercials","record_index":0},{"post_id":4771,"post_title":"Find every commercial ever aired during the Big Game with Algolia SuperSearch","post_date":1454589450,"post_date_formatted":"Feb 4th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/find-every-commercial-ever-aired-during-the-big-game-with-algolia-supersearch\/","categories":["Community","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/02\/indexing-part2-copy-1-360x200.png","time_to_read":3,"content":"\u2026 in as close to real time as possible and have already included a few that were pre-released. (Let\u2019s not talk about puppymonkeybaby.)\nWe really hope you enjoy SuperSearch as much as we\u2019ve enjoyed building it. Have a look for some of your favorite commercials, and don\u2019t forget to leave a comment and tell us which ones you like (and don\u2019t like).\nGo to SuperSearch","record_index":1},{"post_id":4731,"post_title":"Building a better iStyles shopping experience with Algolia search","post_date":1453815529,"post_date_formatted":"Jan 26th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/building-a-better-istyles-shopping-experience-with-algolia-search\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/01\/Pasted-image-at-2016_01_25-19_35-360x200.png","time_to_read":5,"content":"iStyles, a fashion accessories provider for consumer electronics, boasts more than 200K products and 1600 designs, as well as support for over 800 devices. With the already large and constantly growing number of products offered,\u00a0iStyles needs their customers to be able to find what they want quickly and easily. That\u2019s where we come in. Here\u2019s the scoop on how iStyles uses technology, including Algolia, to provide the best shopping experience for their customers.\nThe iStyles story\niStyles was originally formed in 1997 as a web development firm that offered web services and built applications and e-commerce infrastructure for companies worldwide. As such, iStyles has always been a technology company at its core and continually evaluates and leverages the best tech out there to provide an optimal shopping experience.\nIn 2004, after acquiring some product distribution contacts, iStyles decided to make their e-commerce store the main focus of the company and has since grown the business into the fashion accessories provider it is today.\nThe e-commerce challenge\nThe market has become very competitive over the years, and it is increasingly difficult to attract new customers in a saturated marketplace. iStyles needs to stand out from the crowd and prove to customers why we are a much better place to shop than Amazon or other online or brick-and-mortar retailers. We have built a loyal base of customers over the years but still have to continually work hard to attract new ones through marketing and other innovations.\nWhen we first launched in 2004, we based the store on an existing e-commerce platform. The problem with using a popular e-commerce platform is that your competitors can (and will) deploy stores using the same platform. There\u2019s no innovation\u2014no differentiator. The other problem we faced was the rigidity of such e-commerce platforms. One size may fit many, but it will not fit all. If you need to present or link products more meaningfully or have better","record_index":0},{"post_id":4731,"post_title":"Building a better iStyles shopping experience with Algolia search","post_date":1453815529,"post_date_formatted":"Jan 26th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/building-a-better-istyles-shopping-experience-with-algolia-search\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/01\/Pasted-image-at-2016_01_25-19_35-360x200.png","time_to_read":5,"content":"\u2026 control over your search results, you have no choice but to customize the solution.\nThe technical difficulties\u00a0of customization\nA good e-commerce store allows customers to explore the product range available and get to what they need or want quickly and easily. We know the significant role a search engine plays in this process, but the custom search algorithms we developed to overcome the limitations of the default e-commerce platform search engine were inadequate. It was very difficult to build both speed and flexibility.\nPerformance is an important part of the shopping experience, and we are obsessed with making things faster, but it was a large technical hurdle as our product range and traffic grew. Searches on our previous search engine took as long as 2 to 3 seconds for basic searches. It honestly sometimes feels like we\u2019re trying to overcome the limits of physics. And let\u2019s be real, a good search engine is so much more complicated than simply presenting precisely matching results. We didn\u2019t have the resources or time to build another Google from scratch.\nHow can we have great search with our limited resources?\nWe needed a search engine that would return relevant results quickly and present them clearly. We also needed better control over result ranking, a way to allow for typos and the ability to match on synonyms. So we started looking. We were looking for a solution that would allow us to deploy \"really awesome search functionality\" without having to re-invent the wheel or have more back-end infrastructure to worry about.\nExactly what we were looking for\u2014Algolia\nWe signed on for and evaluated trial accounts on several search services, but Algolia quickly emerged as the obvious choice during our 14-day trial period. We were impressed most by the comprehensive documentation and feature set.\nWe designed and built our new Algolia powered search functionality from the ground up and were thrilled it took us less than a week to do so thanks to the","record_index":1},{"post_id":4731,"post_title":"Building a better iStyles shopping experience with Algolia search","post_date":1453815529,"post_date_formatted":"Jan 26th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/building-a-better-istyles-shopping-experience-with-algolia-search\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/01\/Pasted-image-at-2016_01_25-19_35-360x200.png","time_to_read":5,"content":"\u2026 excellent Algolia documentation and easy to use Search API clients.\nOur front-end is now powered by Algolia's JavaScript Search API and custom scripts that tie everything together. We keep the search indexes updated via their PHP API at the back-end.\nIt works, it really works!\nSearch performance is critical for us, and we're glad to say that Algolia exceeds our expectations. There is something to be said when search performance improves from 3 seconds to 0.03 seconds. Search results are now virtually instant\u2014they are so fast that we can update results as the user types.\nWith this virtually instant response, our user search experience was completely transformed. The search is a pleasure to use, and the engine returns more relevant results as we are able to index more comprehensive data with no loss in performance and have more control over how results are presented.\nAlgolia Isn't just a simple search bar either. Users can search for anything they want in the search bar and narrow down the selection easily via the filter options (facets) in the left column. These facets also update virtually instantly based on what the user selects and\/or types.\nUsers can now get to what they want more quickly and in fewer steps. We have no doubt that the Algolia search implementation was part of the reason why our conversion rates this year are consistently higher than last year's.\nWe're planning on building new sections on iStyles, and it's likely that these new areas will be powered by Algolia at the backend.\nWe think you should give it a try\nAs developers, we are always trying to build better products. One way to do so is leveraging the great technology available to us. When we started the project to revamp the iStyles search functionality, our main aim was to allow customers to search more effectively. Algolia enabled us to not only build really good search functionality, but also essentially build a whole new way to experience iStyles.\nThere are options out there that may","record_index":2},{"post_id":4731,"post_title":"Building a better iStyles shopping experience with Algolia search","post_date":1453815529,"post_date_formatted":"Jan 26th 2016","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/building-a-better-istyles-shopping-experience-with-algolia-search\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2016\/01\/Pasted-image-at-2016_01_25-19_35-360x200.png","time_to_read":5,"content":"\u2026 better suit some projects depending on the resources you want to throw into it, but if you're looking for a fast, flexible and reliable backend that you can use to build your dream search engine, give Algolia a try.","record_index":3},{"post_id":4684,"post_title":"How we updated our JS libraries for React Native","post_date":1451985604,"post_date_formatted":"Jan 5th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-updated-our-js-libs-for-react-native\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/react-native-360x200.png","time_to_read":4,"content":"Our mission at Algolia is to make sure that every website or app has the best search experience possible. That's why we created several clients for iOS (in Swift and objective-C), Android, Windows\/C# and JavaScript.\nBut we're always looking for new\u00a0ways to help developers and are very interested in the possibilities introduced by React Native, Facebook\u2019s latest initiative to change the way we build\u00a0apps.\nBased on the awesome React library (that we already use), React Native lets you develop applications for Android and iOS using Javascript, hopefully making the development process easier and more consistent. What\u2019s even cooler is that it can leverage what's already available on NPM such as algoliasearch-client-js or algoliasearch-helper.\nBut React Native is a new execution environment different from Node or the web. Some expected features are implemented, some are not. When we tried to use our libraries to build an app using React Native, we hit a few bumps. We want to share with you what those were and how we got over them.\nTL;DR : we made our JS libraries compatible with React Native, and it\u2019s available right now! And we made a sample to help you get started.\nAlgolia and React Native in action.\nNot a Node polyfill\nThe first thing we noticed when working with React-Native is that even though it looks similar to Node, it\u2019s not quite the same. React Native lets you use  but doesn't polyfill the Node standard libraries like Browserify or Webpack do. The reason it doesn\u2019t do this is that it\u2019s not using the same tool to do the packaging and they made the choice not to implement this compatibility layer.\nIn order to make our module work in this new environment, we need to provide the core Node modules used by our libraries. Hopefully, all those Node standard libraries are available on NPM and we just need to add them to our package.json file as we are installing them ( for the win).\nBut won't we now run into issues with the module running on Node because","record_index":0},{"post_id":4684,"post_title":"How we updated our JS libraries for React Native","post_date":1451985604,"post_date_formatted":"Jan 5th 2016","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-we-updated-our-js-libs-for-react-native\/","categories":["Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/react-native-360x200.png","time_to_read":4,"content":"\u2026 we\u2019ve modified the package.json? Nope! Because of the built-in\u00a0rules of  in Node, we know that it will always use the built-in version if it is available.\nThe web but not exactly\nMost of the work in our JS-client library is based on HTTP calls. And hopefully, React Native implements a version of XHR for us. Unfortunately,\u00a0its XHR implementation comes with some particularities. Some of these differences break stuff, whereas some simplify the work. Let\u2019s see in more detail what differs from the web:\n\nReact-Native is for creating native apps, and they are not restricted to a single domain like web pages, so implementing CORS capabilities doesn't make sense.\nReact-Native doesn\u2019t come in many versions, and the set of features is clearly defined, so there\u2019s no need for web fallbacks\/compat hacks such as JSONP.\nOur library also relies on XHR timeout,\u00a0but it is not implemented there, so we rely on the good old setTimeout to implement this.\n\nThose changes were only needed in the\u00a0JS Client that handles the request to our API. The JS-Helper only relies on the client and some libraries, so it does not need further modifications.\nNew JS territories, here we come!\nWe applied what we learned about React Native\u00a0to make our JS libraries, the JS client and the JS Helper compatible with it, so now you have a new way to build your mobile applications on Android, iOS and more. Let us know if you have any questions.\nGet started with Algolia using React Native using our HN search sample.","record_index":1},{"post_id":4611,"post_title":"Robots vs. humans: Who will emerge supreme in the battle of the image tags?","post_date":1450130507,"post_date_formatted":"Dec 14th 2015","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/robots-vs-humans-who-will-emerge-supreme-in-the-battle-of-the-image-tags\/","categories":["Community","Technology"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/slack-imgs.com_-360x200.png","time_to_read":2,"content":"One of the most exciting challenges in tech today is undoubtedly trying to beat the human brain at its own game. Whether it\u2019s with the super \u201chelpful\u201d automatic checkout at the grocery store or IBM\u2019s supercomputer Watson, technology has come a long way in keeping pace with and occasionally surpassing human capabilities.\nInterpreting images, however, is still considered a human\u2019s game as images are processed faster by the brain than a machine, and human minds consistently outperform technology when it comes to image analysis.\nAs a full-text search engine, Algolia searches a lot of images but cannot actually search the images directly. We rely instead on metadata associated with the image such as title, description, location and other tags to help us make sense of the image. We often run into situations where an image has no associated words or tags but customers still want to search within those images. So what happens then?\nLuckily, we have seen a lot of text-based algorithm tools emerge in the past decade. One such tool, Imagga, boasts an image recognition software that can\u2014amongst other things\u2014automatically generate these missing image tags. And it just so happens to be an API too! Imagga is essentially the missing link that enables searching raw images within Algolia. Algolia provides a powerful way to explore large amounts of data, and Imagga brings to the table the ability to create textual data from a set of images.\nWe\u2019ve devised a game combining both tools to compare how humans and machines tackle image tagging and how this affects your ultimate search experience. Find out who comes out on top in\u00a0Human vs. Robot: Battle of the image tags.\nPlay Clash of Tags\nIllustration by Martin David","record_index":0},{"post_id":4625,"post_title":"Bringing Algolia\u2019s faceting experience to mobile\u2013the evolution of AppApp","post_date":1449618382,"post_date_formatted":"Dec 8th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-algolias-faceting-experience-to-mobile-the-evolution-of-appapp\/","categories":["Customers","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/appappio-360x200.png","time_to_read":5,"content":"We built Algolia to make search easier for everyone, and a lot of our customers implement search within already existing websites or mobile applications.\nBut then we also have customers like AppApp.\nAppApp, an independent search engine for the iOS App Store, was actually conceived after its founders realized the potential of Algolia while building solutions for existing clients. Based on faceting, grouping\u00a0search results into categories, AppApp lets you search within the official App Store in ways the native search does not allow. AppApp immediately saw an opportunity to bring this faceting functionality to their mobile interface and reached out to us to share their process. Dan Singerman, its\u00a0founder, explains.\n\n \nFaceting with Algolia\nWith a typical desktop faceted search design, the instant nature of Algolia's faceting allows you to quickly see the breadth of data, understand the results of your actions and know whether you have applied overly specific criteria. Toggling a filter takes fractions of a second, so there is very little perceived cost to experimenting with filter combinations to produce the desired results.\nAlgolia obviously offers a great faceting experience, but the wheels were in motion\u2014how do you fit it into a mobile design?\nMaking it work for mobile\nFrom the beginning, we wanted\u00a0the search results to visibly change as the faceting was taking place. This is, after all, the essence of instant search. Most mobile faceting interfaces take you to a separate page when refinements are applied, and you lose the instant feedback you get with a desktop design.\nA good example of this is Amazon.com. While they have a beautiful interface, a user would be unlikely to experiment with a large number of different faceting criteria. Who wants to keep navigating back and forth between the search categories and the results?\n\n\nBuilding AppApp.io for mobile\nOur first mobile implementation was a naive attempt to replicate the desktop approach with facets","record_index":0},{"post_id":4625,"post_title":"Bringing Algolia\u2019s faceting experience to mobile\u2013the evolution of AppApp","post_date":1449618382,"post_date_formatted":"Dec 8th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-algolias-faceting-experience-to-mobile-the-evolution-of-appapp\/","categories":["Customers","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/appappio-360x200.png","time_to_read":5,"content":"\u2026 vertically aligned alongside results.\n\nIt was immediately obvious that this was not going to be successful. The touch targets for the facets were far too small, and the design would not extend to support the large number of facet categories and values we were planning to have. We realized that showing the facet categories and their values simultaneously with the results was just not going to be possible on a mobile screen.\nOur next iteration moved the facet values into a modal while leaving the facet categories vertically aligned with the results.\nThis iteration allowed facet selection to instantly update the search results, even if it was occurring in the background. Given that we didn\u2019t have the space of a desktop, we felt this was a good compromise. The more pressing issue with this approach was that on a mobile device, the result set was very narrow and did not make good use of space.\nWith apps, text descriptions alone do not give a good sense of whether an app is what you\u2019re searching for. We like the official App Store\u2019s approach of showing screen shots in line with results and wanted to replicate that in our design. Because of this, we felt that constraining the search results to only two thirds the width of a mobile screen was not optimal. We wanted to make use of the full width available whilst also having an effective faceting interface.\nSo our next iteration did just that.\n\nWe floated the faceting component over the results and moved it below the default title position for the first result so the title wasn\u2019t hidden. Now we had full width results alongside facets. In retrospect, this was a bad idea. Nobody wants components to sit permanently on top of each other. What were we thinking? But trying this on a real device very quickly told us that.\nSo, our next evolution allowed the floating facet component to be hidden.\n\nThis was definitely an improvement in that the facet controls could be hidden so as not to obscure the result set, but it felt","record_index":1},{"post_id":4625,"post_title":"Bringing Algolia\u2019s faceting experience to mobile\u2013the evolution of AppApp","post_date":1449618382,"post_date_formatted":"Dec 8th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/bringing-algolias-faceting-experience-to-mobile-the-evolution-of-appapp\/","categories":["Customers","UX"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/12\/appappio-360x200.png","time_to_read":5,"content":"\u2026 unnatural. You had to actively decide to hide the facets when scrolling through the results if you wanted to be unencumbered by the floating facet component. Weren\u2019t facets the whole point of this project? This wasn\u2019t going to work, but it did take us another step toward our current design.\nWe figured it out\u2014our current design\n\nWe wanted facets (filters) that would appear consistently when activated by the user but naturally disappear as the user focused on the results. We moved the facet control below the search box and added a tab so it can be opened in a drawer-like fashion.\nWe chose to label the tab instead of assigning it an icon because we felt an icon alone would not be a strong or clear enough call to action. We decided on the label \u2018filters\u2019 because we thought this would be much more meaningful to users than \u2018facets\u2019 or \u2018refinements\u2019. We also made the tab bright orange so it would stand out to users and could not be easily overlooked.\nThe facet drawer also naturally closes as you start to scroll, which gives the results more screen real estate.\nDoes it work?\nThe initial evidence is positive for this approach. Since launching a month ago, our analytics tell us that 61% of users on mobile interact with the facet controls via this interface. While we\u2019d definitely like to increase that number, this is a good start. The analytics also show that users who interact with this control do indeed try many different combinations of facets. Ultimately, we\u2019re pleased with the results, even though it took several iterations to get here, and continue to be impressed by the faceting capabilities of Algolia that inspired this project.\nIllustration by Martin David","record_index":2},{"post_id":4575,"post_title":"How Algolia keeps all your data safe but searchable on Solid","post_date":1448466982,"post_date_formatted":"Nov 25th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-keeps-all-your-data-safe-but-searchable-on-solid\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/11\/Solid.io-Blog-image-360x200.png","time_to_read":5,"content":"In today\u2019s world of easily accessible data, privacy is key. But protecting data is often a manual and expensive process requiring extensive planning. Solid, an app designed to help you run easy and effective meetings, faces a unique struggle when it comes to privacy. They have access to large amounts of personal data and need it to be searchable yet secure. \nAligned with their mission to make meetings easy and effective, they needed a way to keep privacy easy and effective while not minimizing any search functionality. So, they called Algolia.\u00a0\nThibaut Davoult, Growth Manager at Solid, recently reached out to share the Solid\/Algolia experience with us. Here\u2019s what he had to say.\n\nWhat does Solid do?\nSolid is a meetings management app used by managers, employees and freelancers around the world. Our platform has hosted hundreds of thousands of meetings and counting, and clients such as LinkedIn, Dropbox, Deezer and Airbnb trust us with their private data. Solid, in return, places the utmost importance on privacy.\nProtecting your data\nIn fact, privacy is one of the stepping stones for Solid\u2019s success. Meeting invitations can contain very sensitive data. Even meeting names alone can be revealing, so we work hard to prevent any of this info from leaking. And this\u00a0has been the case from the very beginning.\u00a0We largely manage to keep privacy in check thanks to deep provider integration. We currently integrate with Office 365 and Google through their OAuth system and rely on ACL to keep users out of others\u2019 meetings.\nBut we needed search...\nThe need for a search feature quickly became evident while trying to manage all these\u00a0meetings. We had a large amount of data and no clear way to organize and find information. Thus, the problem to solve was quite straightforward:\nHow do we implement a search feature that:\n1) Keeps our users\u2019 privacy in check\n2) Doesn\u2019t hurt our app\u2019s performance\n3) Can be implemented as quickly as possible\nWe had a few options:\n1)","record_index":0},{"post_id":4575,"post_title":"How Algolia keeps all your data safe but searchable on Solid","post_date":1448466982,"post_date_formatted":"Nov 25th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-keeps-all-your-data-safe-but-searchable-on-solid\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/11\/Solid.io-Blog-image-360x200.png","time_to_read":5,"content":"\u2026 Develop a less-satisfying search via MySQL+Redis\n2) Implement open source solutions like ElasticSearch or Solr\n3) Go with a 3rd party API \nThe latter seemed like the way to go. Here\u2019s why it seemed like our best option and how we chose Algolia.\nWhy Algolia?\nAlgolia is the leading hosted search API and the only one that delivers instant and relevant results from the first keystroke. Once we realized this, it didn\u2019t take long for us to reach out to\u00a0learn more. What followed, as you\u2019ll see, cemented our decision. We knew it was a no brainer to implement them.\nAn\u00a0easy but careless way to keep search performance up would have been to go without a backend proxy so the JavaScript code could directly send requests to the search engine from the end-user browser or device. But that\u2019s not easy. Since JS is executed client side, it would expose all the code and access keys to users, allowing them to search through other meetings. Not very secure. We needed to find a way to secure this information without going through a backend\u2014and secured\u2014proxy.\nWe recognized that Algolia\u2019s Secured API Keys were the appropriate solution. They allow you to securely apply filtering on the query, done via the back end for optimal security. That means the JS can directly and securely request the Algolia API without any hiccups or slowdowns.\nWe also needed to add a tagging system to ensure users could and would only access meetings that were meant for them. Each indexed meeting contains a tag array with a list of participants that looks like this: `[\u201cuser_xyz\u201d, \u201cuser_abc\u201d]`.\nWhen users start a search, their searches are automatically filtered with their associated tags. As a result, they will only be able to search meetings that contain both their keyword AND their user ID. This way, we\u2019re guaranteed not to show anyone else\u2019s results.\n\nTags are associated with the meeting\u00a0when they are indexed.\n\nView the code on Gist.\n\nThe back end fixes the tag filters and prepares a","record_index":1},{"post_id":4575,"post_title":"How Algolia keeps all your data safe but searchable on Solid","post_date":1448466982,"post_date_formatted":"Nov 25th 2015","author_name":"Guillaume Dumortier","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/37e607f71b47cc1e06a61e51d9cf5770?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/how-algolia-keeps-all-your-data-safe-but-searchable-on-solid\/","categories":["Community","Customers"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/11\/Solid.io-Blog-image-360x200.png","time_to_read":5,"content":"\u2026 session-secured token for the search using Algolia\u2019s php client.\n\nView the code on Gist.\n\nThe data for search is sent to the front end through a specific endpoint.\n\nView the code on Gist.\n\nThe JS client uses the token and the filters to directly request the Algolia search endpoint.\n\nView the code on Gist.\nSolid also allows users to ignore events on a case-by-case basis. We know you don\u2019t want to see meetings you\u2019ve ignored in your search results, so we filter them out per your settings.\nPerformance at scale\nCould we have done this using other methods such as Elasticsearch? Sure, but at what cost? To get the same performance level and still keep up with security, we would have had to pour quite a bit more resources into the project. Resources, ultimately, that we just don\u2019t have.\nWhile we did all of the above in only five days using Algolia, it would have taken close to three more weeks to complete on Elasticsearch. That\u2019s not even accounting for the maintenance work we would have needed to perform as we processed our initial 500k meetings. This would have been increasingly difficult to scale quickly as we onboard new users. We usually love it when our developers tackle difficult problems (and they do too), but this is just an unnecessary problem to have. But protecting your privacy? That\u2019s a problem that's right up our alley! And giving you great search? That\u2019s right up Algolia\u2019s!","record_index":2},{"post_id":4526,"post_title":"Announcing instantsearch.js: Everything you need for great search in one library","post_date":1447857591,"post_date_formatted":"Nov 18th 2015","author_name":"Alexandre Stanislawski","author_image_url":"https:\/\/secure.gravatar.com\/avatar\/9b5b54cd240a4e451639d3185ed8045d?s=40&d=mm&r=g","permalink":"https:\/\/blog.algolia.com\/announcing-instantsearch-js-everything-you-need-for-great-search-in-one-library\/","categories":["News"],"image":"https:\/\/blog.algolia.com\/wp-content\/uploads\/2015\/11\/illus-360x200.png","time_to_read":4,"content":"We\u2019re very excited to announce the launch of instantsearch.js, our new library of UI widgets to help you build the best instant-search experience with Algolia\u2019s hosted search API.\nSo how does this library change search?\nOur mission at Algolia is to make sure any website or application has\u00a0the best search experience. But we don\u2019t want to make this only available to developers with extensive coding knowledge. Which leads us to the second part of our mission\u2014making great search available to everyone. This is where instantsearch.js comes into play.\nWith this new library, we\u2019re separating the logic and design of search, empowering developers and designers alike to create unique search experiences.\n\nThrough hundreds of search implementations, we\u2019ve identified\u00a0and developed best practices for delivering a top-notch search experience. instantsearch.js packages those best practices into a single JavaScript library of UI widgets that you can assemble and customize as you wish.\nWhen\u00a0building or adding an instant-search experience to your website, you can now think in terms of end results and how pages will look and feel, rather than having to carefully create and refine\u00a0an Algolia query.\ntl;dr; Using instantsearch.js, you can:\n\nBuild an instant-search UI\u00a0faster than ever\nFocus entirely on building the UI, forget about the complex code\nPlug and play\u2014everything is included in one library\n\nHow did we get here?\nThere are two ways to implement Algolia on\u00a0the front end. The first way is with an autocompletion menu.\nThis kind of UI can be broken down into two elements: the text input and the search suggestions. As the user types, the search suggestion list refreshes and refines, providing the user with only the most relevant results.\nAlgolia's autocomplete.js in action\n \nThe second way, what we like to refer to as instant search, is more difficult to implement as it is made up of multiple elements such as search filters, resul