# PlanningAlerts ![example workflow](https://github.com/openaustralia/planningalerts/actions/workflows/rubyonrails.yml/badge.svg) [![View performance data on Skylight](https://badges.skylight.io/status/JANyIYPM9FwR.svg?token=dDtVtXbOqFPrEZjsGnMPoT-Ss9h3UHtYtZZjFLE9KWo)](https://www.skylight.io/app/applications/JANyIYPM9FwR) Find out and have your say about development applications in your area. This is the code for the [web application](https://www.planningalerts.org.au/) side of things written using Ruby on Rails. If you're interested in contributing a scraper read our [step-by-step guide to writing scrapers](https://www.planningalerts.org.au/how_to_write_a_scraper) on our scraping platform, [morph.io](https://morph.io/). PlanningAlerts is brought to you by the [OpenAustralia Foundation](https://www.oaf.org.au/). It was created by Matthew Landauer and Katherine Szuminska. ## Development ### Install Dependencies * [Docker compose](https://docs.docker.com/compose/install/) ### Checkout The Project ```bash git clone https://github.com/openaustralia/planningalerts.git cd planningalerts ``` ### Resetting Your Environment To force a rebuild from scratch, stopping and removing containers and volumes (including the databases): ```sh docker compose down -v --rmi all # Check if this volume is "still in use" docker volume rm planningalerts_gem_cache ``` If you get a `volume is in use` error, remove the container using the given id and then retry, for example: ```sh > Error response from daemon: remove planningalerts_gem_cache: volume is in use - [98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b] docker rm 98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b > 98ba717d944d01976ac9074e6a1119e70d3aebd53f5d5449957324926f0bbb4b docker volume rm planningalerts_gem_cache ``` ### Overriding DB host and port for non docker dev To use docker for the database, but allow you to run the application locally, for example to simplify single step debugging in IDE's, you can override the following ENV vars in `.envrc` (for `direnv`) or manually: ``` export DB_HOST=localhost export DB_PORT=15432 ``` ### Alternative web port You can set up a second web server port so all your user and passwords from various projects are not all mixed together. For example, add the following to `docker-compose.override.yml` to use port 30PA (3072): ```yaml services: web: ports: - "3072:3000" ``` This adds an extra port mapping, local port 3000 will still be mapped to port 3000 on the container as well. ### Setup The Database Set up the databases - `docker compose run web bin/rake db:setup` This will trigger a build if needed. You can manually trigger a build by first running - `docker compose build` ### Start the application * `docker compose up` * Point your browser at Append `--build` to rebuild any changes to Dockerfile. ### Run The Tests * In a separate window - `docker compose run web bin/guard` * Press enter to run all the tests ### Setup Development Data Initially `db/seeds.rb` (called by `db:setup` will have set up: * user `admin@example.com` * confirm via the email at http://localhost:1080/ * Add the admin role using the rails console command `User.first.add_role(:admin)` * Authority: Marrickville Council, with * Application to demolish a swimming pool at 28 Grey Street, Emu Plains NSW 2750 Note: external links to morph.io and council site for this example are broken as the external site details have changed in the last 5+ years. #### Using production data locally If you want to use production data locally, follow the instructions in the [doc/using_production_data_locally.md](doc/using_production_data_locally.md) file. ### Emails in development In development all emails are sent locally to [mailcatcher](https://mailcatcher.me/). The emails can be viewed at . Email templates are developed using [Maizzle](https://maizzle.com/). The development workflow goes as follows: ```sh cd maizzle npm run dev production ``` Then point your browser at . As you edit the templates in `maizzle/src/templates`, the templates will get automatically compiled to the erb template in the main rails app tree and will also refresh the browser. ### Type checking We're using [Sorbet](https://sorbet.org/) to add type checking to Ruby which otherwise is a dynamic language. To run the type checker: ```sh docker compose run web bin/srb ``` To update the rbi files, use these commands: ```sh docker compose run web bin/tapioca gem docker compose run web bin/tapioca dsl docker compose run web bin/tapioca dsl --environment=test ``` We use Shopify's [tapioca](https://github.com/Shopify/tapioca) gem to manage all our rbi files. We **don't** use `bundle exec srb rbi ...`. ### Other useful docker commands * `docker compose ps` - View running containers and their status * `docker compose logs [-f] [service]` - View logs for all or a specific service, add `-f` to continue watching (the up command also does this) * `docker compose restart web` - Restart the web container (eg when you make a change to the code) * `docker compose down` - Stop and remove containers (keeps volumes and images intact for quick restart) * `docker compose exec web bash` - Run a shell in the running container * `docker compose run --rm web bin/rails console` - Run a once-off command in new container and then clean it up afterwards * `docker system prune --all` - Remove all stopped containers, orphaned images / networks, build cache etc. (Remove the -v if you want to keep your databases and gem cache) #### If things go wrong with gem updates If things go wrong, force a rebuild from scratch (preserving DB volumes): ```shell docker compose down docker system prune --all docker volume rm planningalerts_gem_cache docker compose up --build ``` Then rebuild sorbet again in another terminal window: ```shell docker compose run web bin/srb docker compose run web bin/tapioca gem docker compose run web bin/tapioca dsl docker compose run web bin/tapioca dsl --environment=test ``` ## Deployment The code is deployed using Capistrano. To deploy to production run: ```sh bundle exec cap production deploy ``` This command is defined in `config/deploy.rb`. Sometimes you want to deploy an alternate branch, for instance when deploying to the `test` stage. In this case you'll need to set the `branch` variable after recipes are loaded by using the `--set` argument instead of `--set-before`, e.g. ```sh bundle exec cap staging --set branch=a-branch-i-want-to-test deploy ``` View more available Capistrano commands with: ```sh bundle exec cap --tasks ``` ## Upgrading Ruby in production Upgrading Ruby in production is an unbelievably painful process right now. I'm sorry. Let's make it simpler but in the meantime: TODO: This section is now out of date as we now have a blue/green deploy setup that allows us to build a new image with a new ruby version and deploy it side by side with the old version and switch over between them Note for new deploy on new servers: Make sure you run "bundle exec cap production puma:install" first. Otherwise the "puma:reload" capistrano task will fail ### Update the application code * Change `.ruby-version`. Run tests to make sure nothing has broken. ### Upgrade ruby in staging * In the `oaf/infrastructure` repo update `roles/internal/planningalerts/meta/main.yml` to add the new ruby version before the current one. The last listed one is the default. We don't yet want to change the default * Install the new ruby on the server by running `ansible-playbook site.yml -l planningalerts`. Remember to set your python virtual environment if you're using that. * Deploy new version of the application with upgraded `.ruby-version` to staging by running `bundle exec cap staging deploy` * Login to each webserver in turn (as root user). Then, `cd /srv/www/staging/current; gem install bundler:1.17.3` * Login to each webserver in turn (as deploy user). Then `cd /srv/www/staging/current; bundle install --gemfile Gemfile --path /srv/www/staging/shared/bundle --deployment --without development test`. This step is necessary if you're upgrading a ruby major version. You might be able to skip it if not. * Edit `roles/internal/planningalerts/templates/default` to change the ruby version used by passenger for staging in staging to the new version * Run ansible again with `ansible-playbook site.yml -l planningalerts` * Check deployed staging is still working by going ### Upgrade ruby in production * Deploy new version of the application with upgraded `.ruby-version` to production by running `bundle exec cap production deploy` * Check deployed production is still working by going * Login to each webserver in turn (as deploy user). Then `cd /srv/www/production/current; bundle install --gemfile Gemfile --path /srv/www/production/shared/bundle --deployment --without development test`. This step is necessary if you're upgrading a ruby major version. You might be able to skip it if not. * Edit `roles/internal/planningalerts/templates/default` to change the ruby version used by passenger for production in staging to the new version * Run ansible again with `ansible-playbook site.yml -l planningalerts` * Check deployed production is still working by going During this keep a close eye on disk space on the root partition as this might get close to full ### Tidy up * Remove old ruby version from `roles/internal/planningalerts/meta/main.yml` in the `oaf/infrastructure` repo * Rerun ansible with `ansible-playbook site.yml -l planningalerts` * If you did a ruby major version upgrade (e.g. ruby 2.6.6 to 2.7.4) then you should also clean up an old unused bundler directory that is taking up a lot of space. Login to each webserver in turn (as deploy user). Then `cd /srv/www/staging/shared/bundle/ruby` and remove the unused directory (e.g. `2.6.0`). Do the same for production `cd /srv/www/production/shared/bundle/ruby`. ### Adding a new authority Someone has just written a new scraper for PlanningAlerts, woohoo! :tada: Now we need to add it to the site. The first step is to fork their repository into the @planningalerts-scrapers organisation. This gives us control over the repository. If we didn't do that then the person could potentially inject bad data without us noticing. A more likely problem is that they go off and do something else and we have no control of the repository to fix things. Once you've done that add it to morph and do an initial scrape to get some data. It's always a good idea to check that the scraper is getting the data we expect. Just like you'd do if someone had fixed a scraper and opened a pull request. Speaking of pull requests, because we've forked the scraper GitHub turns off issues on forked repositories. It's a good idea to switch it back on for ours so that other people can open issues and pull requests against the @planningalerts-scrapers repository. Now that we have a working scraper and some data we can add the new authority to PlanningAlerts. First, log into the admin backend and browse to the authorities section: Click _New Authority_ in the top-right of the page. Now fill out all the details, here's what needs to go in each field: * Full name: The full name of the authority that's displayed throughout the site, e.g. Bellingen Shire Council * Short name: Used in the URL (must be unique), e.g. bellingen * Email: email address that comments get sent to. Try and find a specific DA comment address or failing that use the main contact email address for the council, e.g. * Scraping, morph name: The morph name of the scraper you just forked, e.g. planningalerts-scrapers/bellingen Click _Create Authority_. Now scrape some applications so you can see them on the new authority page - click _Scrape_. Visit the new authority page, e.g. You should see that some applications have already been collected. If not wait a few seconds and refresh the page. Once you've got some do a quick check on a few by clicking _Browse all recent applications_ and selecting a few. Make sure the comment form is visible (that means you set an email address). If all looks good then thank the contributor for helping tens of thousands of people get PlanningAlerts by tweeting about it from @PlanningAlerts. It's always fun to @mention the council too, sometimes we get a RT :grinning: >We've just added @BellingenShire thanks to @LoveMyData. Another 12,886 people can now get PlanningAlerts! e.g. ## Periodic jobs We use `sidekiq-cron` gem to run system-wide periodic jobs that are not tied to a specific server. As noted in `Gemfile` Use "regular" cron for jobs that need to run on every machine (though none are currently defined for the `deploy` user). See `config/cron.yml` for the list of periodic jobs. ## Utilities You can run rake tasks on the server using capistrano, for example: ```shell bundle exec cap production rake:invoke TASK=my:rake_task # Or with maintenance on before and off afterwards bundle exec cap production rake:invoke:with_maintenance TASK=my:rake_task ``` Useful rake tasks: * `db:version` - Retrieve the current schema version number * `db:stats` - Show table count and size Or you can run them locally, eg `bundle exec rake db:stats` ## Contributing * Fork the project on GitHub. * Make a topic branch from the main branch. * Make your changes and write tests. * Commit the changes without making changes to any files that aren't related to your enhancement or fix. * Send a pull request against the main branch. ## Credits Our awesome contributors can be found on the [PlanningAlerts site](https://www.planningalerts.org.au/about). This project is tested with BrowserStack. ## License GPLv2, see the LICENSE file for full details.