ChangeLog for GoatCounter
=========================
This list is not comprehensive, and only lists new features and major changes,
but not every minor bugfix. The goatcounter.com service generally runs the
latest master.

2023-12-10 v2.5.0
-----------------
This release requires Go 1.21.

Features:

- Quite a few tables are rewritten to a more efficient format. For small to
  medium instances this will take a few minutes at the most, but if you have
  very large instances this may take a few hours. It also requires enough free
  disk space to rewrite the `hits` table.

  If you want to run steps manually then you can view the migration with:

      % goatcounter db migrate -show 2023-05-16-1-hits

  Or if you use PostgreSQL:

      % goatcounter db migrate -show -db postgresql+dbname=goatcounter 2023-05-16-1-hits

- The `User-Agent` header is no longer stored; only the browser and system
  parsed out of there. It's pretty reliable, and especially mobile browser
  User-Agents are ridiculously unique. It was always stored only "in case the
  detection got it horribly wrong", but this has never been needed.

- Add `proxy` option in `serve -tls` flag,  to give a hint that a secure
  connection will be used, so we know what value to use for the cookie
  secure/samesite flags.

- Add *experimental* "dark mode"; this needs to be enabled explicitly in the
  user settings. I need help to make this decent:
  https://github.com/arp242/goatcounter/issues/586#issuecomment-1287995673

- Show difference of pageviews compared to previous period on the dashboard.

- Make setup of a new installation a bit easier: instead of telling people to
  use the CLI, display a form when the database is 100% empty.

Fixes:

- Collecting stats was broken when "sessions" was disabled in the site settings.

- Use navigator.sendBeacon by default in count.js. This will allow using click
  events on things like PDF files and external URLs:

      <a href="file.pdf" data-goatcounter-event="file.pdf">
      <a href="http://example.com" data-goatcounter-event="ext-example.com">

- Sometimes the order of pages was wrong when using PostgreSQL.

- Few smaller bugfixes.

2022-11-15 v2.4.1
-----------------
- Fix regression that caused the charts for SQLite to be off.

2022-11-08 v2.4.0
-----------------
- Add a more fully-featured API that can also retrieve the dashboard statistics.
  See https://www.goatcounter.com/help/api for documentation.

  This is still as "v0" because some details may still change.

- Default API ratelimit is now 4 requests/second, rather than 4 requests/10
  seconds. You can use the `-ratelimit` flag to configure this.

- Can now also merge paths instead of just deleting them (the "Settings → Delete
  pageviews" tab was changed to "Manage pageviews").

- Add `goatcounter dashboard`, which uses the new API to display the dashboard
  in the terminal (only a basic non-interactive overview for now).

- Add a "Show fewer numbers" user setting; this is intended to still give a
  reasonably useful overview of what happens on your site but prevent an
  “obsession” over the exact number of visitors and stats.

- No longer store or display "pageviews": always store and display "visitors"
  instead.

  The visitor count is the only thing that's interesting in pretty much all
  cases; the "raw" pageviews are still stored for some future purposes (such as
  "time on page"), but are no longer stored in most other contexts.

- Add infrastructure for "dark mode".

  This is not yet enabled by default because all "dark mode" themes look "bad"
  on my eyes, and I'm not really sure what works well for people who do like it.

  So some help is needed here. See:
  https://github.com/arp242/goatcounter/issues/586#issuecomment-1287995673

2022-10-17 v2.3.0
-----------------
- Expand campaigns: the `utm_campaign` or `campaign` parameter now is tracked
  separately, and add a dashboard panel for campaigns. See:
  https://www.goatcounter.com/help/campaigns

  Old data isn't backfilled as this information wasn't stored.

- There are now binaries for Windows, macOS, {Free,Open}BSD, and illumos.

- WebSockets are now disabled by default, as it turned out a lot of people had
  trouble proxying them. You can enable it with `goatcounter serve -websocket`.

- Add `-dbconn` flag for `serve` to allow setting the maximum number of
  connections. The default is also lowered from 25 to 16 for PostgreSQL.

- Add `-store-every` flag to control how often to persist pageviews to the
  database.

- Add "Sites that can embed GoatCounter" setting to allow embedding GoatCounter
  in a frame.

- Add "Hide UI for public view" setting to allow hiding the UI chrome and
  display only the charts.

- Quite a few bugfixes and minor additions.

2022-02-16 v2.2.0
-----------------
- The database connection string changed; you now need to use `-db
  engine+connect string` rather than `engine://connect string`:

      -db sqlite+[sqlite connection string]
      -db postgresql+[sqlite connection string]

  Various aliases such a `sqlite3`, `postgres` also work.

  The previous "url-like" strings conflicted with PostgreSQL's URL connection
  strings, causing confusion.

  `://`-type strings without a `+` will be rewritten, but will issue a warning.

- GoatCounter can now collect language statistics as well, from the
  `Accept-Language` HTTP header. This is disabled by default, but can be enabled
  in the site settings.

- Charts are now drawn as a line chart by default; you can choose to use bar
  charts in the widget settings menu by selecting the "chart style" for the
  "Paths overview" and/or "Total site pageviews"

  Both charts are also completely reïmplemented by drawing on a canvas instead
  of aligning divs in a flexbox because rendering thousands of divs in a flexbox
  is actually fairly slow.

- The "View as text table" button in the header moved to the "Chart style"
  section mentioned above; this checkbox was added before the configurable
  dashboard feature, and especially now that you can set a chart style it makes
  more sense to set it there.

- Data is now sent over a WebSocket, rather than rendering everything. The
  upshot of this is that the perceived performance is better: it only needs to
  calculate the data that's initially visible, and it's okay to wait a bit for
  the data that's not. The downside is that you need JavaScript, but that was
  already the case to render the charts.

- There is a "server management" tab in the settings which allows viewing and
  editing some server internals. This page is only available to users with the
  (new) "server management" access.

  All sites with just one user have this user's permissions automatically
  "upgraded"; sites with more than one user since I don't know which user should
  have which permissions.

  To prevent updating users, you can use (*before* running migrations):

      % goatcounter db query "insert into version values ('2021-12-13-2-superuser')"

  To update an existing user, you can use:

      % goatcounter db update users -access superuser -find=martin@arp242.net

- Add `-ratelimit` flag to configure the built-in ratelimits (the default values
  are unchanged). See `goatcounter help serve` for details.

- New translations: Italian, Spanish (Chilean), Turkish.

2021-12-01 v2.1.0
-----------------
Aside from a number of small fixes and improvements, major changes include:

- Support for translations; see https://www.goatcounter.com/translating for
  details how to translate GoatCounter.

- The import path is now updated to use "zgo.at/goatcounter/v2" so that e.g. "go
  install zgo.at/goatcounter/v2" works. This should have been done with the 2.0
  release, but I didn't realize how this all worked.

- The visitor counter now supports the `start` and `end` parameters and the JSON
  endpoint returns `count` as well, to get the total pageview count.

- You can now make the dashboard viewable to anyone who has a secret token (e.g.
  https://mystats.example.com?access-token=5g4..)

This release requires Go 1.17 to build.

2021-04-13 v2.0.4
-----------------
- Deal with duplicate entries in the `user_agents` table in the migration
  instead of erroring out; mostly fixes a situation that could happen if you ran
  the broken migrations in 2.0.0 or 2.0.1.

2021-04-02 v2.0.3
-----------------
- Fix if you had already run the broken migrations in 2.0.0 or 2.0.1.

- Handle failures in `goatcounter import` a bit more gracefully.


2021-04-02 v2.0.2
-----------------
- Fix migration order.

- Don't display the expected "Memstore.Init: json: cannot unmarshal number /
  into Go struct field storedSession.paths of type int64" error log on startup;
  this got displayed once, but was a bit confusing.

- Display a message on startup after the first update to direct people towards
  the 2.0 release notes and "goatcounter reindex".


2021-03-29 v2.0.1
-----------------
- Fix migrations 🤦 They worked when they were written, but a bunch of things
  changed in GoatCounter and some older ones didn't run any more.

- Add `-test` flag to `goatcounter db migrate` to rollback a migration, so it's
  easier to test if migrations will run correctly without actually changing the
  database.


2021-03-29 v2.0.0
-----------------
The version is bumped to 2.0 because this contains a number of incompatible
changes: several CLI commands got changed, and it includes some large database
migrations – running them is a bit more complex than the standard migrations.

An overview of **incompatible** changes:

- There are some rather large changes to the database layout for better
  efficiency; this means:

  - Somewhat faster queries.
  - Greatly reduced disk space requirements for the database.
  - The Browsers, systems, size, and location stats are filtered if you enter
    something in "filter paths". Previously this always displayed the site
    totals.
  - "Purge path" now works as expected for all stats.
  - Easier to add new statistics in the future.

  To update:

  1. You **must** first update to 1.4.2 and run all migrations from that.
     **Updating from older versions directly to 2.0.0 will not work!**

  2. Run the migrations with `goatcounter serve -automigrate` or `goatcounter
     migrate`.

  3. You probably want to manually run `VACUUM` (or `VACUUM FULL` for
     PostgreSQL) after the migration to free up unused rows. This isn't strictly
     required, but frees up disk space, and removes some of the autovacuum
     pressure that will run in the background.

  4. Run `goatcounter reindex`.

  All of this may take a while if you've got a lot of data. For about 500,000
  pageviews it takes about 3 minutes on SQLite, but if you've got millions of
  pageviews it may take an hour or more.

  If you want to keep pageviews while this is running you can:

  1. Write it to a logfile from a proxy or temporary HTTP server and run
     `goatcounter import` on this after the migrations are done.

  2. Use `goatcounter buffer`.

- `goatcounter migrate` is now `goatcounter db migrate`. It also behaves a bit
  different:

  - `goatcounter db migrate pending` lists only pending migrations, and will use
    exit code 1 if there are any pending migrations.
  - `goatcounter db migrate list` lists all migrations, always exits with 0.

- If you use PostgreSQL you need PostgreSQL 12 or newer; this was already the
  case before and you could run in to some edge cases where things didn't work,
  but this is enforced now.

- The `none` value got removed from the `-tls` flag; use `tls=http` to not serve
  TLS. This was confusingly named as you can do `-tls=none,acme` to still
  generate ACME certificates, but `none` implies that nothing is done.

- `goatcounter create` is now `goatcounter db site create`, and some flags got
  changed:

  - `-domain` is now `-vhost`.
  - `-parent` is now `-link`.
  - `-email` is now `-user.email`.
  - `-password` is now `-user.password`.

- The `-port` flag for `goatcounter serve` is renamed to `-public-port`. This
  should clarify that this isn't the *listen* port, but just the port
  GoatCounter is publicly accessible on.

- The `-site` flag got removed from `goatcounter import`; you can now only use
  `-url` to set a GoatCounter site to import to. The automagic API key creation
  was more confusing than anything else.

  You can use `goatcounter db create apitoken` to create an API key from the CLI.

- If you build from source, the build flag to set the version changed from:

      -ldflags="-X main.version=..."

  to:

      -ldflags="-X zgo.at/goatcounter.Version=..."

- The CSV export format was increased to `2`; it now includes the parsed browser
  and system values in addition to the User-Agent header. Version 2.0 will not
  be able to import the older exports from version `1`.


**Other changes**:

- You can read pageviews from logfiles with the `goatcounter import` command;
  you can also send pageviews to goatcounter.com with this (you don’t need to
  self-host it). See `goatcounter help import` and the site code documentation
  for details.

- You can now create multiple users; before there was always a single one. You
  can add users in *Settings → Users*.

  As a consequence, "Site settings" and "User preferences" are now split in to
  two screens. The Settings button in the top-right now displays only site
  settings, and clicking on your email address in the top right displays user
  preferences, which every user can configure to their liking.

- You can now configure what's displayed on the dashboard, in what order, and
  configure some aspects of various "widgets". You can set it in *User
  preferences → Dashboard*. Some settings from the main settings page have moved
  there.

- You can save a default view for the dashboard. Instead of always loading the
  last week by default, you can now configure it to load the last month, or view
  by day, or anything you want really.

- You can choose which data to collect; you can disable collecting any
  User-Agent, location, Referrer information.

- Ability to record state/province/district in addition to country, so it
  records "US-TX" or "NL-NB" instead of "United States" or "Netherlands".

  This option can be disabled separately from recording the country (enabled by
  default) and you can set which countries to record it for (defaults to `US,
  RU, CH`).

  This requires specifying the path to a GeoIP City database, which isn't
  included since it's ~30M.

- There are now stable `count.v*.js` scripts that can use subresource integrity.
  See the integration code for a list and hashes.

- You can use `data-goatcounter-settings` on the `<script>` tag to load the
  settings (requires `count.v2.js` or newer).

- New `goatcounter buffer` command; this allows buffering of pageviews in case
  the backend is down, running migrations, etc. See `goatcounter help buffer`
  for more information.

- The database for PostgreSQL is now created automatically; you no longer need
  to do this manually.

- You can copy settings from a site to other sites in *Settings → Sites*.

- Add `goatcounter db` command; you can now edit and delete sites, users, and
  API keys from the CLI. The `create` and `migrate` commands are now merged in
  to this as subcommands.

- Add a `gcbench` utility for inserting random pageviews in a database; for
  testing and comparing performance. This might be useful for end-users too in
  some cases, for example to see how much performance difference SQLite and
  PostgreSQL will give you, or to test if frobbing with server settings makes a
  difference:

      $ go run ./cmd/gcbench -db sqlite://db/gcbench.sqlite3 -ndays=90 -npaths=100 -nhits=1_000_000
      $ go run ./cmd/gcbench -db postgresql://dbname=gcbench -ndays=90 -npaths=100 -nhits=1_000_000

  Right now it doesn't try super-hard to simulate read-world usage patterns: the
  distribution is always uniform, but it still gives a reasonably accurate
  indication for comparison purposes.

- Many other minor changes and improvements.

- For changes since RC1 see:
  https://github.com/arp242/goatcounter/compare/v2.0.0-rc1...v2.0.0


2020-11-10, v1.4.2
------------------

- Add a "visitor counter" image you can add to your website to display the
  number of visitors, similar to old-style counters back in the ’90s.

- Other than this, it's mostly contains a few minor bugfixes and the like. You
  can see a list of changes in the git log:
  https://github.com/arp242/goatcounter/compare/v1.4.1...v1.4.2


2020-09-04 v1.4.1
-----------------

A few small updates, fixes, and performance enhancements. Nothing major.

You can see a list of changes in the git log:
https://github.com/arp242/goatcounter/compare/v1.4.0...v1.4.1


2020-08-24 v1.4.0
-----------------

- **Change defaults for `-listen`**.

  The default for the `-listen` flag changed from `localhost:8081` to `:443`,
  which is probably a better and less confusing default for most people. There
  is also some more detailed docs available in `goatcounter help listen`.

- Set Cache-Control header for static files.

  The `Cache-Control` header is now set for static files. Since the "cache
  busting" happens based on the goatcounter version it's now recommended to set
  this if you're compiling GoatCounter yourself. See the updated README for
  instructions.

- Add multi-factor auth.

  TOTP-based multi-factor auth is now supported.

- Better export, export API, add import feature.

  You can now import the CSV exports, useful for migrating from self-hosted to
  goatcounter.com or vice versa, or for migrating from other systems. There is a
  web interface and a `goatcounter import` command.

  The export now supports a "pagination cursor", so you can export only rows you
  didn't previously export. This is especially useful with the new export API.
  which should make it easy to sync GoatCounter data with another external
  platform.

  See http://goatcounter.com/api for details on the export API.

- API for sending pageviews.

  Doing that with the regular `/count` is actually quite painful, as you quickly
  run in to ratelimits, need to set specific headers, etc. Adding an API
  endpoint for that makes things much easier.

- API for creating and editing additional sites.

- Some redesigns.

  The "Totals" is now placed below the Pages; I think it makes more sense there.
  The Y-axis for the totals is now also independent. There's also been a quite a
  few restylings.

- Add "text view" mode.

  View your data as a simple table without too much graphics; only the main
  "Pages" overview is implemented for now.

- Make it easier to skip your own views.

  Previously this required adding custom code, but now loading any page with
  `#toggle-goatcounter` added will enable/disable the GoatCounter tracking for
  that browser.

- Can now manage "additional sites" from self-hosted GoatCounter.

  This wasn't possible before for no other reason than laziness on my part 🙃

- public/count.js is now ISC licensed.

  Previously the EUPL applied, which is fairly restrictive and may prevent
  people from including/self-hosting the count.js script.

- Add `goatcounter db` command

  This is mostly useful for writing deploy scripts: `goatcounter db
  schema-sqlite` prints the SQLite schema, `schema-pgsql` prints the PostgreSQL
  schema, and `goatcounter db test` tests if the database exists.

- Session hashes are no longer persisted to the database

  This is kind of an internal change, but session hashes are now stored in
  memory only and never recorded to the database. There's no real reason to
  persistently store this information, and this is a (small) privacy/GDPR
  compliance improvement.


2020-06-01 v1.3.0
-----------------

Note: this release contains quite a few database migrations; they make take a
minute to run (depending on your table size), and you may want to run a `VACUUM`
afterwards.

- Remove email auth, replace `-auth` with `-email-from`.

  As mentioned in the 1.2 release the email authentication is now removed. You
  can still reset the password for old accounts.

  Since the email auth no longer exists the `-auth` parameter no longer makes
  sense. It's now replaced with `-email-from`, which can be set to just an email
  address.

  **Action required**: if you set the email address with `-auth` you'll have to
  change it to `-email-from`.

- Add OS stats, improve accuracy of browser stats.

  GoatCounter now tracks the OS/platform in addition to just the browser, and
  the accuracy of the browser stats should be improved.

  **Action required**: you'll need to populate the `system_stats` table:

      $ goatcounter reindex -table system_stats

  If you want to process all browser stats with the new logic too, then use this
  instead:

      $ goatcounter reindex -table system_stats,browser_stats

- Improve performance.

  Increase performance by quite a bit on large sites and time ranges.

- Remove the per-path scaling.

  Previously GoatCounter would scale the Y-axis different for every path in the
  dashboard, but this was more confusing than helpful. It's now always scaled to
  the maximum of all paths in the selected date range and filter, with a field
  to scale it lower on-demand if desired.

- Add totals overview.

  Add chart with totals for the selected date range and filter.

- Add `goatcounter.url()`, `goatcounter.filter()`.

  Adds two new methods to the `count.js` script so it's easier to use write own
  implementation. In addition the script will now issue a `console.warn()` if a
  request isn't being counted for some reason.


2020-05-18 v1.2.0
-----------------

There are a number of changes in 1.2, and a few which require a bit of action
when updating. Also see: https://www.arp242.net/goatcounter-1.2.html

- Password authentication.

  The email-based authentication has been deprecated in favour of password
  authentication.

  **Action required** Use the interface to set a password (you will get a
  notification about this). Email authentication still works, but will be
  removed in the next release, after which updating the password will be tricky.

- Unique visit tracking.

  GoatCounter now tracks unique visits (without using cookies).

  Technical documentation about the implementation is in
  [doc/sessions.md](doc/sessions.md).

  There are two ways to display the older stats:

  1. Do nothing; meaning that "visits" will be 0 for previous date ranges.

  2. Assign a new 'session' to every hit, so that unique visits will be the same
     as the number of pageviews.

  Doing option 2 is a potentially expensive database operation and not everyone
  may care so it's not done automatically; instructions for doing this are:

  - SQLite (do *not* do this on a running system; as far as I can tell there's
    no good way to get the next sequence ID while incrementing it):

        delete from sessions;
        update hits set session=id, first_visit=1;
        update sqlite_sequence set seq = (select max(session) from hits) where name='sessions';

  - PostgreSQL:

        update hits set session=nextval('sessions_id_seq'), first_visit=1;

  And then run `goatcounter reindex`.

- Improve bot detection.

  The bot detection is now improved; this will be applied to older pageviews in
  the database with a migration, but the cached statistics aren't updated
  automatically (as it can take a while for larger sites). Use the `reindex`
  command to fully update older pageviews (this is entirely optional).

- Track events.

  There is now better support to track events; see the updated documentation on
  the Site Code page for details.

- Better support for campaigns.

  There is now a "campaign parameters" setting; if the URL matches one of these
  parameters it will be set as the referrer (overriding the `Referer` header).

  The default is `utm_campaign, utm_source, ref`.

- Better export.

  The export was a quick feature added in the first version, but didn't scale
  well to larger sites with a lot of pageviews. This now works well for any
  number of pageviews.

- Many small improvements and bug fixes

  It's almost 2 months of work, and there have been many small changes, fixes,
  and improvements. I didn’t keep track of them all 😅


2020-03-27 v1.1.2
-----------------

- Fix small issue with the domain not showing correct in the site code 😅


2020-03-27 v1.1.1
-----------------

- Small bugfix release which fixes some small issues and improves a few small
  documentation issues. List of changes:
  https://github.com/arp242/goatcounter/compare/v1.1.0...v1.1.1

- The biggest change is that the `saas` command no longer works (and is no
  longer documented). It was only ever useful for hosting goatcounter.com, and
  has a number of assumptions and hard-coded values.

  If you're using `saas`, then you can migrate to `serve` by setting a custom
  domain (`sites.cname`) for all the sites. The `serve` command should work
  after that.


2020-03-18 v1.1.0
-----------------

- **Incompatible** Improve CLI UX.

  The entire CLI has been redone; the original wasn't very user-friendly for
  self-hosting. See `goatcounter help` for the full docs, but in brief:

      o Use "goatcounter serve" instead of just "goatcounter".
      o Create new sites with "goatcounter create".
      o Good support for TLS hosting and ACME certificates (see -tls flag).
      o Invert -prod to -dev (i.e. just drop -prod for production services, add -dev for development).
      o -smtp flag is no longer required.
      o -dbconnect                 →  -db
      o -pgsql                     →  -db postgresql://...
      o -staticdomain              →  no longer needed, but if you really want it you can
                                      append to domain: -domain example.com,static.example.com
      o -emailerrors               →  -errors mailto:...
      o goatcounter -migrate       →  goatcounter migrate
      o goatcounter -migrate auto  →  goatcounter serve -automigrate

- **Action required** Show top referrals.

  To populate the ref_stats and size_stats tables for older data, update first
  and then run:

      $ goatcounter reindex -confirm -table ref_stats
      $ goatcounter reindex -confirm -table size_stats

- Charts are displayed in local timezone.

- Add "IgnoreIPs" setting to ignore your own views.

- Link to paths by adding a new domain setting.

- Add configurable data retention.

- Allow configuring the thousands separator.

- Allow filtering pages in the dashboard.

- Improve the integration code.

- Allow sending emails without a relay.

- Add get_query() to count.js to get query parameter.

- Allow viewing the charts by day, instead of only by hour.


2020-01-13 v1.0.0
-----------------

Initial stable release.