# Developing Welcome to the Mapbox Maps SDK for Android project. This document describes every step of the development cycle of our project and make sure you have everything you need to get your pull request (PR) landed. Your contributions are appreciated. ## Building The buildsystem of Mapbox Maps SDK for Android is [Gradle](https://gradle.org/) but there is a [Make](https://www.gnu.org/software/make/manual/make.html) file to make some tasks more convenient and discoverable to execute. Familiarity with `make` is a plus but not a requirement. This document will assume you are familiar with [git](https://git-scm.com/) and will not get into details about `git` commands. When not explicitly specified, commands described are assumed to be executed from the root of the project tree. ### Setting up the repository Clone the git repository: ``` $ git clone git@github.com:mapbox/mapbox-maps-android.git && cd mapbox-maps-android ``` ### Installing the dependencies The only requirement is to install the latest version of [Android Studio](https://developer.android.com/studio). After installed, run the [`sdkmanager`](https://developer.android.com/studio/command-line/sdkmanager) command line or through the GUI interface to make sure the extra requirements to build Android are met. Note: Android Gradle Plugin v7.0+ is used to build the project. That requires JDK 11 or higher installed and `JAVA_HOME` environment variable pointing on it. ``` $ ${ANDROID_HOME}/tools/bin/sdkmanager \ "platform-tools" \ "platforms;android-33" \ "build-tools;30.0.3" \ "build-tools;33.0.0" \ "extras;android;m2repository" \ "patcher;v4" \ "extras;google;m2repository" \ "extras;m2repository;com;android;support;constraint;constraint-layout;1.0.2" \ "ndk;23.2.8568313"\ "cmake;3.22.1" ``` Optional : In case of errors related to ndk version, make sure the ndk path is set in Android Studio. You can set ndk path manually by navigating to `File -> Project Structure -> Android NDK Location` or follow the official guide [Install and configure NDK](https://developer.android.com/studio/projects/install-ndk) ### Opening the Maps SDK This repository is a regular Android Studio project. Open Android Studio and import this repository’s root `mapbox-maps-android` folder. ### Configuring the project Before building, the project needs to be configured. A couple of configurations need to be in place before a successful build and test run can be made. Either export the following values as environement variables or update them inline in the source code: #### SDK Registry token The SDK_REGISTRY_TOKEN is a Mapbox access token, used during compile time, with a scope set to `DOWNLOADS:READ`. This token allows to download all required Mapbox dependencies from a Mapbox Maven instance. The DOWNLOADS:READ scope can be set when creating a new access token on https://account.mapbox.com/. The token configuration can be found in the root `build.gradle.kts` of the project. #### Mapbox Access token The MAPBOX_ACCESS_TOKEN is a mapbox access token, packaged as part of test application, to load Mapbox tiles and resources. This token can be configured on https://account.mapbox.com/ and doesn't require any specific scopes. After first compilation, a gitignored file, is generated in `app/src/main/res/values/developer-config.xml`. ### Initial build To verify if everything has been correctly setup so far, you can run the following make command to compile the project: ``` $ make sdk-build ``` ## Checks Before pushing your code and triggering the CI, it is important that all the checks are passing. Checks are mostly code formatters and code generators. In order to run the checks you need to execute the `check` target. ``` $ make check ``` This target encapsulates validating: - ktlint - lint - kdoc validation - code generation As a convenience, the CI will run the checks again and will let you know if the code is completely formatted and if everything that was supposed to be generated was generated. ### Secret shield This project uses [secret-shield](https://github.com/mapbox/secret-shield) to help block secrets such as access tokens from being exposed. Install `secret-shield` by entering `npm install -g @mapbox/secret-shield`. Install the pre-commit hook by running `scripts/install-pre-commit/install-pre-commit.sh` ### Code formatting We follow [Kotlin Coding Conventions](https://kotlinlang.org/docs/reference/coding-conventions.html) and official [Kotlin Style Guide](https://developer.android.com/kotlin/style-guide). All the projects are configured with `kotlin.code.style=official` in gradle.properties, so you may autoformat the code in Android Studio by default. #### ktlint We use `ktlint` to validate formatting and there's a check on CI preventing PR from merge if `ktlint` fails. Check if format is correct with `ktlint` or `./gradlew ktlint`. Run `ktlintFormat` or `./gradlew ktlintFormat` or `make fix` to automatically reformat everything and fix all the issues. ### Public API changes tracking We use [binary-compatibility-validator](https://github.com/Kotlin/binary-compatibility-validator) and [Metalava](https://android.googlesource.com/platform/tools/metalava/) for tracking binary and source compatibility of the APIs we ship. * When doing code changes that do not imply any changes in public API, no additional actions need to be performed. The bots named `check-api-*` will validate if there's any API breaking changes. * When doing code changes that imply changes in public API, whether it is a new API or adjustments to the existing one, `check-api-*` tasks will start to fail. `make update-api` should be executed manually, the resulting diff in `.api` file should be verified: only signatures you expected to change should be changed. * Commit the resulting `.api` diff along with code changes. ## Code ### Project overview The Mapbox Maps SDK for Android is a multi module gradle project with Android Libraries and Android applications. #### sdk-base This module is the shared dependency across modules and contains primarily interfaces. This module enables us to independently develop other modules against interfaces and not rely on any concrete implementation. By having this strict decoupling, we makes sure we think well about the contracts a module has with other and what public API it exposes to consumers. #### sdk The sdk module is the main module developers integrate in their project. It contains the most important building blocks of the product that couldn't have been decoupled from the underlying c++ implementation at this time. #### module If a module is prefixed with the name `module-` it indicates it's a required module and will be enforced to be available during compilation time. This enforcement is enabled and configured through the annotation processor of [mapbox-base-android](https://github.com/mapbox/mapbox-base-android). #### plugin If a module is prefixed with the name `plugin-` it indicates it's an optional module that will be loaded automatically at runtime if it's part of the compilation classpath. Plugins can be excluded with a gradle exclusion to disable a plugin fully and to save binary size. By default we package the majority of the plugins found in this repository. #### extension If a module is prefxied with the name `extension-` it indicates it's a collection of extension functions that makes it more easy and fluent to integrate with an existing, low-level, API.s The code in those modules use [Kotlin extension functions](https://kotlinlang.org/docs/reference/extensions.html) to add APIs to existing classes. #### app The app module is the internal test and development application. It hosts a variety of different example use-cases to test the code from all the other modules. This module can be referenced when building out other examples. ### Generated code The project contains a large number of code generated files that should not be touched manually. These can be recognised with a top file comment as `// This file is generated.`. Code generators are hosted externally from this repository and generated code requires to be up-to-date when making a pull request. ## Testing Mapbox Maps SDK for Android relies on unit, instrumentation, robo and benchmark test to verify our code correctness. No code is accepted by our CI if any of these tests are failing. When developing a new feature, you are required to write a test and make sure that the test coverage will not go down. A feature will be considered tested if an unit test is provided that verifies the business logic and a instrumentation test is provided that executes the codepath down to the C++ code and back. When fixing a bug, you are also required to provide a test that fails if the fix for the bug is not applied. The test in this case, triggers the faulty behavior and will make sure the bug is not going to be reintroduced and our code regresses. ### Unit tests Unit tests are the tests that validate the business logic of the SDK. Since unit test on Android are executed on the JVM of the development machine, we aren't able to load native libraries and call into our C++ stack. Any of those interactions will need to be either mocked or shadowed. To achieve this we are using a couple of frameworks as [Mockk](https://mockk.io/) and [Robolectric](http://robolectric.org/). Unit tests, for all modules, can be run locally with: ``` $ make unit-tests ``` ### Mutation tests Mutation testing is a technique used to check quality of unit tests and improve them. The main idea is to change the production code and cause the tests to fail. Each mutated version is called a mutant and tests detect and reject mutants by causing the behavior of the original version to differ from the mutant. Read more about [mutation testing](https://en.wikipedia.org/wiki/Mutation_testing). For generating mutants on the JVM we are using [Pitest](http://pitest.org/) library. Because of such testing type takes lot of time you can only run it on necessary module: ``` $ ./gradlew plugin-compass:pitestDebug ``` ### Instrumentation tests Instrumentation tests are integration test that validate the integration with the Android OS and our C++ code. It allows to render the actual Map on an Android device or emulator. It's important that features are accompanied by a instrumentation test as C++ issues will only show at runtime. Instrumentation tests are run on [Firebase TestLab](https://firebase.google.com/docs/test-lab). The CI push a test application and an application with the tests to a physical device on the Firebase device farm. This device is fully allocated to our test at the moment the test is running. The build step that pushes the app to Firebase will give you an URL that can be used for checking the phone logs in case of a crash or test failure. If you don't have access to Firebase, please ask a member of the Maps team. In order to run tests on multiple devices, the last commit in a branch must have `multiple_devices` message. We also run tests on multiple device automatically when the release tag is added. The CircleCI bot will wait for the Firebase test to finish, collect the results, publish them as build artifact and fail/succeed accordingly. Instrumentation tests are divided across 3 modules: - sdk: contains instrumentation tests that validate C++ integration of the SDK - app: contains instrumentation tests that validate specific feature integrations - extension-style-app: contains instrumentation tests for validating the style integration Instrumentation tests can be run locally on a device or emulator with: ``` $ make instrumentation-tests ``` #### Generated Instrumentation tests Next to hand written tests, we generate a large bulk of our tests. Either these are generated as part of a code generator which will generate a test specific to a feature or we generate smoke test that validate integration of a feature within our test application. Generation of the latter, the sanity smoke tests, can be done with: ``` $ make generate-sanity-test ``` The source code of this generator is found under `scripts/sanity-test` and uses [ejs templates](https://ejs.co/). ### Robo tests [Firebase Robo tests](https://firebase.google.com/docs/test-lab/android/robo-ux-test) are integration tests that involves providing invalid, unexpected, or random data as inputs to the test application. This allows us to find bugs that arise in edge-cases situations. Robo tests are executed as part of a CI run and run for 10 minutes. ## Commit messages Commit messages should read as prose. A commit message should tell a story about the what, how and why a change is happening. All commit messages should be pre fixed with the module their are changing, omitting the type of module. ``` [logo] Rename visibility feature to enabled. This is to align plugin vocabulary across the codebase and the plugin system. Enabled is now made part of the Plugin interface and removed from the Logo specific interface. All tests and referneces in the SDK and test applications have been updated to resemble this change. ``` ## Changelog If your change is not trivial, it deserves a changelog entry, so we can tell our users about it on the next release. Please include a changelog entry as shown by the PR template. If no changelog needs to be added, please include `skip changelog` label. ## API documentation Kotlin API documentation can be generated with [Dokka](https://github.com/Kotlin/dokka) in `build/dokka/htmlCollector` with running ``` $ make dokka-html ``` ## Code review Congratulations, you made a pull request. You most certainly added tests, a good PR description and commit message and all the bots are happy about your contribution. The humans working on the project might have a constructive feedback to your code. You will need the approval of at least one team member of the Maps SDK team to land your patch. It is a rare occasion, except for trivial changes, that a patch will get approved with no comments. Please take a moment to appreciate the reviewer feedback. Reviewing pull requests is an arduous and time consuming task. You can make the life of the reviewing easier by making small pull requests. It is easier to land a feature one small self contained functionality at a time. It is also OK to disagree with the reviewer or ask for more details on why a change is needed. Sometimes we will approve with a "nit" or trivial request. One example is, "PR looks good but please fix that typo or let's use a more descriptive name for variable foobar". In this case you can fix your PR, push it again and land it as soon as the bots are green. A reviewer will always trust the developer. It is the developer responsibility to test the feature end-to-end and make sure we can ship a state of the art map rendering engine to our users. And as always, everyone is invited to review code. ## Making releases ### Versioning The SDK, the default extensions and the default plugins use the same versioning, which is defined by the `VERSION_NAME` variable in the root project's gradle.properties. However, the project also allows to use separate versioning for individual subprojects/modules. Subprojects can define the `MODULE_VERSION_NAME` in the subproject-level gradle.properties to overwrite the `VERSION_NAME` defined in the root project. ### Publish new releases The CI is setup to publish new releases when new tag that follows the release tag pattern is created. * To publish a Maps SDK release with all the default plugins/extensions, create a release git tag `v*`. ## Opening tickets Everyone is welcome to open issues or make feature requests on the Mapbox Maps SDK for Android repository. Make sure, in case of an issue, it can be reproduced with the latest version of Mapbox Maps SDK for Android, otherwise it is very likely it was already fixed. Search our issue tracker for similar issues, if you found a problem, maybe someone else also found the same problem and you may contribute to the issue report. Same goes for feature requests, it might be that someone else already requested the feature you want. ### Issue checklist A great bug report will have many of the following: - Detailed steps to reproduce the issue. - The version of the SDK causing the problem. - The hardware in use when the problem was detected (i.e. Samsung Galaxy A71). - The version of the operating system (i.e. Android KitKat). - The style JSON or URL used as the map style. - The camera configuration (coordinates, zoom, bearing and pitch). - When adding a GeoJSON source to the map, the URL for the GeoJSON. - If customizations such as overlays and custom layers are in use and how. - If using runtime styling, a list of changes used to mutate the style. - A symbolicated backtrace in case of a crash. - Screenshot or video demonstrating the problem. - A test reduction, meaning a code example that can reproduce the problem. ## Symbolicating native crashes The Mapbox Maps SDK for Android uses C++ code under the hood. By default, the used C++ binary will be stripped from symbols which means that stacktraces produced are not readable. To symbolicate C++ traces you can use [ndk-stack](https://developer.android.com/ndk/guides/ndk-stack) and a binary that has the symbols attached. ### Download the unstripped binary Next to Mapbox Maven distribution mechanism, Mapbox Android binaries can be downloaded as a zipped archive. This zipped version contains the additional shared objects with debug symbols. Replace in following command the SDK_REGISTRY_TOKEN and VERSION with the appropriate values: ``` $ curl --user mapbox:{SDK_REGISTRY_TOKEN} https://api.mapbox.com/downloads/v2/mobile-maps-android-core/releases/android/{VERSION}/android-core-all.zip --output android-core-all.zip ``` ### Running NDK stack After unzipping the archive you can run ndk-stack on the unstripped native binary and your stacktrace that was saved as a txt file. ``` $ ndk-stack -sym obj/arm64-v8a -dump trace.txt ``` Note that the stacktrace _must_ be formatted properly, otherwise ndk-stack will not be able to symbolicate the stacktrace. Different crash reporting tools often format stacktraces differently, so you might need to reformat the stacktrace. The stacktrace should be formatted as follows: ``` *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** #00 pc acfed libmapbox-maps.so ... #12 pc 51f408 libmapbox-maps.so (BuildId: 331a309c6d4e90a8) ``` e.g. start with asterisks line, followed by stack entries. Stack entries should start with at least one space, then a hash sign with a stack entry number, then `pc`, then the address (_without_ the `0x` prefix), then the library name. ## Working with snapshots The Mapbox Maps SDK for Android publishes snapshot releases to our API downloads infrastructure. These snapshots can be used in downstream project to verify bug fixes or preview new features. However, we do not recommend using snapshot releases in production environment. ### Automatic snapshot releases By default, each commit to the main branch of the repository will trigger a snapshot release for the sdk, as well as all the plugins/extensions/modules, with an unique version name. The name of the snapshot is defined in the `VERSION_NAME` in the project's `gradle.properties` file, with the commit hash inserted before the `-SNAPSHOT`. An example of snapshot for the sdk is `com.mapbox.maps:android:10.0.0-a1d95e7a-SNAPSHOT`. ### Manually triggered snapshot releases The Mapbox Maps SDK for Android also features manually triggered snapshot releases from a developing branch. If a last commit in a developing branch has `publish_android_snapshot` in a title or description, the `build-modules-and-instrumentation-tests` CI step will create a snapshot release. The name of the snapshot release would be: ${LAST_RELEASE_VERSION_BEFORE_THIS_COMMIT}-${BRANCH_NAME}-SNAPSHOT for example: `10.0.0-rc.1-peng-commit-message-based-snapshot-SNAPSHOT` ### Use snapshot releases in your app Snapshot releases are distributed from a different Downloads API endpoint, you will need to add this API endpoint to your project's `build.gradle` as follows: ```gradle allprojects { repositories { maven { url 'https://api.mapbox.com/downloads/v2/snapshots/maven' authentication { basic(BasicAuthentication) } credentials { // Do not change the username below. // This should always be `mapbox` (not your username). username = 'mapbox' // Use the secret token you stored in gradle.properties as the password password = project.properties['MAPBOX_DOWNLOADS_TOKEN'] ?: "" } } } } ``` Where the password is the secret token and can be configured following the [installation guide](https://docs.mapbox.com/android/beta/maps/guides/install/#configure-credentials) And then update the Mapbox Maps SDK's version name to the snapshot version in your app's `build.gradle`. ## Working with traces Mapbox Maps SDK for Android allows enabling additional Mapbox traces that could be of help troubleshooting performance issues. Working with traces is possible only when app is debuggable: ```xml ... android:debuggable="true" ... ``` If the app is not debuggable and `MapboxTracing` will be used - runtime exception will be thrown. ### Enabling Mapbox traces There are 2 types of traces that could be enabled: - Native rendering engine traces could be enabled with `MapboxTracing.enableCore()`. Those traces include native render calls, style loading, requesting tiles and more. - Android platform traces could be enabled with `MapboxTracing.enablePlatform()`. Those traces include Android render thread calls such as preparing / destroying surface. It is also possible to enable all the traces with `MapboxTracing.enableAll()`. Tracing could be disabled with `MapboxTracing.disableAll()`. ### Recording traces using Android studio There are several ways of capturing traces for Android which are described in [official documentation](https://developer.android.com/topic/performance/tracing). As a reference we will describe the steps of recording traces using Android Studio Profiler: 1. Run your app in profile mode e.g. by clicking following icon in Android Studio. image 2. When app is run, click on CPU in Profiler window and select `System Trace` radio-button. When you are ready to actually record the traces, hit the `Record` button. image 3. When you are done with recording, press `Stop` button, wait Android Studio to parse your traces and hit `Export system trace recording` to save trace recording locally. image ### Reading Mapbox traces using Perfetto After we exported trace recording we could use [Perfetto](https://ui.perfetto.dev/) to investigate the Mapbox traces. 1. Open [Perfetto](https://ui.perfetto.dev/) and drag-and-drop your local trace recording file there. 2. When it is loaded - navigate to your application package and un-squash it. 3. Mapbox native render engine traces added by `MapboxTracing.enableCore()` could be found at the bottom of your app process in separate groups. Those traces start with `mbx:`. image 4. Mapbox platform specific traces will be located inside other Android threads and not in dedicated groups, so you could use global search by `mbx:` to locate them. image ### Other ways to record traces There are several other ways how to record traces. #### From ui.perfetto.dev over WebUSB image Enable Atrace events (other optional probes can be enabled if needed). image After `stop recording` is pressed, trace will be visible in the UI. #### Directly from the device (supported on Google Pixel devices) #### Using command line utility (/Library/Android/sdk/platform-tools/systrace) `python systrace.py -a "com.mapbox.maps.testapp" -o trace.html -b 20000`