# Contributing to the Azure native provider

## Building

### Dependencies

- Go 1.21
- NodeJS 18.X.X or later
- Python 3.10 or later
- .NET 6 or later
- Gradle 8 or later

Please refer to [Contributing to Pulumi](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) for installation
guidance.

### Building locally

Ensure to clone this repository including submodules:

```shell
git clone --recurse-submodules https://github.com/pulumi/pulumi-azure-native
```

Run the following commands to install Go modules, generate all SDKs, and build the provider:

```bash
make
```

Note: When building via make, go workspaces are ignored.

### Running a nodejs example

Navigate to one of the `examples` and run Pulumi:

```bash
make install
cd ./examples/simple
yarn link @pulumi/azure-native
pulumi up
```

## Adding Resource Documentation

Documentation is mostly generated automatically from the specifications. Additional documentation can be added for specific resources by adding markdown files in the [docs/resources](./docs/resources/) folder. Each filename is the combination of the module and resource name as it appears in the registry: `[module]-[Resource].md` e.g. `sql-Server.md`.

Once documentation has been added, run `make generate generate_docs` to re-generate the SDKs and the registry docs. Commit the changes and open a pull request.

## Azure Versions

Key facts about Azure Versions:

- Azure's REST API has many versions of each service which are published per-service on their own schedule.
- Service versions are in the form of an ISO date (e.g. `2020-01-01`) followed by an optional suffix (e.g. `-preview` or `-privatepreview`).
- Not all versions of the service contain all resources. Some services will only publish the parts of their API which have changed since the previous version.
- Some services haven't had a stable (non-suffixed) release for a number of years - and the 'preview' versions are generally considered stable for every-day use.
- New specification versions are sometimes release before the updated service is deployed.

### Terminology

Azure's API specifications are organised into "Resource Providers" (or "Services"), Namespaces and API Versions. We use the term **"API Version"** in this doc to refer to Azure's Resource Provider API Versions.

### Version Sources

1. `spec.json`: the `azure-rest-api-specs` are checked out as a sub-module and include all OpenAPI specifications (for every Resource Provider API Version). The `spec.json` is a simplified report of versions in the spec - each version and the resources within.
2. `active.json` lists the 'active' API versions - the versions of each API which are actually deployed into Azure's data centers. It is generated by running `az provider list`, stripping excess information (which is written to `provider_list.json`) then gets summarised into `active.json`.

### Removing Redundant Versions (Squeezing)

Due the very large number of versions, we only include the latest version, or versions which have incompatible resource shapes when compared with the next version.

This should only be generated at the point of releasing a new major version.

This is currently generated by running `make schema_squeeze`.

## Default Version

To provide simpler discovery and usability, the Pulumi Azure Native provider adds a single 'default' version for each API which is the combination of one or more API versions. The aim is to include all resources, at the latest available version, where possible.

Updates to the version selection is done conservatively within the same major version to avoid breaking changes.

### Configuration Files

Version configuration files are prefixed with the major version they're for (e.g. `v2-`) and live within the `versions/` folder.

1. Config: (e.g. [`v2-config.json`](./versions/v2-config.json)) is a hand-edited file to control how the Spec is updated.
2. Spec: (e.g. [`v2-spec.yaml`](./versions/v2-spec.yaml)) is automatically appended to but can be manually edited.
3. Removed resources/invokes: (e.g. [`v2-removed-resources.json`](./versions/v2-removed-resources.json))
4. Removed versions: (e.g. [`v2-removed.json`](./versions/v2-removed.json))

The default versions file (e.g. [`v2.yaml`](./versions/v2.yaml)) is calculated as a part of the `make schema` target and shouldn't be modified directly.

### Config

The config file performs three functions for each service:

1. Control which versions will be considered for the spec.
2. Assert expected properties of the spec - raising warnings when not met.
3. Document information relevant to the selection of versions and resources.

### Spec

A spec file consists of service names, each specifying:

- `tracking` specifies a single version of the API to fetch all resources from.
- `additions` specified a map of specific resource (or invoke) name to API version to include in the default version.

Both `tracking` and `additions` can be specified together but if the set of resources overlaps (the same resource exists) in both the tracking version and the additions, then the versioning program will fail with an error.

From the Spec file, we calculate the default versions file (e.g. [`v2.yaml`](./versions/v2.yaml)) which contain the fully expanded set of resource versions for each module. These are the files used at the point of generating the schema.

### Creating a new major version default selection

1. Take a copy of previous config.
2. Create an empty spec.
3. Update Makefile: `PROVIDER_VERSION` and dependencies specific to the previous version (e.g. `v2-*` mentions).
4. Calculate removable resources: `make schema_squeeze`.
5. Generate schema: `make schema`.

### Type Compatibility

Supporting every version of every API causes the SDK size to be very large. One method we've used to reduce the required size is to only have a single set of types for each API which are re-used across all versions. However, not all the types are compatible and therefore a type with the same name might be incompatible in the next version. This incompatibility is currently detected but ignored on a case-by-case basis for now.

Ideally, at the point a new incompatible version of the type is added, it should then be created with a unique, stable name. Alternatively, we could create a union of the two possible types

### Default Versions File

The default version is calculated and written to a YAML file which list every resource (or invoke) at a specific API version. This file is read in during the schema and SDK generation. Currently there is a `v1.json` and `v2.json` file. These files should not be edited directly, but should be calculated by the versioner tool using a specific algorithm.

### Additional Reports

- `deprecated.json` is a list of API versions which are older than the versions included in the default version **and** at least 2 years old. These will be removed in the next major version of the provider.
- `pending.json` is a list of new API versions which aren't yet included in the default version. These should be included in the default version at the next major release of the provider.

## New Go SDK

As the size of the Go SDK is close to exceeding the limit of 512Mb, we've created a new SDK which we're publishing in parallel which defined a Go module per Azure namespace rather than a single root Go module. The additional go modules are auto-generated with the required dependencies in [provider/cmd/pulumi-gen-azure-native/main.go](./provider/cmd/pulumi-gen-azure-native/main.go#L312).

This new SDK is published to its own repository at [github.com/pulumi/pulumi-azure-native-sdk](https://github.com/pulumi/pulumi-azure-native-sdk). This is separate partly due to the large number of tags which are created in this new repository per release, but also to remove the need to commit the SDK code into this repository.

## Sub-resources

Some resources which are also settable on a parent resource. For instance, the subnets of a virtual network can be specified inline with the virtual network resource:

```typescript
new network.VirtualNetwork("inline", {
    subnets: [
      { name: "default", addressPrefix: "10.4.1.0/24" },
    ]
})
```

But they can also be specified as stand-alone resources:

```typescript
new network.Subnet("third", { ... })
```

In this case, we call `VirtualNetwork.subnets` a _sub-resource property_.

The choice between inline and stand-alone sub-resource definitions offers flexibility but needs some special handling in the CRUD lifecycle.

On Create: when the user opts for stand-alone representations of the sub-resource, they omit the sub-resource property on the parent. For instance, `new network.VirtualNetwork` will be defined without `subnets`. In this example, however, creating a `VirtualNetwork` without subnets will fail in Azure because it's a required property. Therefore, on Create, we find sub-resource properties that are not set, and set them to their default value in the request payload. This happens in the `azureNativeProvider.setUnsetSubresourcePropertiesToDefaults` method.

On Update: consider the naive implementation. When a virtual network `v` is updated, any stand-alone subnets are not in `v.subnets`, therefore they would be removed on update. To prevent this, we need to retrieve the existing sub-resources and fill them into the parent's sub-resource property. This is done in the `azureNativeProvider.maintainSubResourcePropertiesIfNotSet` method. For example:

```typescript
new network.VirtualNetwork("vnet", { ... })

new network.Subnet("sub1", { ... })
```

When `vnet` is updated, the provider first reads `vnet` from Azure and populates `vnets.subnets` with the subnets from the response. This way, the subnets are simply round-tripped and not removed on update.

On Read: when reading a parent resource, the Azure response will contain the sub-resources. If the user defined them stand-alone, we need to reset them to the empty value to avoid recording them in state. This is done in the `azureNativeProvider.resetUnsetSubResourceProperties` method.

A note on the "default value" mentioned above: it's hard-coded as a an empty array currently. We haven't seen any other type, and it's the only one that really makes sense for sub-resources: they must be a variable number of items.

Relevant PRs:

- #2755
- #2950
- #3054