Some heading
" 'An action was performed. Now do you want to do the next thing.
' ) % next_thing_url messages.success(request, msg, extra_tags="safe") ``` Note the `safe` tag which allow HTML to be included in the message. ## Avoid model forms [Django's model-forms](https://docs.djangoproject.com/en/2.1/topics/forms/modelforms/) are a useful crutch for rapidly building web projects. However for a late-stage Django project (like ours), they are best avoided apart from the most degenerate, simple scenarios. Why? Because they conflate validation and persistence logic which, over time, leads to hard-to-maintain code. As soon as you need to add more sophisticated actions (than a simple DB write) to a successful form submission, model-forms are the wrong choice. Once you start overriding `.save()`, you're on the path to maintenance hell. Future maintainers will thank you if you ensure forms are only responsible for validating dictionaries of data, nothing more, single-responsibility principle and all that. Instead, use plain subclasses of `form.Form` and Django's `fields_for_model` function to extract the form fields you need. Handle the `form_valid` scenario in the view with a single call into the domain layer to perform all necessary actions. One advantage of this is you can pluck form fields off several models to build a sophisticated form, which allows views to be kept simple and thin (handling multiple forms in the same view should be avoided). Example: ```python from django import forms from myproject.someapp import models class SampleForm(forms.Form): # Grab fields from two different models user_fields = forms.fields_for_model(models.User) profile_fields = forms.fields_for_model(models.Profile) name = user_fields["name"] age = profile_fields["age"] ``` The same principle applies to other ORM-constructs, like DRF's model serializers, which trade-off good structure and long-term maintainability for short-term development speed. This is an important step in extricating a project from Django's tight grip, moving towards treating Django as a library rather than framework. ## Avoid field validators [Django's field validators](https://docs.djangoproject.com/en/4.0/ref/validators/) should not be used. Like model forms, they can be useful in an early stage project, but are better avoided in a late-stage project like Kraken. This is enforced by flake8 rule `K306`. Why? They make it look like a field will always have its value validated when in practice it will not. Example: ```python from django.core import validators from django.db import models class SomeModel(models.Model): constrained_number = models.FloatField( default=0.5, validators=[ validators.MinValueValidator(0.0), validators.MaxValueValidator(1.0), ], ) ``` This looks like a field that cannot be set to values below 0 or above 1. But the validation exists in code rather than the database, so it is possible to store values outside this range. Django only runs field validators when calling `.full_clean()` on a Model. This is done when using Django Admin to build forms or when using model forms. Neither of these are the recommended method of creating forms in Kraken. They are also not pulled into the form fields when we use `forms.fields_for_model`. As such, field validators are not run automatically, while looking like they are. Validation should instead be performed explicitly when validating input. [Constraints](#integrity-at-the-database) should be used to provide data integrity in the database. This makes it clear when validation is and isn't being run. ## Avoid multiple domain calls from an interface component An interface component, like a view class/function or Celery task, should only make one call into the domain layer (i.e. the packages in your codebase where application logic lives). Recall: the job of any interface component is this: 1. Translate requests from the language of the interface (i.e. HTTP requests or serialized Celery task payloads) into domain objects (e.g. `Account` instances) 2. Call into the domain layer of your application to fetch some query results or perform an action. 3. Translate any returned values (or exceptions) from the domain layer back into the language of the interface (e.g. into a 200 HTTP response for a successful query, or a 503 HTTP response if an action wasn't possible). Note, this step only applies to interfaces components that can _respond_ in some way - this doesn't apply to fire-and-forget Celery tasks where there's no results back-end. This convention mandates that step 2 involves a single call into the domain layer to keep the interface easy-to-maintain and to avoid leaking application logic into the interface layer. So avoid this kind of thing: ```python class SomeView(generic.FormView): def form_valid(self, form): account = form.cleaned_data["account"] payment = form.cleaned_data["payment"] result = some_domain_module.do_something(account, payment) if result: some_other_domain_module.do_something_else(account) else: form.add_error(None, "Couldn't do something") return self.form_invalid(form) some_logging_module.log_event(account, payment) return shortcuts.redirect("success") ``` where the view class is making multiple calls into the domain layer. Instead, encapsulate the domain functionality that the interface requires in a single domain call: ```python class SomeView(generic.FormView): def form_valid(self, form): try: some_domain_module.do_something( account=form.cleaned_data["account"], payment=form.cleaned_data["payment"], ) except some_domain_module.UnableToDoSomething as e: # Here we assume the exception message is suitable for end-users. form.add_error(None, str(e)) return self.form_invalid(form) return shortcuts.redirect("success") ``` Note the use of domain-specific exceptions to handle failure conditions. Since fire-and-forget Celery tasks can't respond, their implementation should be simple: just loading the appropriate domain objects and making a single call into the domain layer. Something like: ```python @app.task(queue=settings.SOME_QUEUE) def perform_some_action(*, foo_id, bar_id, **kwargs): foo = Foo.objects.get(id=foo_id) bar = Bar.objects.get(id=bar_id) some_domain_module.perform_action(foo, bar) ``` ## Check permissions in `dispatch` method If using class-based views, perform all access-control and pre-condition checks in the `dispatch` method (or the `get` method if a read-only view). Because: - This method is expected to return a `HttpResponse` instance and so is a natural place to return a 403 if the requesting user does not have permission to access the requested object. - This method is called for _all_ HTTP methods and avoids possible security holes if permissions are only checked in, say, the `get` method. In particular, avoid checking permissions in other commonly subclassed methods like `get_context_data` or `get_form_kwargs`. When checking pre-conditions, avoid adding business logic to [the `dispatch` method](https://docs.djangoproject.com/en/4.0/ref/class-based-views/base/#django.views.generic.base.View.dispatch). Instead encapsulate the check as a function in the domain layer and call that from `dispatch` - something like this: ```python from django.views import generic from django import http from django.http import response from project.interfaces import acl from project.domain.frobs import checks class ActOnFrob(generic.FormView): def dispatch( self, request: http.HttpRequest, *args, **kwargs ) -> response.HttpResponseBase: # Check if the request user is allowed to mutate the resource. if not acl.can_user_administer_frob(request.user, self.frob): return http.HttpResponseForbidden("...") # Check the pre-conditions for the resource to be mutated. if not checks.can_frob_be_mutated(self.frob): return http.HttpResponseForbidden("...") return super().dispatch(request, *args, **kwargs) ``` ## Load resources in `setup` method If using class-based views, perform all model loading in [the `setup` method](https://docs.djangoproject.com/en/4.0/ref/class-based-views/base/#django.views.generic.base.View.setup) (or the `get` method if a read-only view). This method should perform any initialization prior to dispatching. Putting model loading in this view allows the retrieved models to be used by any other method on the view class. ```python from django.views import generic from django import shortcuts, http from project.data import models class ActOnFrob(generic.FormView): def setup(self, request: http.HttpRequest, *args, **kwargs) -> None: super().setup(request, *args, **kwargs) # Load the resource specified in the URL. self.frob = shortcuts.get_object_or_404(models.Frob, id=kwargs["id"]) ``` ## DRF serializers Serializers provided by Django-REST-Framework are useful, not just for writing REST APIs. They can be used anywhere you want to clean and validate a nested dictionary of data. Here are some conventions for writing effective serializers. Be liberal in what is accepted in valid input. - Allow optional fields to be omitted from the payload, or have `null` or `""` passed as their value. - Convert inputs into normal forms (by e.g. stripping whitespace or upper-casing). In contrast, be conservative in what is returned in the `validated_data` dict. Ensure `validated_data` has a consistent data structure no matter what valid input is used. Don't make calling code worry about whether a particular key _exists_ in the dict. In practice, this means: - Optional fields where `None` is not a valid value have a default value set in the serializer field declaration. - If optional string-based fields have `allow_null=True`, then convert any `None` values to the field default. To that end, use this snippet at the end of your `validate` method to ensure there are no missing keys in the `validated_data` dict and that `None` is only included as a value in `validated_data` when it's a field's default. ```py def validate(self, data): ... # Ensure validated data includes all fields and None is only used as a value when it's # the default value for a field. for field_name, field in self.fields.items(): if field_name not in data: data[field_name] = field.initial elif data[field_name] is None and data[field_name] != field.initial: data[field_name] = field.initial return data ``` Misc: - Ensure validation error messages end with a period. ## Out-of-band form errors In a `FormView` sometimes an error occurs _after_ the initial payload has been validated. For instance, a payment partner's API might respond with an error when registering a new bankcard even though the submitted bankcard details appeared to be valid. To handle this situation in a `FormView` subclass, use the following pattern: ```py from django.views import generic from . import forms class CreateSomething(generic.FormView): form_class = forms.Something ... def form_valid(self, form): try: thing = domain.create_a_thing( foo=form.cleaned_data['foo'], bar=form.cleaned_data['bar'], ) except domain.UnableToCreateThing as e: # Handle *anticipated* exceptions; things we know might go wrong but # can't do much about (e.g. vendor errors). Here we assume the exception # message is suitable for end-users but often it needs some # adjusting. form.add_error(None, str(e)) return self.form_invalid(form) except Exception as e # Optional handling of *unanticipated* errors. When this happens, we want to send # details of the error to Sentry so such errors can be investigated # and fixed. It's optional as we could omit this block and let the # exception percolate up to generate a 500 response which Sentry # will capture automatically. It's a bit friendlier to the user to # show them some custom copy rather than a generic 500 response. # Ensure your logger has the Raven handler so the exception is sent # to Sentry. logger.exception("Unable to create a thing") # Depending on context, it might not be appropriate to # show the exception message to the end-user. If possible, offer # some guidance on what the end-user should do next (e.g. contact # support, try again, etc). Ensure the tone is apologetic. msg = ( "Really sorry, we weren't able to do the thing as an unanticipated " "error occurred. Error message: %s" ) % e form.add_error(None, msg) return self.form_invalid(form) else: # Handle successful creation... message.success(self.request, "Created the thing!") return http.HttpResponseFound(...) ``` ## Use database constraints for enforcing data integrity Don't rely on the Python application for data integrity. Instead, use database constraints to prevent the storage of corrupt data. Application-level validations aimed at the user experience (such as form validation) are acceptable as long as they aren't used as substitutes for data integrity, but as an extra. Some Django `QuerySet` methods such as `bulk_update()`, `bulk_create()`, and `update()` don't call the model's `save()` method. If a model has validation logic living in `save()`, it becomes susceptible to data inconsistencies. The same risks apply to functions [encapsulating model mutations](#encapsulate-model-mutation) and to other application-level validators, such as [model field validators](#field-validators). When asserting data integrity is important, don't do this: ```python def save(self, *args, **kwargs) -> None: if self.field_a is None and self.field_b is None: raise exceptions.ValidationError( "At least one of field_a or field_b must have a value." ) return super().save(*args, **kwargs) ``` Assert it at the database level instead: ```python class MyModel(models.Model): # model fields... class Meta: constraints = [ models.CheckConstraint( check=~Q(field_a__isnull=True, field_b__isnull=True), name="at_least_one_of_field_a_or_field_b_not_null", ) ] ``` ## Use path converters in URLs instead of regular expressions Where possible we prefer Django path converters to old style URLs. The format is `