Handling nested serializer validations in Django Rest Framework

I understand that the title of this post is a little confusing. Recently, while working on the Projects API in Weblate, I came across an interesting issue. The Projects API in Weblate allowed you to get an attribute called source_language. Every project has only one source_language and in the API, it was a read-only property.

{
  "name": "master_locales",
  "slug": "master_locales",
  "web": "https://example.site",
  "source_language": {
    "code": "en",
    "name": "English",
    "direction": "ltr",
    "web_url": "http:/example.site/languages/en/",
    "url": "http://example.site/api/languages/en/"
  },
  "web_url": "http://example.site/projects/master_locales/",
  "url": "http://example.site/api/projects/master_locales/",
  "components_list_url": "http://example.site/api/projects/master_locales/components/",
  "repository_url": "http://example.site/api/projects/master_locales/repository/",
  "statistics_url": "http://example.site/api/projects/master_locales/statistics/",
  "changes_list_url": "http://example.site/api/projects/master_locales/changes/",
  "languages_url": "http://example.site/api/projects/master_locales/languages/"
}

As you can see, unlike the other relational fields, it's not a HyperLinkedIdentityField. It uses the nested language serializer to show all the attributes of the source_language.

Now, previously, when a project was created via API, a default language was always assigned to the project and there was no way to define the source_language while creating the project via API.

Problem?

Doing GET on Language Serializer when sending POST on Project Serializer

So we needed to add the feature to define the source_language of the project when we send a POST request to the Project API. And also edit the project via API to update the source_language. So, to use the same serializer, the request body for the POST request would look something like this:

{
  "name": "master_locales",
  "slug": "master_locales",
  "web": "https://example.site",
  "source_language": {
    "code": "ru",
    "name": "Russian",
    "direction": "ltr",
  }
}

Now, in general, we would have a python serializer like this:

class LanguageSerializer(serializers.ModelSerializer):
    web_url = AbsoluteURLField(source="get_absolute_url", read_only=True)
    class Meta:
        model = Language
        fields = ("code", "name", "direction", "web_url", "url")
        extra_kwargs = {
            "url": {"view_name": "api:language-detail", "lookup_field": "code"}
        }


class ProjectSerializer(serializers.ModelSerializer):
    source_language = LanguageSerializer(required=False)
    # ...
    # Other parts of the serializer

The problem with having code like this is, when the ProjectSerializer gets a request like shown above and tries to validate the data in the request, it also validates the LanguageSerializer part. The LanguageSerializer part whenever it gets data, it will automatically try to validate the data. The code property of Language model has a unique constraint. So, when LanguageSerializer tries to validate

{
    "code": "ru",
    "name": "Russian",
    "direction": "ltr",
}

it will throw an error "This field must be unique" for code property in case a language with codename ru already exists in the database.

Solution

So there are few steps to get this done.

Remove validators from code field

extra_kwargs = {
    "url": {"view_name": "api:language-detail", "lookup_field": "code"},
    "code": {"validators": []},
}

Add "code": {"validators": []} to the extra_kwargs to remove the validator from the LanguageSerializer on every data request it receives.

Add manual validation for code field

Removing validator will also remove the validation while doing POST request. Now, the LanguageSerializer in Weblate specifically doesn't support POST, but in any case, you would manually need to add a validation function to the LanguageSerializer so if someone checks for validity before adding language, it throws an error. To do that, add a function validate_code like this:

def validate_code(self, value):
    check_query = Language.objects.filter(code=value)
    if check_query.exists() and not (
        isinstance(self.parent, ProjectSerializer)
        and self.field_name == "source_language"
    ):
        raise serializers.ValidationError(
            "Language with this Language code already exists."
        )
    if not check_query.exists():
        raise serializers.ValidationError(
            "Language with this language code was not found."
        )
    return value

Note: The name of the function must be validate_{field_name} when you are trying to validate a field based on how DRF handles validation.

Overwrite create() in ProjectSerializer

Finally, we would want to overwrite the create() function of ProjectSerializer to:

• Validate source_language data using the above validation to check if the language with that code exists
• Modify source_language key of the validated_data to have the Language model object rather than the dictionary, so it can be used to create a project with the foreign key.
• Lastly, create a project with the new validata_data

The code would look something like this:

def create(self, validated_data):
    source_language_validated = validated_data.get("source_language")
    if source_language_validated:
        validated_data["source_language"] = Language.objects.get(
            code=source_language_validated.get("code")
        )
    project = Project.objects.create(**validated_data)
    return project

And now, if you create a project, using the source_language key, you can define the source language for the project while using the Project API. There might be several other ways to go about it. But this is one of the ways I found works.

Also, this feature is now live in Weblate 4.* versions which allows you to define the source_language via the API.

Creating Custom Whoosh Plugin

Recently, while trying to work on a query parser feature in Weblate, I came across this search engine library called Whoosh. It provides certain nice features like indexing of text, parsing of search queries, scoring algorithms, etc. One good thing about this library is most of these features are customizable and extensible.

Now, the feature I was trying to implement is an exact search query. An exact search query would behave in a way such that the backend would search for an exact match of any query text provided to it instead of the normal substring search. Whoosh provides a plugin for regex, which can be accessed via whoosh.qparser.RegexPlugin(). So we can technically go about writing a regex to do the exact match. But a regex search will have worse performance than a simple string comparison.

So, one of the ways of doing a new kind of query parsing is creating a custom whoosh plugin. And that's what this blog is going to be about.

Simple Whoosh Plugin

In some cases, you will probably not need a complicated plugin, but just want to extend the feature of an existing plugin to match a different kind of query. For example, let's say you want to extend the ability of SingleQuotePlugin to parse queries wrapped in either single-quotes or double-quotes.

class QuotePlugin(whoosh.qparser.SingleQuotePlugin):
    """Single and double quotes to specify a term."""
    expr = r"(^|(?<=\W))['\"](?P<text>.*?)['\"](?=\s|\]|[)}]|$)"

In the above example, QuotePlugin extends the already existing SingleQuotePlugin class. It just overrides the expression to parse the query. The expression, mentioned in the variable expr is usually a regex expression with ?P<text> part denoting the TermQuery. A TermQuery is the final term/terms searched for in the database. So in the above regex, we say to parse any query such that the TermQuery is wrapped in between single-quotes or double-quotes.

Query Class

A query class is the class, whose instance the final parsed term will be. Unless otherwise mentioned, it's usually <Term>. So if we want our plugin to parse the query and show it as an instance of a custom class, we need to define a custom query class.

class Exact(whoosh.query.Term):
    """Class for queries with exact operator."""

    pass

So, as you can say, we can just have a simple class just extending whoosh.query.Term so that while checking the parsed terms, we can get is as an instance of Exact. That will help us differentiate the query from a normal Term instance.

Custom Whoosh Plugin

After writing the query class, we will need to write the custom plugin class.

class ExactPlugin(whoosh.qparser.TaggingPlugin):
    """Exact match plugin with quotes to specify an exact term."""

    class ExactNode(whoosh.qparser.syntax.TextNode):
        qclass = Exact

        def r(self):
            return "Exact %r" % self.text

    expr = r"\=(^|(?<=\W))(['\"]?)(?P<text>.*?)\2(?=\s|\]|[)}]|$)"
    nodetype = ExactNode

In the above example, unlike the simple case, we extend TaggingPlugin instead of any other pre-defined plugin. Most of the pre-defined plugins in whoosh also extend TaggingPlugin. So it is a good fit as a parent class.

Then, we create a ExactNode class. This we will assign to the node type for the custom plugin. A node type class basically defines the query class to be used in this custom plugin, along with various representations and properties of the parsed node. qclass will have the query class created before to denote the Exact instance to the final parsed term.

Apart from that, we have the expr which contains the regex just like in the simple example to parse the query term.

Finally...

After creating the custom plugin, you can:

• add this plugin to the list of plugins defined in the whoosh query parser class
• use the query class to make an isinstance() check when making database queries
• check for the node type in the different nodes used by the parser

Configuring Jest with React and Babel

module.exports = api => {
  const isTest = api.env('test');
  if (isTest) {
    return {
      presets: [
        [
          '@babel/preset-env',
          {
            modules: false,
          },
        ],
        '@babel/preset-react',
      ],
      plugins: [
        "@babel/plugin-transform-modules-commonjs",
      ],
    }
  } else {
    return {
      presets: [
        [
          '@babel/preset-env',
          {
            modules: false,
          },
        ],
        '@babel/preset-react',
      ],
    }
  }
};

module.exports = {
  moduleNameMapper: {
    '^~/(.*)$': '<rootDir>/path/to/jsRoot/$1'
  }
}

Making cleaner imports with Webpack and Babel

What's a ShadowDOM?

I first came to know about Shadow DOM from my curiosity about how browsers implement tags like <video> with all the controllers or <input> which changes based on the type attribute. We can never see any HTML code for these implementation and yet they are shown. How? That is when I stumbled upon this blog by Dimitri Glazkov which explains beautifully the concept of shadow DOM encapsulation used by browsers to implement such tags.

However, none of the browsers allowed developers to write their own custom shadow DOM (though google chrome had a v0 version implemented). I stumbled upon shadow DOM again while looking at an issue in jQuery to fix. So now, since 2018, most of the browsers have started supporting shadow DOM APIs and hence jQuery needed to implement support for that too.

https://caniuse.com/#feat=shadowdomv1

So, what is this shadow DOM and why do we even use it?

What's a DOM?

W3C specification describes it as "a method of combining multiple DOM trees into one hierarchy and how these trees interact with each other within a document, thus enabling better composition of the DOM".

Now, to understand that, we need to understand what a DOM is. DOM or Document Object Model is a tree-like structure containing the different elements (or tags) and strings of text that are shown by the markup language (like HTML, XML, etc.).

So, let's say we have a HTML code something like this:

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Title</title>
  </head>
  <body>
      <div>
        <h1>This is header</h1>
        <p>This is a  
          <a href="https://www.saptaks.blog">
            link
          </a>
        </p>
      </div>
  </body>
</html>

So, visually you can show the DOM structure as something like:

Shadow DOM

Now, shadow DOM allows us to create separate hidden DOM trees that are attached to the elements of a regular DOM tree. This allows you to implement functional encapsulation, i.e. someone parsing the regular DOM tree and applying styling to the regular DOM tree doesn't know or affect the properties and functionalities of the shadow DOM tree. Hence you use the shadow DOM without knowing the intricate details of how the DOM is implemented. This is important, because this follows the basic ideas of Object Oriented Programming.

The shadow DOM tree starts with a shadow root and can then have any regular DOM element attached underneath it.

Let's see an example:

HTML

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Title</title>
  </head>
  <body>
      <div id="shadowHost">
      </div>
  </body>
</html>

JavaScript

const shadowHost = document.getElementById('shadowHost');
const shadowRoot = shadowHost.attachShadow({mode: 'open'});
shadowRoot.innerHTML = '<h1>Hello Shadow DOM</h1>';

So, this will create a shadow DOM. Visually you can represent this as:

So, as you can see, there are few different parts in a shadow DOM apart from it being just another DOM.

• Shadow tree: The DOM tree inside the shadow DOM.
• Shadow boundary: the place where the shadow DOM ends, and the regular DOM begins.
• Shadow root: The root node of the shadow tree.
• Shadow host: The regular DOM node that the shadow DOM is attached to.
• Shadow child: The tree below the shadow root node.

The shadow DOM cannot be attached to few different elements as mentioned in the spec. Some of the reasons are:

• The different form tags such as <input>, <textarea>, etc or any other html tag for which the browser implements its own Shadow DOM
• Elements like <img> or <br> or <hr> which are usually self enclosing tags and don't usually contain a child node.

Also if you see in the code there is a "{mode: open}". The mode determines whether you can edit the style of the shadow DOM from outside the shadow DOM or not.

Why do we need Shadow DOM anyways?

There are few different scenarios where you might want to use shadow DOM. The most important functionality of shadow DOM is the implementation of the concept of encapsulation. So basically, someone using the shadow DOM host doesn't need to care the style and implementation of the DOM inside. So few use-cases would be following:

• The browser implements Shadow DOM for various different tags such as <video>, <audio>, <input>, <select>, etc.
• You can make your own custom Shadow DOM when you need to create an element that you want to not be modified by the styling of remaining DOM.
• You can also use Shadow DOM, when you want to create a separation of a particular DOM element from the main DOM element.

So is Shadow DOM very shadow-ey? Well maybe, since it stays hidden from the main DOM traversal. But at the same time it is often very useful because of its encapsulation properties.

What to expect from GSoC?

iTerm + zsh + tmux for your MacOS

brew install zsh

sh -c "$(curl -fsSL https://raw.githubusercontent.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"

brew install tmux

chsh -s `which zsh`

# set shell
set -g default-shell /bin/zsh

killall tmux