Ever wanted to add a field that does not exist in our out-of-the-box models and you thought, “We don’t need a smart model to do this. I just need to look at these certain words.” ? For this use-case we provide a predictor you can control through the predictor settings.

In the example below we describe the implementation for the field email_coming_from with the tag options no_reply and info, so they will match emails sent from no-reply@contract.fit and info@contract.fit respectively.

Fields and tags need to be added in a specified rule as a regex in the predictor settings. On the swagger page of your environment you can find the endpoint /predictor_settings/{scope}. The scope is the inbox/project for which you would want this predictor to run. Inside key_value_pairs::rule_config you can specify per tag_field which regexes you want to match for a given field_name. Note that the field_name and tag options have to match exactly (case sensitive). Otherwise the prediction will be empty.

{
  "key_value_pairs": {
    "rule_config": {
      "email_coming_from": {
        "no_reply": {
          "rules": [
            {
              "confidence": 97,
              "+rule": ["L:no-reply@contract.fit"]
            }
          ]
        },
        "info": {
          "rules": [
            {
              "confidence": 97,
              "+rule": ["L:info@contract.fit"]                
            }
          ]
        }
      }
    }
  }
}

Per tag field (level of no_reply and info) you can specify a list of rules with a confidence per rule. The rule that matched with the highest confidence will be presented in the prediction. If we see no-reply@contract.fit somewhere in the uploaded document, a prediction with confidence 97 for the tag_option no_reply will be returned.

“+rule”

The simplest version of a rule is specified in +rule. The value with this key is a list of strings prefixed with L: or D:. This list will be concatenated into one regex.

L stands for literal and prefixes a normal regex. Note that the regex should be double escaped, so the regex for digit becomes \\d instead of \d.

D stands definition which prefixes a variable.

Specifying the search space

The search space is the area of the document our rules apply to, ie. where they try to find a matching result. Coming back to our first example above, the rules here applied to the whole document. This is too broad for this use-case (“is the sender no-reply@contract.fit or info@contract.fit ?”). In this case we only want to search inside the email field which specifies the sender.

For this we can use the where_to_search option.

This function contains 4 options, all are optional:

1. preprocess_text: This is a boolean specifying if the text needs to be cleaned (= removal of noise such as unexpected characters eg. multiple dashes).

2. search_in (for email): This field specifies which part of the email we look in. There are 5 options which can be combined. When left empty we will look in all these options:

   1. email_from

   2. email_to

   3. email_subject

   4. email_body

   5. attachment

3. limits: This field specifies which limits we apply to our search_space. See below.

4. Granularity

Search_in

In the case of our example, we only want to look in the email_from, so we add the following:

{
  "confidence": 97,
  "+rule": ["L:no-reply@contract.fit"]
  "where_to_search": {"search_in": ["email_from"]}
}

Limits

You can apply different limits to the search space of your query.

These are the 4 options to limits:

1. document_types: list of document types, these can be combined.

2. pages: list of slices to specify which pages you want to search in.

3. lines: list of slices to specify which lines you want to search in.

4. characters: list of slices to specify which characters you want to search in.

For option 2, 3 and 4 we use the method of a Python Slice. They require a specified start and stop value to be defined.

[start, stop]    # items from start through stop-1
[-start:-stop]   # items from start (counting from end) through stop-1 (counting from end)
[start]          # items from start through end (only allowed for the last slice)
[-start]         # items frpm start (counting from end) through end (only allowed for the last slice)

Eg. Imagine that in our example we only want to look in the first 10 and last 20 characters of the email_from. In this case we would change our rule as follows:

{
  "confidence": 97,
  "+rule": ["L:no-reply@contract.fit"],
  "where_to_search": {
    "search_in": ["email_from"],
    "limits": {
      "characters": [[0,10], [-20]]
    }
  }
}

Granularity

The granularity allows you to specify in which blocks of text we want to search.

The options are:

1. full (default if nothing is specified)

2. page

3. sentence

4. paragraph

5. line

Logical additions to “+rule”

In some cases matching just regexes - which already have and, or and not operators built-in - does not accomodate your rules anymore. To satisfy your needs we give you the ability to combine different regexes with higher level and-or-not operators and allowed to extend them with different low level operators like the +rule. A not operator you can specify by replacing the + in front of your operator by a -.

This is the full list of operators with a dash between the higher level and low level operators:

• +and: list of higher or lower level operators which should all match

• -and: not version of and

• +or: list of higher or lower level operators for which one should match

• -or: not version of or

• +rule: list of strings starting with L: or D: . When evaluating they will be appended into one regex.

• -rule: not version of rule

• +lemma: list of strings which are contained in the lemma: more info here.

• -lemma: not version of lemma

Let’s say we want to match the emails coming from no-reply@contract.fit if and only if in the body we don’t see a phone number. Our rule would now look like this:

{
  "confidence": 97,
  "+and": [
    {
      "+rule": ["L:no-reply@contract.fit"],
      "where_to_search": {
        "search_in": ["email_from"],
        "limits": [[0,10], [-20]]}
    },
    {
      "-rule": ["L:\\+32\\d{9}"],
      "where_to_search": {
        "search_in": ["email_body"]
      }
    }
  ]
}

Variables

When writing rules you might come across the case where you have added the same regex in many rules. With variables you can avoid this problem. Specify them on the highest level in a dictionary and use them with the prefix D: in your rules. Re-using the telephone number, our full dictionary would now look like this:

{
  "no_reply": {
    "variables": {
      "var1": ["L:\\+32\\d{9}"]
    },
    "rules": [
      {
        "confidence": 97,
        "+and": [
          {
            "+rule": ["L:no-reply@contract.fit"],
            "where_to_search": {
              "search_in": ["email_from"],
              "limits": [[0,10], [-20]]}
          },
          {
            "-rule": ["D:var1"],
            "where_to_search": {
              "search_in": ["email_body"]
            }
          }
        ]
      }
    ]
  }
}

Lemma

This is a list of strings that are contained in the granularity. Here we don’t look at the original text, but the lemmatised version of the text.

FAQ

"rules": [
  {
      "confidence": 97,                     
      "+rule": ["L:no-reply@contract.fit"]     
      "where_to_search":
        {
          "search_in": ["email_subject"]
        }
  }
]

 How can I test if my regex has the right format?

In the front-end you can go to a certain page and select the text view (see attached image). This way you can copy the text you are searching in. On this site you can test the regex you created: https://regex101.com/. PS: Don’t forget to set the language to python on the left hand side of the screen and remove the double escaping.

{
  "email_coming_from": {
    "other": {
      "rules": [
        {
          "confidence": 96,
          "+rule": [
            "L:."
          ]
        }
      ]
    }
  }
}