How would you define a Matcher based on a regular expression?

[rsoika]

Hi,
I have a question about using the matcher feature of spaCy.
Is it possible to define a matcher based on a regular expression? I want to extract IBAN/BIC from a invoice text.
For example a regular expression for IBAN looks like this:

\b[A-Z]{2}[0-9]{2}(?:[ ]?[0-9]{4}){4}(?!(?:[ ]?[0-9]){3})(?:[ ]?[0-9]{1,2})?\b    

Background: currently I do NER and train my model on invoices giving the position of an IBAN/BIC. Even as I have trained multiple thousands of training data the recognition of IBAN/BIC is not so good as I hoped which may be the missing text context in an invoice document.
Now I think about if it would be better to detect IBAN/BIC by patterns using the matcher feature? I have seen pattern formats like the LIKE_EMAIL and I guess this is also some regular expression. But how can I define my own matcher pattern?

And can I combine the matcher with my NER? Or do I need two models for that?

Can someone give me a hint how to get started?

[polm]

For IDs with regular structure like IBAN it makes sense to use regular expressions instead of NER models. If you have cases where you have things that match your regex but may not be IDs for some reason an NER model can help but I think IBAN numbers are too long for that to be an issue.

For a guide to writing Matcher patterns, which are not just regular expressions, see the rule-based matching documentation, which also covers using matchers with NER.

I'm not really familiar with IBANs, but based on your regex above, it looks like you may have variation in the number of tokens. In that case it can sometimes make sense to use a regex directly against the text of the doc and use Doc.char_span, see here.

[rsoika]

Thanks for your answer. What did you mean with "If you have cases where you have things that match your regex but may not be IDs" - can you give me an example.
Another example illustrating my problem are invoice that can have 2 VAT-IDs - one of the customer and one of the supplier. But I am interested only in the suppliers ID. Is this the case where the combination between matcher and NER comes into the play?

[polm]

Thanks for your answer. What did you mean with "If you have cases where you have things that match your regex but may not be IDs" - can you give me an example.

American phone numbers without area codes look like 555-5555. You can match that with a regular expression, but in the sentence "My favorite 80s song is 867-5309", the number is a phone number but also a song title, so you may not want to match it. Similarly you could imagine something like "my refrigerator is model XYZ999-2000", where the number is not a phone number at all. As mentioned above, I think IBANs are too long and specific for this to be an issue, but it is a problem to be aware of.

Another example illustrating my problem are invoice that can have 2 VAT-IDs - one of the customer and one of the supplier. But I am interested only in the suppliers ID. Is this the case where the combination between matcher and NER comes into the play?

This is semantic role labelling and it's a challenging problem. Our usual suggestion is to use NER labels (whether rule-based or from a model) and then to use a rule based component to disambiguate the types and see how far it gets you. If you have complete sentences you can use dependency information, if you don't have complete sentences you can use nearby words. Search Discussions for "semantic role labelling" and you should find some examples.

[rsoika]

I already use spacy to detect IBAN/BIC with NER in invoice documents (no sentences only numbers/words). It works not to bad. But sometimes in new invoices the results are not so good. So my idea/hope is that I can improve this using the entityruler. The documentation says "...can significantly improve accuracy in some cases" - which gives me hope ;-)