apiary.apib

FORMAT: 1A
HOST: https://research.democracy.ovh

# argument-analysis-research

A collection of code used for the research project at ARG-tech aiming for argument analysis and natural language processing

##### About

The purpose of this project is to provide tools and services used in our argument analysis research.

Parts of this research project will happen inside [Colaboratory](https://colab.research.google.com/drive/1iGL_J01I-SAtw2HG8uoJMLgYhYqMzzAK) as well. So this repository is a dependency there and should be treated accordingly.

## Argument Analysis API

Some parts of this repo get combined into a simple JSON API for use in other places (like Colaboratory).
This currently includes segmenting, keyword extraction as well as ADW based segment comparison.

### API Types

#### Document

Documents are the core ingestion format for the argument analysis API.
A document stores content (text) and depending of its state processed results like a list of segments or keywords.

```
{
    "hash": "string", // a sha256 over the content field. usually autogenerated by the API
    "content": "string", // the documents content. this is the single required field for every document
    "segments": [] // array of segments as defined below
}
```

## Segmenting [/argument/segment]

### Single Input [POST /argument/segment]

You may supply a string of text to analyze and get a document containing segements.

+ Request (application/json)

            {
                "input": "Hello darkness my old friend"
            }

+ Response 200 (application/json)

    + Body

            {
                "content": "Hello darkness my old friend",
                "segments": [
                    {
                        "text": "Hello darkness my old friend"
                    }
                ]
            }
            
### Bulk Input [POST /argument/segment/bulk]

You may supply a list of strings to analyze and get a list of documents each containing segements.

+ Request (application/json)

            {
                "inputs": [
                    "Hello darkness my old friend",
                    "Foo"
                ]
            }

+ Response 200 (application/json)

    + Body

            {
                "documents": [
                    {
                        "content": "Hello darkness my old friend",
                        "segments": [
                            {
                                "text": "Hello darkness my old friend"
                            }
                        ]
                    },
                    {
                        "content": "Foo",
                        "segments": [
                            {
                                "text": "Foo"
                            }
                        ]
                    }
                ]
            }
        
## Keyword Extraction [/argument/keyword]

For Keyword extraction we currently use RAKE. The endpoint adds found keywords to the passed in segment.

**TODO:** This endpoint requires some love as it is barely used at the moment and should take documents instead of basic input.

### Get a segment with keywords from basic input [POST /argument/keyword]

+ Request (application/json)

            {
                "input": "Hello darkness my old friend"
            }

+ Response 200 (application/json)

    + Body

            {
                "text": "Hello darkness my old friend",
                "keywords": [
                    {
                      "key": "old friend",
                      "value": 4
                    },
                    {
                      "key": "darkness",
                      "value": 1
                    }
                ]  
            }

## ADW API [/argument/adw]

This almost primitive service wraps around [ADW](https://github.com/pilehvar/ADW) to expose a very simple API for checking text similarity of a json request.

### Get distance for two strings [POST /argument/adw]

+ Request (application/json)

            {
                "text1": "We should stop eating meat",
                "text2": "Don't buy so much stuff"
            }

+ Response 200 (application/json)

    + Body

            {
                "result": 0.507
            }

## Linker API [/argument/link]

The Linker API provides a service for checking a set of documents for links based on their segments.
It takes a pre segmented list of documents to find and return found links.

The service itself is async and acts like a database with background processing.
It provides an easy interface that manages multiple linker instances in the background.
If a new instance is created (by sending a linking POST without specifying any linkerID), the new ID gets returned by the service.
To add documents to this instance set the linkerID as shown below.

The linker compares all its stored documents based on their segments doing a two way rating through the rating service defined per linker instance.

### Insert one or many Documents into the Linker [POST /argument/link]

This request either sends to or creates a new linker which stores and handles the provided documents.
The `id` value is optional. If none is set, a new linker is being created and the output will contain the new ID for sending further documents or querying the results.
If `id` is set, the existing linker found under its value will get used.

The `rater` value is optional and defaults to `https://research.democracy.ovh/argument/adw`.
The `threshold` value is the minimal weight two segments need between eachother to get stored as links. It defaults to `0`.

Setting `rater` and `threshold` after the linker was created will have no impact on any existing linkers and get ignored if `id` is set.

+ Request (application/json)

            {
                "id": "",
                "rater": "",
                "threshold": 0.25,
                "documents": [
                    {
                        "content": "Keep encouraging regulations reducing pollutants into our atmosphere.",
                        "segments": [
                            {
                                "text": "Keep encouraging regulations reducing pollutants into our atmosphere."
                            }
                        ]
                    },
                    {
                        "content": "The eruption of two supervolcanoes would take care of it. At least for a while.",
                        "segments": [
                            {
                                "text": "The eruption of two supervolcanoes would take care of it."
                            },
                            {
                                "text": "At least for a while."
                            }
                        ]
                    }
                ]
            }

+ Response 200 (application/json)

    + Body

            {"id": "8c2ab258-60dc-4416-9648-edd1efbf3a84"}
            
+ Response 500 (application/json)

    + Body
        
            {
                "error": "error message or object"
            }
            
### List all existing linkers [GET /argument/linkers]

This request returns all the linkers persisted in the linker database.

+ Response 200

    + Body
    
            {
                "linkers": [
                    {
                        "metadata": {
                            "id": "c6b5dc9b-48fa-40d1-9d30-a134c045e851",
                            "rater": "https://research.democracy.ovh/argument/adw",
                            "threshold": 0.45
                        }
                    }
                ]
            }
            
### List all links found so far [GET /argument/links{?id}{?docs}]

This request queries all documents and links for the provided linker `id`.
To not query all documents, the `docs` parameter can be set to false and the linker won't return them.

+ Parameters

    + id (string) - The linker id to query links from

    + docs (bool, optional) - If set to false, no documents will be returned, just links.
        + Default: `true`

+ Response 200 (application/json)

    + Body
        
            {
                "documents": [
                    {
                        "hash": "0d53f89084673365c99b79e8e0b0da359d38c341f6886e14dc3aa824080acc10",
                        "content": "If every human being on the planet dropped dead right now, the worst effects of climate change would still occur. There's no stopping it at this point. Anyone who tries to tell you differently is deluded.",
                        "segments": [
                            {
                                "text": "If every human being on the planet dropped dead right now, the worst effects of climate change would still occur."
                            },
                            {
                                "text": "There's no stopping it at this point."
                            },
                            {
                                "text": "Anyone who tries to tell you differently is deluded."
                            }
                        ]
                    },
                    {
                        "hash": "af9eca75be519629fef2b63164714ba16cc03a2eee4e7966b2a4d61af563ea4b",
                        "content": "There is nothing we can do as individuals. No amount of energy saving light bulbs will remove CO2 from the atmosphere. There is nothing we can do as nations. Even human extinction would have no effect. The carbon is there. It's not going anywhere. There is no hope for reversing it. Now I'm sure lots of people who read this will consider me a pessimistic asshole with nothing of value to add to the conversation. Those people are delusional or misinformed and their hopes are based on nothing.",
                        "segments": [
                            {
                                "text": "There is nothing we can do as individuals."
                            },
                            {
                                "text": "No amount of energy saving light bulbs will remove CO2 from the atmosphere."
                            },
                            {
                                "text": "There is nothing we can do as nations."
                            },
                            {
                                "text": "Even human extinction would have no effect."
                            }
                        ]
                    },
                ],
                "links": [
                    {
                        "source": {
                            "doc": "af9eca75be519629fef2b63164714ba16cc03a2eee4e7966b2a4d61af563ea4b",
                            "seg": 3
                        },
                        "target": {
                            "doc": "0d53f89084673365c99b79e8e0b0da359d38c341f6886e14dc3aa824080acc10",
                            "seg": 0
                        },
                        "weight": 0.58415395
                    },
                    {
                        "source": {
                            "doc": "0d53f89084673365c99b79e8e0b0da359d38c341f6886e14dc3aa824080acc10",
                            "seg": 0
                        },
                        "target": {
                            "doc": "af9eca75be519629fef2b63164714ba16cc03a2eee4e7966b2a4d61af563ea4b",
                            "seg": 3
                        },
                        "weight": 0.58415395
                    }
                ]
            }