📍Search API

This section explains different REST APIs which enable the users to insert or search data within Engines.

List Private Engines

GET /v1/engine/user/list

This endpoint allows the users to list all the Private Engines associated with a private API key.

x-api-key: user private api key

Request Example:

curl --location --request GET \
'https://api.sepana.io/v1/engine/user/list' \
--header 'x-api-key: <replace-with-your-api-key>'
Private Engines: Response
[
    {
        "engine_id": "<engine-id-1>",
        "engine_name": "<engine-name-1>",
        "namespace": "4439953c-885a-4cb0-b895-b2e0c186f073",
        "network": "",
        "total_records": 186,
        "total_size": 23178,
        "created_at": {
            "val": "11/24/2022 12:29:15 PM ",
            "_spec_type": "<class 'datetime.datetime'>"
        }
    },
    {
        "engine_id": "<engine-id-2>",
        "engine_name": "<engine-name-2>",
        "namespace": "4439953c-885a-4cb0-b895-b2e0c186f073",
        "network": "",
        "total_records": 410,
        "total_size": 48688,
        "created_at": {
            "val": "11/25/2022 01:11:23 PM ",
            "_spec_type": "<class 'datetime.datetime'>"
        }
    }
]

List Public Engines

GET /v1/engine/public/list

This endpoint allows the users to list all the Public Engines existing within Sepana via any private API key belonging to them.

Header:

x-api-key: user private api key

Request Example:

curl --location --request GET \
'https://api.sepana.io/v1/engine/public/list' \
--header 'x-api-key: <replace-with-your-api-key>'
Public Engines: Response
[
    {
        "engine_id": "<engine-id-1>",
        "engine_name": "documentation",
        "namespace": "d9601b27-e91b-4987-a664-ec9af8769e63",
        "network": "",
        "total_records": 3,
        "total_size": 7059,
        "created_at": {
            "val": "12/01/2022 12:11:23 PM ",
            "_spec_type": "<class 'datetime.datetime'>"
        }
    },
    {
        "engine_id": "<engine-id-2>",
        "engine_name": "new_test",
        "namespace": "e5d35741-0794-4f45-b312-b08c84524a20",
        "network": "",
        "total_records": 996,
        "total_size": 114913,
        "created_at": {
            "val": "11/24/2022 10:21:06 AM ",
            "_spec_type": "<class 'datetime.datetime'>"
        }
    }
]

Insert Data

POST /v1/engine/insert_data

This endpoint is responsible for ingesting the data within a Sepana Engine. The data is provided in the form of JSON documents. Upon successful call to this endpoint, an internal data indexing job is created which is responsible for persisting the data. A maximum of 500 records can be inserted at a time.

Note: While ingesting the data, users can send unique _id along with each record. This acts as an identifier for that row in the dataset. If not specified explicitly by the user, Sepana automatically assigns one to each row.

Example below:

[
    {
        "_index": "<engine-id-1>",
        "_type": "_doc",
        "_id": "RfEFzoQBxqv6fmT099iM",
        "_source": {
            "Stages": "Series A",
            "Date": "2021-08-31T00:00:00.000Z",
            "Investors": "Accel,Bertelsmann India Investments"
       }
     },
     {
        "_index": "<engine-id-1>",
        "_type": "_doc",
        "_id": "RvELzoQBxqv6fmT0odgB",
        "_source": {
            "Stages": "Series B",
            "Date": "2021-08-31T00:00:00.000Z",
            "Investors": "Siopp Investments"
         }
     }    
]

RfEFzoQBxqv6fmT099iM and RvELzoQBxqv6fmT0odgB are system generated identifiers (_id) for the two records present within <engine-id-1>. If the user wants to update the data within these two data points in future, they can simply create an ingestion request with new data while adding _id identifier to it. This will overwrite the older data present with _id.

Header:

x-api-key: api key with a minimum of Read-Write access

Parameters:

docs: data documents in JSON format
engine_id: The id of the Engine where data is to be inserted

e.g.

{
  "engine_id": "string",
  "docs": [
    {.....},
    .......,
    {.....}
  ]
}

Request Example:

curl --location --request POST 'https://api.sepana.io/v1/engine/insert_data' \
    --header 'x-api-key: <replace-with-your-api-key>' \
    --header 'Content-Type: application/json' \
    --data-raw '{
        "engine_id": "<replace-with-your-engine-id>",
        "docs": [
            {
                "_id": 1,
                "age": 30,
                "email": "abc@gmail.com",
                "name": "ABC Sepana"
            },
            {
                "_id": 2,
                "age": 34,
                "email": "def@gmail.ai",
                "name": " DEF Sepana",
                "nested": {
                    "field_1": {
                        "field_2": ["trying", "nested", "list", "sample", "together"]
                    }
                },
                "sports": ["soccer", "baseball", "cricket"] 
            }
        ]
    }'
Insert Data: Response
{
    "job_id": "<job-id>",
    "engine_id": "<engine-id>",
    "request_timestamp": "12/01/2022, 12:14:46",
    "status": "queued"
}

Delete Data

DELETE /v1/engine/data/delete

This endpoint is responsible for deleting data from a Sepana Engine. The users can delete the data providing delete_query and engine_id as inputs. Users can initially perform search with a query and delete those records using the same query as delete_query. To perform deletion, the user must own an API key for an Engine with a minimum of Admin access.

Header:

x-api-key: api key with a minimum of Admin access

Parameters:

engine_id: The id of the Engine whose data is to be deleted
delete_query: query to selective delete data points from an Engine

Eg.:
{
  "engine_id": "string",
  "delete_query": {
      .....
  }
}

Request Example:

Delete the records from Engine "a88d58e6-15bf-409c-a09d-84603894c56e" where user_name is Louis Pearson

curl --location --request DELETE 'https://api.sepana.io/v1/engine/data/delete' \
--header 'x-api-key:  <replace-with-your-api-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "engine_id": "<replace-with-your-engine-id>",
    "delete_query": {
        "query": {
            "match": {
                "user_name": "Louis Pearson"
            }
        }
    }
}'
Delete Data: Response
{
    "detail": "Deleted 5 records from <engine_id>",
    "query": {
        "query": {
            "match": {
                "user_name": "Louis Pearson"
            }
        }
    }
}

Engine Remapping

POST /v1/engine/remap_fields

This endpoint allows users to reconfigure Engines for which they have admin access. It allows users to redefine and update the field data types. Upon calling this endpoint, an asynchronous pipeline is created which updates the data mapping of the engines based on the past configurations. The example mentioned updates the Engine mapping from previous endpoint (/engine/insert_data). Changes field "age" from "float" to "integer" and adds another field "expense" with data type "integer"

Header:

x-api-key: api key with a minimum of Admin access

Parameters:

properties: new data schema of the Engine
engine_id: The id of the engine whose mapping is to be updated

e.g.

{
  "engine_id": "string",
  "properties": {}
}

Request Example:

curl --location --request POST 'https://api.sepana.io/v1/engine/remap_fields' \
--header 'x-api-key: <replace-with-your-api-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "engine_id": "<replace-with-your-engine-id>",
    "properties": {
                "age": {
                    "type": "integer"
                },
                "email": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "name": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "nested": {
                    "properties": {
                        "field_1": {
                            "properties": {
                                "field_2": {
                                    "type": "text",
                                    "fields": {
                                        "keyword": {
                                            "type": "keyword",
                                            "ignore_above": 512
                                        }
                                    }
                                }
                            }
                        }
                    }
                },
                "sports": {
                    "type": "text",
                    "fields": {
                        "keyword": {
                            "type": "keyword",
                            "ignore_above": 256
                        }
                    }
                },
                "expense": {
                    "type": "integer"
                }
            }
}
'
Engine Remmaping: Response
{
    "job_id": "<job-id>",
    "engine_id": "<engine-id>",
    "mapping": {
       ....
    },
    "request_timestamp": "12/01/2022, 13:17:13",
    "status": "queued"
}

Job Status

GET /v1/job/status/{job_id}

Both Data Ingestion and Engine Reconfiguration are long running tasks. The system, therefore, creates asynchronous pipelines to execute those tasks. Every task is assigned a Job ID. You can track and update Job IDs using this endpoint.

Header:

x-api-key: private api key

Parameters:

job_id: id of the task

Request Example:

curl --location --request \
GET 'https://api.sepana.io/v1/job/status/<replace-with-your-job-id>' \
--header 'x-api-key: <replace-with-your-api-key>'
Job Status: Response
{
    "job_id": "job_id",
    "engine_id": "<associated-job-id>",
    "request_timestamp": "10/03/2022, 01:41:51",
    "status": "<status>",
    "message": "<success/error message>",
    "completion_timestamp": "10/02/2022, 20:11:55"
}

POST /v1/search

This endpoint allows users to search through data in a single or multiple Engines. There are various filters and options to choose from.

Header:

x-api-key: api key with a minimum of Read-Only access to all the Engines searched

Parameters:

query (Data)
    engine_ids: Optional[List]
    indices_boost: Optional[List]
    query: Optional[Dict]
    aggs: Optional[Dict]
    fields: Optional[List]
    timeout: str = "2s"
    sort: Optional[List]
    collapse: Optional[Dict]
    post_filter: Optional[Dict]
    highlight: Optional[Dict]
    rescore: Optional[Dict]
    search_after: Optional[List]
    track_total_hits: bool = True
    response_fields: Optional[List]
    page: Optional[int] = 0  # from
    size: Optional[int] = 20

Request Examples:

curl --location --request POST 'https://api.sepana.io/v1/search' \
--header 'x-api-key: <replace-with-your-api-key>' \
--header 'Content-Type: application/json' \
--data-raw '{
    "engine_ids": [
        "<engine-id-1>",
        "<engine-id-2>"
    ],
    "query": {
        "query_string": {
            "query": "india"
        }
    },
    "size": 2,
    "page": 0
}
'
Search: Response
{
    "hits": {
        "total": {
            "value": 4,
            "relation": "eq"
        },
        "max_score": 0.2876821,
        "hits": [
            {
                "_index": "<engine-id-1>",
                "_type": "_doc",
                "_id": "RfEFzoQBxqv6fmT099iM",
                "_score": 0.2876821,
                "_source": {
                    "Stages": "Series A",
                    "Date": "2021-08-31T00:00:00.000Z",
                    "Investors": "Accel,Bertelsmann India Investments",
                    "Description": "Decentralized personal loans marketplace",
                    "Founder": "Mayank Tewari,Prerit Srivastava",
                    "Website": "https://www.skeps.com/",
                    "Project": "Skeps",
                    "Category": "CeFi",
                    "Sub-categories": "Lending/Borrowing",
                    "Announcement": "",
                    "Fundraising Round": "Skeps - Series A",
                    "Record ID": "recufCOPL5Y7csrNj",
                    "Name": "Skeps",
                    "category_name": "CeFi",
                    "subcategory_name": "Lending/Borrowing",
                    "fundraising_round": "Skeps - Series A",
                    "Investors_name": "Accel, Bertelsmann India Investments",
                    "founders_name": "Mayank Tewari, Prerit Srivastava",
                    "Created": "2021-09-06T08: 22:21.000Z",
                    "Amount": "9500000",
                    "Investors copy": "",
                    "Valuation": ""
                }
            },
            {
                "_index": "<engine-id-2>",
                "_type": "_doc",
                "_id": "RvELzoQBxqv6fmT0odgB",
                "_score": 0.13353139,
                "_source": {
                    "indicator": {
                        "id": "NY.GDP.MKTP.CD",
                        "value": "GDP (current US$)"
                    },
                    "country": {
                        "id": "IN",
                        "value": "India"
                    },
                    "countryiso3code": "IND",
                    "date": "2021",
                    "value": 3173397590816.91,
                    "unit": "",
                    "obs_status": "",
                    "decimal": 0
                }
            }
        ]
    },
    "aggregations": null
}

Last updated