HTTP API¶
MARV provides an HTTP API for integration with other services. There is a dedicated API endpoint that can be used to run complex queries on the MARV database. For performing actions like tagging or commenting there are no dedicated endpoints at this time, and we recommend using the frontend API endpoints documented below for the time being.
To work with the API on the command line it is handy to have curl and jq.
sudo apt-get install curl jq
Set an environment variable for the API.
export MARV_API=http://localhost:8000/marv/api
Auth¶
For some API calls you need to be authenticated, let’s get a token, and set handy environment vars.
echo -n "Enter username: "; read -s MARV_USER && echo && \
echo -n "Enter password: "; read -s MARV_PASS && echo && \
TOKEN=$(curl -s -X POST -H "Content-Type: application/json" \
-d '{"username": "'$MARV_USER'", "password": "'$MARV_PASS'"}' \
$MARV_API/auth | jq -r .access_token) && \
echo $TOKEN
Query¶
Database queries can be performed by posting a corresponding request to
v1/rpcs
. The endpoint expects a json encoded object with an rpcs
key and a list of actions to run. An empty request and its response would look
like:
curl -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"rpcs": []}' \
$MARV_API/v1/rpcs
{
"data": {}
}
Query actions are represented by dictionaries with a key query
. The simplest
query payload {"model": "dataset"}
asks for all entries of a single entity,
in this case dataset. The valid entities are dataset
, file
, comment
,
tag
, user
, and group
. The complete HTTP request to get all datasets
looks like:
curl -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"rpcs": [{"query": {"model": "dataset"}}]}' \
$MARV_API/v1/rpcs
{
"data": {
"dataset": [
{
"id": 1,
"collection": "bags",
"discarded": 0,
"name": "laser",
"status": 0,
"time_added": 1568903426121,
"timestamp": 1521328251000,
"setid": "dbgjsro7vfazyrj2pavoouk3vq"
},
// and many more ...
}
}
}
Often not all attributes are required on the client side. To restrict the fields
in the response the query payload can contain an attrs
object. Its keys
correspond to the database fields that should be returned, and its values are
always set to true
for real database columns.
{
"model": "dataset",
"attrs": {
"name": true,
"setid": true,
},
}
{
"data": {
"dataset": [
{
"name": "laser",
"setid": "dbgjsro7vfazyrj2pavoouk3vq",
"id": 1
},
{
"name": "map",
"setid": "ob56ua4ztunqeh7iw6ywg2ntum",
"id": 2
},
{
"name": "navsatfix",
"setid": "g5nnjxaejquqrph6dnsencj2mi",
"id": 3
},
{
"name": "odom",
"setid": "gl5zdundbwwsr2qhduous5324y",
"id": 4
},
{
"name": "test",
"setid": "o6yufr25xqhx7nz7tljdr4ogaa",
"id": 5
}
]
}
}
The results can be filtered using the filters
key. Its value is a list of
filter objects to apply. Each filter object consists at least of an operator
op
. Most operators are applied to a field name
and work with a value
value
. The following query payload gets the dataset where the field id
equals 3:
{
"model": "dataset",
"filters": [
{"op": "eq", "name": "id", "value": 3},
],
}
{
"data": {
"dataset": [
{
"id": 3,
"name": "navsatfix",
// other attrs
}
]
}
}
The filters list can contain multiple filter objects and by default the results have to match all. The following operators work on fields and values:
between
: field is in the range of value ([low, high])endswith
: string field ends with valueeq
: field equal to value, for comparison tonull
useis
(see below)gt
: field greater than valuegte
: field greater equal valuein
: field is contained in list of values [val1, val2, …]is
: field is valueisnot
: field is not valuelt
: field is lower than valuelte
: field is lower equal valuene
: field is not equal valuenotbetween
: field is not in the range of value ([low, high])notin
: field is not one of the values in value [val1, val2, …]startswith
: string field starts with valuesubstring
: value is a substring of field
There are three additional operators that implement boolean logic to combine filters. Each of the operators works without a field name and operates on value only:
not
: Negates val which is another filter object itselfand
: “ANDs” all filter objects in a value listor
: “ORs” all filter objects in a value list
Filters can also operate on related tables by using a dotted path notation in
field names. To search for datasets tagged important
use:
{
"model": "dataset",
"filters": [
{"op": "eq", "name": "tags.value", "value": "important"},
],
}
{
"data": {
"dataset": [
{
"id": 1,
"name": "laser",
// other attrs
}
]
}
}
A query can also embed related models into the response by setting its
corresponding attrs
key similarly to regular model fields. Set the value to
true
to embed complete objects or set the value to an object that shall be
used as attrs
for the related model:
{
"model": "dataset",
"attrs": {
"name": true,
"tags": {"value": true}, # populate tag objects only with value field
},
"filters": [
{"op": "eq", "name": "tags.value", "value": "important"},
],
}
{
"data": {
"dataset": [
{
"name": "laser",
"id": 1,
"tags": [
1, 4
]
}
],
"tag": [
{
"value": "autonomous",
"id": 1
}
{
"value": "important",
"id": 4
}
]
}
}
The API supports sorting and paging. To sort the results use the option
order
with a value of ["fieldname" "ORDER"]
with ORDER being either
ASC
or DESC
. Paging is achieved with the limit
and offset
integer options. An example query payload would look like:
{
"model": "dataset",
"order": ["setid", "ASC"], # sort ascending by setid
"limit": 5, # limit number of results to 5
"offset": 10, # skip the first 10 results
}
Listing (deprecated)¶
MARV knows two kind of ids for dataset.
setid; a random 128 bit integer, base32 encoded without padding chars, e.g.
h27zmwsdzcnmu6kqncwdhhvrva
id; id of the dataset within the database, e.g.
42
While the set id is unique for all times and across sites, for many interactions it is more efficient to use the database id.
Fetch id of all datasets:
curl $MARV_API/collection |jq '.listing.widget.data.rows[] | .id'
And likewise for setid:
curl $MARV_API/collection |jq '.listing.widget.data.rows[] | .setid'
Filter¶
curl -G \
--data-urlencode \
'filter={"name": {"op": "substring", "val": "leica"}}' \
$MARV_API/collection \
|jq '.listing.widget.data.rows[] | .setid'
curl -G \
--data-urlencode \
'filter={"tags": {"op": "all", "val": ["bar", "foo"]}}' \
$MARV_API/collection \
|jq '.listing.widget.data.rows[] | .setid'
curl -G \
--data-urlencode \
'filter={"tags": {"op": "any", "val": ["bar", "foo"]}}' \
$MARV_API/collection \
|jq '.listing.widget.data.rows[] | .setid'
List of dataset files¶
curl -X POST \
-H "Content-Type: application/json" \
-d "[42]" \
$MARV_API/file-list
output:
{
"paths": [
"/scanroot/scan_odom_map_test.bag",
],
"urls": [
"dataset/h27zmwsdzcnmu6kqncwdhhvrva/0",
]
}
Comment¶
curl -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"42": {"add": ["comment 1", "comment 2"]}}' \
$MARV_API/comment
output:
{}
Delete¶
curl -X DELETE \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "[1,2]" \
$MARV_API/dataset
output:
{}
Deletion is idempotent.
Tag¶
curl -X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d '{"bags": {"add": {"foo": [42]}, "remove": {"bar": [17,42]}}}' \
$MARV_API/tag
{}
Tagging is idempotent, missing tags are created, unused tags are not automatically cleaned up (see Maintenance).