Indexes

This page details how to create, delete and retrieve indexes on Marqo Cloud. Find your Marqo Cloud API key through heading to the Marqo console, in the API keys tab.

Create index

POST https://api.marqo.ai/api/indexes/{index_name}

Create and index with (optional) settings. This endpoint accepts the application/json content type.

Marqo Cloud creates dedicated infrastructure for each index. Using the create index endpoint, you can specify the type of storage for the index storage_class and the type of inference inference_type. The number of storage instances is defined by number_of_shards, the number of replicas number_of_replicas and the number of Marqo inference nodes by number_of_inferences.

Example

cURLPython

curl -XPOST 'https://api.marqo.ai/api/indexes/my-first-index' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
    "index_defaults": {
        "treat_urls_and_pointers_as_images": false,
        "model": "hf/all_datasets_v4_MiniLM-L6"
    },
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "inference_type": "marqo.CPU.small",
    "storage_class": "marqo.basic",
    "number_of_inferences": 1
}'

import marqo
mq = marqo.Client("https://api.marqo.ai", api_key="XXXXXXXXXXXXXXX")
index_settings = {
    "index_defaults": {
        "treat_urls_and_pointers_as_images": True,
        "model": "hf/all_datasets_v4_MiniLM-L6"
    },
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "inference_type": "marqo.CPU.small",
    "storage_class": "marqo.basic",
    "number_of_inferences": 1
}
mq.create_index("my-first-index", settings_dict=index_settings)

Response: 200 OK

{"acknowledged":true, "shards_acknowledged":true, "index":"my-first-index"}

Path parameters

Name	Type	Description
index_name	String	name of the index

Body Parameters

The settings for the index. The settings are represented as a nested JSON object.

Name	Type	Default value	Description
index_defaults	Dictionary	""	The index defaults object
inference_type	String	marqo.CPU.small	Type of inference for the index. Options are "marqo.CPU.small", "marqo.CPU.large", "marqo.GPU".
storage_class	String	marqo.basic	Type of storage for the index. Options are "marqo.basic", "marqo.balanced", "marqo.performance".
number_of_shards	Integer	1	The number of shards for the index.
number_of_replicas	Integer	0	The number of replicas for the index.
number_of_inferences	Integer	1	The number of inference nodes for the index.

Index Defaults Object

The index_defaults object contains the default settings for the index. The parameters are as follows:

Name	Type	Default value	Description
treat_urls_and_pointers_as_images	Boolean	""	Fetch images from pointers
model	String	hf/all_datasets_v4_MiniLM-L6	The model to use for the index
model_properties	Dictionary	""	The model properties object (for custom models)
normalize_embeddings	Boolean	true	Normalize the embeddings to have unit length
text_preprocessing	Dictionary	""	The text preprocessing object
image_preprocessing	Dictionary	""	The image preprocessing object
ann_parameters	Dictionary	""	The ANN algorithm parameter object

Text Preprocessing Object

The text_preprocessing object contains the specifics of how you want the index to preprocess text. The parameters are as follows:

Name	Type	Default value	Description
split_length	Integer	2	The length of the chunks after splitting by split_method
split_overlap	Integer	0	The length of overlap between adjacent chunks
split_method	String	sentence	The method by which text is chunked (character, word, sentence, or passage)

Image Preprocessing Object

The image_preprocessing object contains the specifics of how you want the index to preprocess images. The parameters are as follows:

Name	Type	Default value	Description
patch_method	String	null	The method by which images are chunked (simple or frcnn)

ANN Algorithm Parameter object

The ann_parameters object contains hyperparameters for the approximate nearest neighbour algorithm used for tensor storage within Marqo. The parameters are as follows:

Name	Type	Default value	Description
space_type	String	cosinesimil	The function used to measure the distance between two points in ANN (l1, l2, linf, or cosinesimil).
parameters	Dict	""	The hyperparameters for the ANN method (which is always hnsw for Marqo).

HNSW Method Parameters Object

parameters can have the following values:

Name	Type	Default value	Description
ef_construction	int	128	The size of the dynamic list used during k-NN graph creation. Higher values lead to a more accurate graph but slower indexing speed. It is recommended to keep this between 2 and 800 (maximum is 4096)
m	int	16	The number of bidirectional links that the plugin creates for each new element. Increasing and decreasing this value can have a large impact on memory consumption. Keep this value between 2 and 100.

Model Properties Object

model_properties is a flexible object that is used to set up models that aren't available in Marqo by default (models available by default are listed here). The structure of model_properties will vary depending on the model.

For Open CLIP models, see here for model_properties format and example usage.

For Generic SBERT models, see here for model_properties format and example usage.

Below is a sample index settings JSON object. When using the Python client, pass this dictionary as the settings_dict parameter for the create_index method.

{
    "index_defaults": {
        "treat_urls_and_pointers_as_images": false,
        "model": "hf/all_datasets_v4_MiniLM-L6",
        "normalize_embeddings": true,
        "text_preprocessing": {
            "split_length": 2,
            "split_overlap": 0,
            "split_method": "sentence"
        },
        "image_preprocessing": {
            "patch_method": null
        },
        "ann_parameters" : {
            "space_type": "cosinesimil",
            "parameters": {
                "ef_construction": 128,
                "m": 16
            }
        }
    },
    "number_of_shards": 3,
    "number_of_replicas": 0,
    "inference_type": "marqo.GPU",
    "storage_class": "marqo.balanced",
    "number_of_inferences": 1
}

Delete index

Delete an index.

Note: This operation cannot be undone, and the deleted index can't be recovered

DELETE /indexes/{index_name}

Example

cURLPython

curl -H 'x-api-key: XXXXXXXXXXXXXXX' -XDELETE https://api.marqo.ai/api/indexes/my-first-index

results = mq.index("my-first-index").delete()

Response: 200 OK

{"acknowledged": true}

List indexes

GET /indexes

List indexes

Example

cURLPython

curl -H 'X-API-KEY: XXXXXXXXXXXXXXX' https://api.marqo.ai/api/indexes

mq.get_indexes()

Response: 200 OK

{
    "results": [
        {
            "docs.count":"0",
            "Created":"2023-08-28T03:18:19.824604",
            "docs.deleted":"0",
            "search.query_total":"0",
            "store.size":"0",
            "data_size":208,
            "index_name":"test",
            "number_of_shards":"1",
            "number_of_replicas":"0",
            "index_status":"READY",
            "number_of_inferences":"1",
            "storage_class":"BASIC",
            "inference_type":"CPU.SMALL",
            "index_defaults":{
                "text_preprocessing":{
                    "split_length":"2",
                    "split_method":"sentence",
                    "split_overlap":"0"
                },
                "ann_parameters": {
                    "parameters":{
                        "ef_construction":"128",
                        "m":"16"
                    },
                    "space_type":"cosinesimil"
                },
                "model":"hf/all_datasets_v4_MiniLM-L6",
                "normalize_embeddings":true,
                "image_preprocessing":{
                    "patch_method":null
                },
                "treat_urls_and_pointers_as_images":false
            },
            "doc_count":0,
            "error_msg":null,
            "endpoint":"https://test-npe2d0-fd0c8262-62e4-47ad-b85b-5ef85cac7c15.dp1.marqo.ai"
        }
    ]
}