Using Marqo Cloud

Exploring console.marqo.ai

Login

Login to your marqo cloud console account. To register for Marqo cloud, apply here.

Indexes

You'll be presented with a table of your indexes and on a separate section you can see your Marqo Endpoint Details.

Each Marqo Endpoint URL is unique.

Below is how the overview page looks:

To view graphs on Data size, Count of documents, Total search count, and Primary shards, select the Monitor tab for the index page.

API Keys

Your API Keys are used to access indexes through the marqo client.

• Up to four API keys are allowed.

Here is an illustration on how to create an API key:

Example

Say you created the API Key below:

Plug the Secret key and Marqo endpoint URL in to your code to connect to the marqo client.

Python

pip install marqo

import marqo

marqo_endpoint_url = "https://XXXXXX.execute-api.us-east-1.amazonaws.com/prod" # the endpoint provided on the "indices" page of the console
api_key = "oCmxAvL7ht6hlmzRoTYwv4eqNsZunysK9EFgJZ0l" # an API key you created
mq = marqo.Client(url=marqo_endpoint_url, api_key=api_key)

mq.index("my-first-index").add_documents([
    {
        "Title": "The Travels of Marco Polo",
        "Description": "A 13th-century travelogue describing Polo's travels"
    },
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection, "
                       "mobility, life support, and communications for astronauts",
        "_id": "article_591"
    }]
)

results = mq.index("my-first-index").search(
    q="What is the best outfit to wear on the moon?"
)

• If you're using a GPU device on the cloud, specify device="cuda" in add documents and search calls on the cloud in order to take advantage of accelerated inference

• mq is the client that wraps themarqo API
• add_documents() takes a list of documents, represented as python dicts, for indexing
• add_documents() creates an index with default settings, if one does not already exist
• You can optionally set a document's ID with the special _id field. Otherwise, marqo will generate one.
• If the index doesn't exist, Marqo will create it. If it exists then Marqo will add the documents to the index.

Sharding on Marqo Cloud

Marqo cloud allows a maximum 20GB per shard. To ensure optimal performance, it's better to keep your index below 10GB per shard. Note that you are not able to edit the number of shards in an index once you have initially specified them, and if you want to increase the index size, you'll need to re-index the data. You can specify a number of shards by using the create index API.

Note that index_defaults are required to be specified when creating a new index. See create index documentation here or refer to the below example:

Example

cURLPython

curl -XPOST 'http://localhost:8882/indexes/my-first-index' -H 'Content-type:application/json' -d '
{
"index_defaults": {
    "treat_urls_and_pointers_as_images": false,
    "model": "hf/all_datasets_v4_MiniLM-L6",
    "normalize_embeddings": true,
    "text_preprocessing": {
        "split_length": 2,
        "split_overlap": 0,
        "split_method": "sentence"
    },
    "image_preprocessing": {
        "patch_method": null
    }
},
"number_of_shards": 5
}'

index_settings = {
"index_defaults": {
    "treat_urls_and_pointers_as_images": False,
    "model": "hf/all_datasets_v4_MiniLM-L6",
    "normalize_embeddings": True,
    "text_preprocessing": {
        "split_length": 2,
        "split_overlap": 0,
        "split_method": "sentence"
    },
    "image_preprocessing": {
        "patch_method": None
    }
},
"number_of_shards": 5
}
mq.create_index("my-first-index", settings_dict=index_settings)