Skip to content

Using Marqo Cloud

Exploring console.marqo.ai

Login

Login to your marqo cloud console account. To register for Marqo cloud, apply here.

Indexes

You'll be presented with a table of your indexes and on a separate section you can see your Marqo Endpoint Details.

Each Marqo Endpoint URL is unique.

Below is how the overview page looks:

Overview Page

To view graphs on Data size, Count of documents, Total search count, and Primary shards, select the Monitor tab for the index page.

Navigating to Index Monitor Page

API Keys

Your API Keys are used to access indexes through the marqo client.

Here is an illustration on how to create an API key:

API Keys Page

Example

Say you created the API Key below:

secret key

Plug the Secret key and Marqo endpoint URL in to your code to connect to the marqo client (or specify as X-API-KEY in the header if you are using the REST API). 

pip install marqo

import marqo

marqo_endpoint_url = "https://XXXXXXXX.api.s2search.io" # the endpoint provided on the "indices" page of the console
api_key = "oCmxAvL7ht6hlmzRoTYwv4eqNsZunysKd9EFgJZ0l" # an API key you created (specify in header as X-API-KEY when calling the endpoint directly). 
mq = marqo.Client(url=marqo_endpoint_url, api_key=api_key)

mq.index("my-first-index").add_documents([
    {
        "Title": "The Travels of Marco Polo",
        "Description": "A 13th-century travelogue describing Polo's travels"
    },
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection, "
                       "mobility, life support, and communications for astronauts",
        "_id": "article_591"
    }]
)

results = mq.index("my-first-index").search(
    q="What is the best outfit to wear on the moon?"
)

curl -XPOST -H 'Content-type: application/json' -H 'X-API-KEY: XXXXXXXXXXXXXXXXXXXX' https://XXXXXXXX.api.s2search.io -d '[
    {
        "Title": "The Travels of Marco Polo",
        "Description": "A 13th-century travelogue describing Polo'\''s travels"
    },
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection, mobility, life support, and communications for astronauts",
        "_id": "article_591"
    }
]'

curl -XPOST http://localhost:8882/indexes/my-first-index/search -H 'X-API-KEY: XXXXXXXXXXXXXXXXXXXX' -H 'Content-type: application/json' -d '{
  "q":"What is the best outfit to wear on the moon?"
}'
  • If you're using a GPU device on the cloud, specify device="cuda" in add documents and search calls on the cloud in order to take advantage of accelerated inference
  • mq is the client that wraps themarqo API
  • add_documents() takes a list of documents, represented as python dicts, for indexing
  • add_documents() creates an index with default settings, if one does not already exist
  • You can optionally set a document's ID with the special _id field. Otherwise, marqo will generate one.
  • If the index doesn't exist, Marqo will create it. If it exists then Marqo will add the documents to the index.

Sharding on Marqo Cloud

Marqo cloud allows a maximum 140GB per shard replica pair. This roughly equates to 3,000,000 vectors of 756 dimension. Shards should be as large as possible to provide optimal performance. Note that you are not able to edit the number of shards in an index once you have initially specified them, and if you want to increase the index size, you'll need to re-index the data. You can specify a number of shards by using the create index API.

Note that index_defaults are required to be specified when creating a new index. See create index documentation here or refer to the below example:

Example

curl -XPOST 'http://localhost:8882/indexes/my-first-index' -H 'Content-type:application/json' -H 'X-API-KEY: XXXXXXXXXXXXXXXXXXXX' -d '
{
"index_defaults": {
    "treat_urls_and_pointers_as_images": false,
    "model": "hf/all_datasets_v4_MiniLM-L6",
    "normalize_embeddings": true,
    "text_preprocessing": {
        "split_length": 2,
        "split_overlap": 0,
        "split_method": "sentence"
    },
    "image_preprocessing": {
        "patch_method": null
    }
},
"number_of_shards": 5
}'

index_settings = {
"index_defaults": {
    "treat_urls_and_pointers_as_images": False,
    "model": "hf/all_datasets_v4_MiniLM-L6",
    "normalize_embeddings": True,
    "text_preprocessing": {
        "split_length": 2,
        "split_overlap": 0,
        "split_method": "sentence"
    },
    "image_preprocessing": {
        "patch_method": None
    }
},
"number_of_shards": 5
}
mq.create_index("my-first-index", settings_dict=index_settings)

Adding Team Members

On Marqo it's not possible to add the same email as a team member to multiple accounts. If you want to add your email to multiple accounts, the suggested workaround is to use the "+" symbol in the email, which will enable you to have an additional email which routes to your usual mailbox but will be separately identified when you log into Marqo.

Example: tom+314532432@example.com Example: tom+devaccount@example.com

How do I create an API key?

To create an API key on Marqo cloud, first log into the Marqo cloud console. Then navigate to the "API Keys" tab, and click the blue "+" button.

Here you can enter the name of the API key. Once you've entered the name, click create. When the API key is generated, you'll be able to press the eye shaped button to unhide the secret key. To use the key, copy this secret key that was hidden and add it either as the API key when you initialise the python client (as above), or specify it in the header as X-API-KEY like below when using the REST API:

curl -XPOST 'https://XXXXXXXX.api.s2search.io/indexes/my-first-index/documents?non_tensor_fields=Title&non_tensor_fields=Genre' \
-H 'Content-type:application/json' -H 'X-API-KEY: XXXXXXXXXXXXXXXXXXXX' -d '
[ 
    {
         "Title": "The Travels of Marco Polo",
         "Description": "A 13th-century travelogue describing the travels of Polo",
         "Genre": "History"
      }, 
    {
        "Title": "Extravehicular Mobility Unit (EMU)",
        "Description": "The EMU is a spacesuit that provides environmental protection",
        "_id": "article_591",
        "Genre": "Science"
    }
]'