Using Marqo Cloud
Exploring console.marqo.ai
Login
Login to your marqo cloud console account. To register for Marqo cloud, apply here.
Indexes
You'll be presented with a table of your indexes and on a separate section you can see your Marqo Endpoint Details
.
Each Marqo Endpoint URL is unique.
Below is how the overview page looks:
To view graphs on Data size
, Count of documents
, Total search count
, and Primary shards
, select the Monitor
tab for the index page.
API Keys
Your API Keys are used to access indexes through the marqo client
.
- Up to four API keys are allowed.
Here is an illustration on how to create an API key:
Example
Say you created the API Key below:
Plug the Secret key
and Marqo endpoint URL
in to your code to connect to the marqo client.
pip install marqo
import marqo
marqo_endpoint_url = "https://XXXXXX.execute-api.us-east-1.amazonaws.com/prod" # the endpoint provided on the "indices" page of the console
api_key = "oCmxAvL7ht6hlmzRoTYwv4eqNsZunysK9EFgJZ0l" # an API key you created
mq = marqo.Client(url=marqo_endpoint_url, api_key=api_key)
mq.index("my-first-index").add_documents([
{
"Title": "The Travels of Marco Polo",
"Description": "A 13th-century travelogue describing Polo's travels"
},
{
"Title": "Extravehicular Mobility Unit (EMU)",
"Description": "The EMU is a spacesuit that provides environmental protection, "
"mobility, life support, and communications for astronauts",
"_id": "article_591"
}]
)
results = mq.index("my-first-index").search(
q="What is the best outfit to wear on the moon?"
)
- If you're using a GPU device on the cloud, specify device="cuda" in add documents and search calls on the cloud in order to take advantage of accelerated inference
mq
is the client that wraps themarqo
APIadd_documents()
takes a list of documents, represented as python dicts, for indexingadd_documents()
creates an index with default settings, if one does not already exist- You can optionally set a document's ID with the special
_id
field. Otherwise, marqo will generate one. - If the index doesn't exist, Marqo will create it. If it exists then Marqo will add the documents to the index.
Sharding on Marqo Cloud
Marqo cloud allows a maximum 20GB per shard. To ensure optimal performance, it's better to keep your index below 10GB per shard. Note that you are not able to edit the number of shards in an index once you have initially specified them, and if you want to increase the index size, you'll need to re-index the data. You can specify a number of shards by using the create index API.
Note that index_defaults are required to be specified when creating a new index. See create index documentation here or refer to the below example:
Example
curl -XPOST 'http://localhost:8882/indexes/my-first-index' -H 'Content-type:application/json' -d '
{
"index_defaults": {
"treat_urls_and_pointers_as_images": false,
"model": "hf/all_datasets_v4_MiniLM-L6",
"normalize_embeddings": true,
"text_preprocessing": {
"split_length": 2,
"split_overlap": 0,
"split_method": "sentence"
},
"image_preprocessing": {
"patch_method": null
}
},
"number_of_shards": 5
}'
index_settings = {
"index_defaults": {
"treat_urls_and_pointers_as_images": False,
"model": "hf/all_datasets_v4_MiniLM-L6",
"normalize_embeddings": True,
"text_preprocessing": {
"split_length": 2,
"split_overlap": 0,
"split_method": "sentence"
},
"image_preprocessing": {
"patch_method": None
}
},
"number_of_shards": 5
}
mq.create_index("my-first-index", settings_dict=index_settings)
Adding Team Members
On Marqo it's not possible to add the same email as a team member to multiple accounts. If you want to add your email to multiple accounts, the suggested workaround is to use the "+" symbol in the email, which will enable you to have an additional email which routes to your usual mailbox but will be separately identified when you log into Marqo.
Example: tom+314532432@example.com Example: tom+devaccount@example.com