Multimodal Image Search with Marqo

First, select your platform:

Marqo Cloud

Marqo Open Source

This guide will walk you through setting up your environment and using Marqo Cloud to perform text-image searches. Follow along for a step-by-step tutorial.

Full code:

If you have any questions or need help, visit our Community and ask in the get-help channel.

Step 1: Get Marqo Cloud API Key

First, we need to obtain our Marqo Cloud API Key. For more information on how you can obtain this, visit our article. Once you have obtained this, replace your_api_key with your actual API Key:

api_key = "your_api_key"

Step 2: Create a Marqo Index

PythoncURL

from marqo import Client

# Initialize the Marqo client with the API URL and your API key.
# The API key is required to authenticate requests to the Marqo API.
mq = Client(
    "https://api.marqo.ai", 
    api_key=api_key
)

# Define a unique name for your index.
index_name = 'image-search'

# Configure the index settings
settings = {
    "model": "ViT-B/32",  # A Vision Transformer model for image embeddings.
    "treatUrlsAndPointersAsImages": True,  # URLs will be treated as image sources.
    "inferenceType": "marqo.GPU",  # Use GPU for faster inference.
}

# Create the index with the specified settings.
mq.create_index(index_name, settings_dict=settings)

curl -X POST 'https://api.marqo.ai/api/v2/indexes/image-search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' \
-d '
{
"treatUrlsAndPointersAsImages": true,
"model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
"inferenceType": "marqo.GPU"
}'

Replace XXXXXXXXXXXXXXX with your actual API Key. To find this, visit Find Your API Key.

Step 3: Add Images to the Index

We now add images to the Marqo index. The images we will use can be found in our examples folder in our GitHub repo.

PythoncURL

from pprint import pprint

# Add a list of documents containing image URLs and associated descriptions to the index.
# These images are hosted on the Marqo GitHub repository.
documents = [
    {"title": "a woman on her phone taking a photo", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"title": "a horse and rider jumping over a fence", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"title": "an aeroplane and the moon", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"title": "man stood by a traffic light", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add the documents to the index with specific mappings and tensor fields.
# The mapping 'image_title_multimodal' combines text and image features with specified weights.
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,  # Add documents one at a time for simplicity.
    mappings={
        "image_title_multimodal": {
            "type": "multimodal_combination",  # Combine text and image modalities.
            "weights": {"title": 0.1, "image": 0.9},  # Assign higher importance to image data.
        }
    },
    tensor_fields=["image_title_multimodal"],  # Specify fields to generate tensors for.
)

# Print the result of adding documents to verify success.
pprint(res)

curl -XPOST 'your_endpoint/indexes/image-search/documents' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"documents": [
    {
    "title": "a woman on her phone taking a photo",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"
    },
    {
    "title": "a horse and rider jumping over a fence",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"
    },
    {
    "title": "an aeroplane and the moon",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"
    },
    {
    "title": "man stood by a traffic light",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"
    }
],
"mappings": {
    "image_title_multimodal": {
    "type": "multimodal_combination",
    "weights": {
        "title": 0.1,
        "image": 0.9
    }
    }
},
"tensorFields": ["image_title_multimodal"]
}'

Replace your_endpoint and api_key with your actual endpoint and API Key. To find these, visit Find Your Endpoint and Find Your API Key.

Step 4: Perform a Search

Use Marqo to search for an image by describing what you're looking for in natural language.

PythoncURL

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

curl -XPOST 'your_endpoint/indexes/image-search/search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
    "q": "A rider on a horse jumping over the barrier"
}'

Replace your_endpoint and api_key with your actual endpoint and API Key. To find these, visit Find Your Endpoint and Find Your API Key.

We can now visualise this result using some code which is featured in our GitHub Python script. This returns the following image which indeed, is an exact match to our query!

Step 5: Clean up

When you are done with the index you can delete it with the following code:

PythoncURL

# Delete your index
mq.delete_index("image-search")

This is a cURL example for the query 'public speaking'.

curl -XDELETE https://api.marqo.ai/api/v2/indexes/image-search \
-H 'x-api-key: XXXXXXXXXXXXXXX'

If you do not delete your index you will continue to be charged for it.

Full Code

image_search_cloud.py

#####################################################
### STEP 1. Obtain Marqo API Key
#####################################################

# 1. Sign Up to Marqo Cloud: https://cloud.marqo.ai/
# 2. Get a Marqo API Key: https://www.marqo.ai/blog/finding-my-marqo-api-key
# Replace this with your Marqo API key:
api_key = "your_api_key"

####################################################
### STEP 2: Initialize Marqo Client
####################################################

from marqo import Client

# Initialize the Marqo client with the API URL and your API key.
# The API key is required to authenticate requests to the Marqo API.
mq = Client(
    "https://api.marqo.ai", 
    api_key=api_key
)

####################################################
### STEP 3: Create a Marqo Index
####################################################

# Define a unique name for your index.
index_name = 'image-search'

# Check if an index with the same name already exists and delete it if necessary.
# This avoids errors when trying to create an index with an existing name.
try:
    mq.index(index_name).delete()
except:
    pass  # It's safe to ignore errors if the index does not exist.

# Configure the index settings, such as the model to use for embedding images,
# whether URLs should be treated as images, and the inference hardware type.
settings = {
    "model": "ViT-B/32",  # A Vision Transformer model for image embeddings.
    "treatUrlsAndPointersAsImages": True,  # URLs will be treated as image sources.
    "inferenceType": "marqo.GPU",  # Use GPU for faster inference.
}

# Create the index with the specified settings.
mq.create_index(index_name, settings_dict=settings)

####################################################
### STEP 4: Add Images to the Index
####################################################

from pprint import pprint

# Add a list of documents containing image URLs and associated descriptions to the index.
# These images are hosted on the Marqo GitHub repository.
documents = [
    {"title": "a woman on her phone taking a photo", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"title": "a horse and rider jumping over a fence", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"title": "an aeroplane and the moon", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"title": "man stood by a traffic light", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add the documents to the index with specific mappings and tensor fields.
# The mapping 'image_title_multimodal' combines text and image features with specified weights.
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,  # Add documents one at a time for simplicity.
    mappings={
        "image_title_multimodal": {
            "type": "multimodal_combination",  # Combine text and image modalities.
            "weights": {"title": 0.1, "image": 0.9},  # Assign higher importance to image data.
        }
    },
    tensor_fields=["image_title_multimodal"],  # Specify fields to generate tensors for.
)

# Print the result of adding documents to verify success.
pprint(res)

####################################################
### STEP 4: Search using Marqo
####################################################

# Define a natural language query to search the indexed documents.
query = "A rider on a horse jumping over the barrier"

# Execute the search and retrieve the results.
search_results = mq.index(index_name).search(query)

# Extract the top result from the search results.
top_result = search_results['hits'][0]

# Print the top result for debugging or verification purposes.
print(search_results['hits'][0])

####################################################
### STEP 5: Visualize the Output
####################################################

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import urllib.request
from io import BytesIO

# Get the image URL from the top search result.
image_url = top_result['image']

# Download the image from the URL and load it as a PIL Image object.
with urllib.request.urlopen(image_url) as url:
    img = Image.open(BytesIO(url.read()))

# Convert the PIL Image object to a NumPy array for compatibility with matplotlib.
img_array = np.array(img)

# Display the image using matplotlib without axis labels or ticks.
plt.imshow(img_array)
plt.axis('off')  # Hide the axis for a cleaner display.
plt.show()

This guide will walk you through setting up your environment and using Marqo to perform image searches. Follow along for a step-by-step tutorial.

Full code: Image Search with Marqo Open Source Code

If you have any questions or need help, visit our Community and ask in the get-help channel.

Step 1: Start Marqo

We need to get Marqo up and running. You can do this by executing the following command in your terminal:

docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest

For more detailed instructions, check the Installation Guide.

Step 2: Create a Marqo Index

PythoncURL

from marqo import Client

# Initialize the Marqo Client
mq = Client("http://localhost:8882")

# Name your index
index_name = 'image-search'

# Define settings for the index
settings = {
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "treatUrlsAndPointersAsImages": True,
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

curl -X POST 'http://localhost:8882/indexes/image-search' \
-H 'Content-type:application/json' \
-d '
{
"treatUrlsAndPointersAsImages": true,
"model": "open_clip/ViT-B-32/laion2b_s34b_b79k"
}'

Step 3: Add Images to the Index

We now add images to the Marqo index. The images we will use can be found in our examples folder in our GitHub repo.

PythoncURL

documents = [
    {"title": "a woman on her phone taking a photo", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"title": "a horse and rider jumping over a fence", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"title": "an aeroplane and the moon", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"title": "man stood by a traffic light", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add the documents to the index with specific mappings and tensor fields.
# The mapping 'image_title_multimodal' combines text and image features with specified weights.
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,  # Add documents one at a time for simplicity.
    mappings={
        "image_title_multimodal": {
            "type": "multimodal_combination",  # Combine text and image modalities.
            "weights": {"title": 0.1, "image": 0.9},  # Assign higher importance to image data.
        }
    },
    tensor_fields=["image_title_multimodal"],  # Specify fields to generate tensors for.
)

# Print the result of adding documents to verify success.
pprint(res)

curl -XPOST 'http://localhost:8882/indexes/image-search/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [
    {
    "title": "a woman on her phone taking a photo",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"
    },
    {
    "title": "a horse and rider jumping over a fence",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"
    },
    {
    "title": "an aeroplane and the moon",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"
    },
    {
    "title": "man stood by a traffic light",
    "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"
    }
],
"mappings": {
    "image_title_multimodal": {
    "type": "multimodal_combination",
    "weights": {
        "title": 0.1,
        "image": 0.9
    }
    }
},
"tensorFields": ["image_title_multimodal"]
}'

Step 4: Perform a Search

Use Marqo to search for an image by describing what you're looking for in natural language.

PythoncURL

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

curl -XPOST 'http://localhost:8882/indexes/image-search/search' \
-H 'Content-type:application/json' -d '
{
    "q": "A rider on a horse jumping over the barrier"
}'

We can now visualise this result using some code which is featured in our GitHub Python script. This returns the following image which indeed, is an exact match to our query!

Step 5: Clean up

When you are done with the index you can delete it with the following code:

PythoncURL

# Delete your index
mq.delete_index("image-search")

curl -XDELETE http://localhost:8882/indexes/image-search

Full Code

GitHub Code: Image Search with Marqo Open Source Code

image_search_open_source.py

from marqo import Client
from pprint import pprint

#####################################################
### STEP 1. Start Marqo Locally
#####################################################

"""
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
"""

####################################################
### STEP 2: Create Marqo Index
####################################################

# Initialize the Marqo Client
mq = Client("http://localhost:8882")

# Name your index
index_name = 'image-search'

# We create the index. Note if it already exists an error will occur
# as you cannot overwrite an existing index. For this reason, we delete
# any existing index 
try:
mq.index(index_name).delete()
except:
    pass # It's safe to ignore errors if the index does not exist.

# Define settings for the index
settings = {
    "model": "ViT-B/32",   # A Vision Transformer model for image embeddings.
    "treatUrlsAndPointersAsImages": True,   # URLs will be treated as image sources.
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

# ####################################################
# ### STEP 3: Add Images to the Index
# ####################################################

# Add a list of documents containing image URLs and associated descriptions to the index.
# These images are hosted on the Marqo GitHub repository.
documents = [
    {"title": "a woman on her phone taking a photo", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"title": "a horse and rider jumping over a fence", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"title": "an aeroplane and the moon", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"title": "man stood by a traffic light", "image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add the documents to the index with specific mappings and tensor fields.
# The mapping 'image_title_multimodal' combines text and image features with specified weights.
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,  # Add documents one at a time for simplicity.
    mappings={
        "image_title_multimodal": {
            "type": "multimodal_combination",  # Combine text and image modalities.
            "weights": {"title": 0.1, "image": 0.9},  # Assign higher importance to image data.
        }
    },
    tensor_fields=["image_title_multimodal"],  # Specify fields to generate tensors for.
)

# Print the result of adding documents to verify success.
pprint(res)

####################################################
### STEP 4: Search using Marqo
####################################################

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

####################################################
### STEP 5: Visualize the Output
####################################################

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import urllib.request
from io import BytesIO

# URL of the image
image_url = top_result['image']

# Open the URL and read the image into a Pillow Image object
with urllib.request.urlopen(image_url) as url:
    img = Image.open(BytesIO(url.read()))

# Convert the PIL Image object to a NumPy array for matplotlib
img_array = np.array(img)

# Display the image using matplotlib
plt.imshow(img_array)
plt.axis('off')  # Hide the axis
plt.show()