Image Search with Marqo

First, select your platform:

Marqo Cloud

Marqo Open Source

This guide will walk you through setting up your environment and using Marqo Cloud to perform image searches. Follow along for a step-by-step tutorial.

Full code:

If you have any questions or need help, visit our Community and ask in the get-help channel.

Step 1: Get Marqo Cloud API Key

First, we need to obtain our Marqo Cloud API Key. For more information on how you can obtain this, visit our article. Once you have obtained this, replace your_api_key with your actual API Key:

api_key = "your_api_key"

Step 2: Create a Marqo Index

PythoncURL

import marqo

# Set up the Client
mq = marqo.Client(
    "https://api.marqo.ai", 
    api_key=api_key
)

# Name your index
index_name = 'image-search'

# Define settings for the index
settings = {
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "treatUrlsAndPointersAsImages": True,
    "inferenceType": "marqo.GPU"
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

curl -X POST 'https://api.marqo.ai/api/v2/indexes/image-search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' \
-d '
{
"treatUrlsAndPointersAsImages": true,
"model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
"inferenceType": "marqo.GPU"
}'

Replace XXXXXXXXXXXXXXX with your actual API Key. To find this, visit Find Your API Key.

Step 3: Add Images to the Index

We now add images to the Marqo index. The images we will use can be found in our examples folder in our GitHub repo.

PythoncURL

from pprint import pprint

# We will use 4 images from our examples folder in our GitHub repo: https://github.com/marqo-ai/marqo/tree/mainline/examples/ImageSearchGuide/data
documents = [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add these documents to the index
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,
    tensor_fields=["image"]
)

# Print out if you wish to
from pprint import pprint
pprint(res)

curl -XPOST 'your_endpoint/indexes/image-search/documents' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
"documents": [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
],
"tensorFields": ["image"]
}'

Replace your_endpoint and api_key with your actual endpoint and API Key. To find these, visit Find Your Endpoint and Find Your API Key.

Step 4: Perform a Search

Use Marqo to search for an image by describing what you're looking for in natural language.

PythoncURL

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

curl -XPOST 'your_endpoint/indexes/image-search/search' \
-H 'x-api-key: XXXXXXXXXXXXXXX' \
-H 'Content-type:application/json' -d '
{
    "q": "A rider on a horse jumping over the barrier"
}'

Replace your_endpoint and api_key with your actual endpoint and API Key. To find these, visit Find Your Endpoint and Find Your API Key.

We can now visualise this result using some code which is featured in our GitHub Python script. This returns the following image which indeed, is an exact match to our query!

Full Code

image_search_cloud.py

#####################################################
### STEP 1. Obtain Marqo API Key
#####################################################

# To find your Marqo API Key, visit this article: https://marqo.ai/blog/finding-my-marqo-api-key
api_key = "your_api_key"

####################################################
### STEP 2: Initialize Marqo Client
####################################################

from marqo import Client

# Set up the Client
mq = Client(
    "https://api.marqo.ai", 
    api_key=api_key
)

####################################################
### STEP 3: Create a Marqo Index
####################################################

# Name your index
index_name = 'image-search-cloud'

# We create the index. Note if it already exists an error will occur
# as you cannot overwrite an existing index. For this reason, we delete
# any existing index 
try:
    mq.index(index_name).delete()
except:
    pass

# Define settings for the index
settings = {
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "treatUrlsAndPointersAsImages": True,
    "inferenceType": "marqo.GPU"
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

# ####################################################
# ### STEP 4: Add Images to the Index
# ####################################################

# We will use 4 images from our examples folder in our GitHub repo: https://github.com/marqo-ai/marqo
documents = [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
]

# Add these documents to the index
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,
    tensor_fields=["image"]
)

# Print out if you wish to
from pprint import pprint

pprint(res)

####################################################
### STEP 4: Search using Marqo
####################################################

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

####################################################
### STEP 5: Visualize the Output
####################################################

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import urllib.request
from io import BytesIO

# URL of the image
image_url = top_result['image']

# Open the URL and read the image into a Pillow Image object
with urllib.request.urlopen(image_url) as url:
    img = Image.open(BytesIO(url.read()))

# Convert the PIL Image object to a NumPy array for matplotlib
img_array = np.array(img)

# Display the image using matplotlib
plt.imshow(img_array)
plt.axis('off')  # Hide the axis
plt.show()

This guide will walk you through setting up your environment and using Marqo to perform image searches. Follow along for a step-by-step tutorial.

Full code: Image Search with Marqo Open Source Code

If you have any questions or need help, visit our Community and ask in the get-help channel.

Step 1: Start Marqo

We need to get Marqo up and running. You can do this by executing the following command in your terminal:

docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest

For more detailed instructions, check the Installation Guide.

Step 2: Create a Marqo Index

PythoncURL

from marqo import Client

# Initialize the Marqo Client
mq = Client("http://localhost:8882")

# Name your index
index_name = 'image-search'

# Define settings for the index
settings = {
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "treatUrlsAndPointersAsImages": True,
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

curl -X POST 'http://localhost:8882/indexes/image-search' \
-H 'Content-type:application/json' \
-d '
{
"treatUrlsAndPointersAsImages": true,
"model": "open_clip/ViT-B-32/laion2b_s34b_b79k"
}'

Step 3: Add Images to the Index

We now add images to the Marqo index. The images we will use can be found in our examples folder in our GitHub repo.

PythoncURL

# We will use 4 images from our examples folder in our GitHub repo: https://github.com/marqo-ai/marqo/tree/mainline/examples/ImageSearchGuide/data
documents = [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
    # Add more documents here 
]

# Add these documents to the index
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,
    tensor_fields=["image"]
)

# Print out if you wish to
pprint(res)

curl -XPOST 'http://localhost:8882/indexes/image-search/documents' \
-H 'Content-type:application/json' -d '
{
"documents": [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
],
"tensorFields": ["image"]
}'

Step 4: Perform a Search

Use Marqo to search for an image by describing what you're looking for in natural language.

PythoncURL

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

curl -XPOST 'http://localhost:8882/indexes/image-search/search' \
-H 'Content-type:application/json' -d '
{
    "q": "A rider on a horse jumping over the barrier"
}'

We can now visualise this result using some code which is featured in our GitHub Python script. This returns the following image which indeed, is an exact match to our query!

Full Code

GitHub Code: Image Search with Marqo Open Source Code

image_search_open_source.py

from marqo import Client
from pprint import pprint

#####################################################
### STEP 1. Start Marqo
#####################################################

"""
docker pull marqoai/marqo:latest
docker rm -f marqo
docker run --name marqo -it -p 8882:8882 marqoai/marqo:latest
"""

####################################################
### STEP 2: Create Marqo Index
####################################################

# Initialize the Marqo Client
mq = Client("http://localhost:8882")

# Name your index
index_name = 'image-search-open-source'

# We create the index. Note if it already exists an error will occur
# as you cannot overwrite an existing index. For this reason, we delete
# any existing index 
try:
    mq.index(index_name).delete()
except:
    pass

# Define settings for the index
settings = {
    "model": "open_clip/ViT-B-32/laion2b_s34b_b79k",
    "treatUrlsAndPointersAsImages": True,
}

# Create the index
mq.create_index(index_name, settings_dict=settings)

# ####################################################
# ### STEP 3: Add Images to the Index
# ####################################################

# We will use 4 images from our examples folder in our GitHub repo: https://github.com/marqo-ai/marqo
documents = [
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image0.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image1.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image2.jpg"},
    {"image": "https://raw.githubusercontent.com/marqo-ai/marqo/mainline/examples/ImageSearchGuide/data/image3.jpg"}
    # Add more documents here
]

# Add these documents to the index
res = mq.index(index_name).add_documents(
    documents,
    client_batch_size=1,
    tensor_fields=["image"]
)

# Print out if you wish to
pprint(res)

####################################################
### STEP 4: Search using Marqo
####################################################

# Define a query
query = "A rider on a horse jumping over the barrier"

# Perform a search for this query
search_results = mq.index(index_name).search(query)

# Obtain the top result 
top_result = search_results['hits'][0]
# Print the top result
print(search_results['hits'][0])

####################################################
### STEP 5: Visualize the Output
####################################################

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
import urllib.request
from io import BytesIO

# URL of the image
image_url = top_result['image']

# Open the URL and read the image into a Pillow Image object
with urllib.request.urlopen(image_url) as url:
    img = Image.open(BytesIO(url.read()))

# Convert the PIL Image object to a NumPy array for matplotlib
img_array = np.array(img)

# Display the image using matplotlib
plt.imshow(img_array)
plt.axis('off')  # Hide the axis
plt.show()