Bring your own CLIP model
Most customers can get great performance from publicly available CLIP models. However, some use cases will benefit even more from a model fine-tuned for their domain specific task. In this circumstance, you should use your own model with fine-tuned weights and parameters. It is very convenient to incorporate your own model in Marqo as long as your model belongs to one of the following frameworks:
To use your fine-tuned model, here are the detailed steps:
1. Fine-tune your model
The first step is to fine-tune your model using the frameworks mentioned
above. Here we use Open CLIP
framework as an example. You should follow the guide
to fine-tune your own model and store the trained model (checkpoint) as a *.pt
file.
2. Upload your model to a server
You need to upload your model (the *.pt
file) to a cloud storage (e.g., Amazon S3, GitHub) and use the downloading
address to reference it in Marqo.
3. Use your model in Marqo
To use your custom model, you need to create an index
in Marqo and define it in your index settings via model
and model_properties
. For an example Open CLIP
model, the code is:
settings = {
"index_defaults": {
"treat_urls_and_pointers_as_images": True,
"model": 'generic-clip-test-model-1',
"model_properties": {
"name": "ViT-B-32-quickgelu",
"dimensions": 512,
"url": "https://github.com/mlfoundations/open_clip/releases/download/v0.2-weights/vit_b_32-quickgelu-laion400m_avg-8a00ab3c.pt",
"type": "open_clip",
},
"normalize_embeddings": True,
},
}
response = mq.create_index("my-own-clip", settings_dict=settings)
"model_properties"
.
3. Preloading your model
There may be cases wherein you want to preload (or prewarm, in other terms) your model before using it to index. This can be done by adding your model (with model
and model_properties
) to the list of models on startup in your marqo configuration.
The syntax for this can be found in Configuring preloaded models