Skip to content

Release 0.0.21

New features

  • Load custom SBERT models from cloud storage with authentication (https://github.com/marqo-ai/marqo/pull/474). Marqo now supports fetching your fine-tuned public and private SBERT models from Hugging Face and AWS s3. Learn more about using your own SBERT model here. For instructions on loading a private model using authentication, check model auth during search and model auth during add_documents.

  • Bulk search score modifier and context vector support (https://github.com/marqo-ai/marqo/pull/469). Support has been added for score modifiers and context vectors to our bulk search API. This can help enhance throughput and performance for certain workloads. Please see documentation for usage.

Bug fixes and minor changes

  • README enhancements (https://github.com/marqo-ai/marqo/pull/482, https://github.com/marqo-ai/marqo/pull/481).

Contributor shout-outs

  • A special thank you to our 3.0k stargazers!

Release 0.0.20

New features

  • Custom model pre-loading (https://github.com/marqo-ai/marqo/pull/475). Public CLIP and OpenCLIP models specified by URL can now be loaded on Marqo startup via the MARQO_MODELS_TO_PRELOAD environment variable. These must be formatted as JSON objects with model and model_properties. See here (configuring pre-loaded models) for usage.

Bug fixes and minor changes

  • Fixed arm64 build issue caused by package version conflicts (https://github.com/marqo-ai/marqo/pull/478)

Release 0.0.19

New features

  • Model authorisation(https://github.com/marqo-ai/marqo/pull/460). Non-public OpenCLIP and CLIP models can now be loaded from Hugging Face and AWS s3 via the model_location settings object and model_auth. See here (model auth during search) and here (model auth during add_documents) for usage.
  • Max replicas configuration (https://github.com/marqo-ai/marqo/pull/465). Marqo admins now have more control over the max number of replicas that can be set for indexes on the Marqo instance. See here for how to configure this.

Breaking changes

  • Marqo now allows for a maximum of 1 replica per index by default (https://github.com/marqo-ai/marqo/pull/465).

Bug fixes and minor changes

  • README improvements (https://github.com/marqo-ai/marqo/pull/468)
  • OpenCLIP version bumped (https://github.com/marqo-ai/marqo/pull/461)
  • Added extra tests (https://github.com/marqo-ai/marqo/pull/464/)
  • Unneeded files are now excluded in Docker builds (https://github.com/marqo-ai/marqo/pull/448, https://github.com/marqo-ai/marqo/pull/426)

Contributor shout-outs

  • Thank you to our 2.9k stargazers!
  • Thank you to community members for the increasingly exciting discussions on our Slack channel. Feedback, questions and hearing about use cases helps us build a great open source product.
  • Thank you to @jalajk24 for the PR to exclude unneeded files from Docker builds!

Release 0.0.18

New features

  • New E5 model type is available (https://github.com/marqo-ai/marqo/pull/419). E5 models are state of the art general-purpose text embedding models that obtained the best results on the MTEB benchmark when released in Dec 2022. Read more about these models here.
  • Automatic model ejection (https://github.com/marqo-ai/marqo/pull/372). Automatic model ejection helps prevent out-of-memory (OOM) errors on machines with a larger amount of CPU memory (16GB+) by ejecting the least recently used model.
  • Speech processing article and example (https://github.com/marqo-ai/marqo/pull/431). @OwenPendrighElliott demonstrates how you can build and query a Marqo index from audio clips.

Optimisations

  • Delete optimisation (https://github.com/marqo-ai/marqo/pull/436). The /delete endpoint can now handle a higher volume of requests.
  • Inference calls can now execute in batches, with batch size configurable by an environment variable (https://github.com/marqo-ai/marqo/pull/376).

Bug fixes and minor changes

  • Configurable max value validation for HNSW graph parameters (https://github.com/marqo-ai/marqo/pull/424). See here for how to configure.
  • Configurable maximum number of tensor search attributes (https://github.com/marqo-ai/marqo/pull/430). See here for how to configure.
  • Unification of vectorise output type (https://github.com/marqo-ai/marqo/pull/432)
  • Improved test pipeline reliability (https://github.com/marqo-ai/marqo/pull/438, https://github.com/marqo-ai/marqo/pull/439)
  • Additional image download tests (https://github.com/marqo-ai/marqo/pull/402, https://github.com/marqo-ai/marqo/pull/442)
  • Minor fix in the Iron Manual example (https://github.com/marqo-ai/marqo/pull/440)
  • Refactored HTTP requests wrapper (https://github.com/marqo-ai/marqo/pull/367)

Contributor shout-outs

  • Thank you to our 2.8k stargazers!
  • Thank you community members raising issues and discussions in our Slack channel.
  • Thank you @jess-lord and others for raising issues

Release 0.0.17

New features

  • New parameters that allow tweaking of Marqo indexes' underlying HNSW graph. ef_construction and m can be defined at index time (https://github.com/marqo-ai/marqo/pull/386, https://github.com/marqo-ai/marqo/pull/420, https://github.com/marqo-ai/marqo/pull/421), giving you more control over the relevancy/speed tradeoff. See usage and more details here.
  • Score modification fields (https://github.com/marqo-ai/marqo/pull/414). Rank documents using knn similarity in addition to document metadata ( https://github.com/marqo-ai/marqo/pull/414). This allows integer or float fields from a document to bias a document's score during the knn search and allows additional ranking signals to be used. Use cases include giving more reputable documents higher weighting and de-duplicating search results. See usage here.

Bug fixes and minor changes

  • Added validation for unknown parameters during bulk search (https://github.com/marqo-ai/marqo/pull/413).
  • Improved concurrency handling when adding documents to an index as it's being deleted (https://github.com/marqo-ai/marqo/pull/407).
  • Better error messages for multimodal combination fields (https://github.com/marqo-ai/marqo/pull/395).
  • Examples of recently added features added to README (https://github.com/marqo-ai/marqo/pull/403).

Contributor shout-outs

Release 0.0.16

New features

  • Bulk search (https://github.com/marqo-ai/marqo/pull/363, https://github.com/marqo-ai/marqo/pull/373). Conduct multiple searches with just one request. This improves search throughput in Marqo by parallelising multiple search queries in a single API call. The average search time can be decreased up to 30%, depending on your devices and models. Check out the usage guide here
  • Configurable number of index replicas (https://github.com/marqo-ai/marqo/pull/391). You can now configure how many replicas to make for an index in Marqo using the number_of_replicas parameter. Marqo makes 1 replica by default. We recommend having at least one replica to prevent data loss. See the usage guide here
  • Use your own vectors during searches (https://github.com/marqo-ai/marqo/pull/381). Use your own vectors as context for your queries. Your vectors will be incorporated into the query using a weighted sum approach, allowing you to reduce the number of inference requests for duplicated content. Check out the usage guide here

Bug fixes and minor changes

  • Fixed a bug where some Open CLIP models were unable to load checkpoints from the cache (https://github.com/marqo-ai/marqo/pull/387).
  • Fixed a bug where multimodal search vectors are not combined based on expected weights (https://github.com/marqo-ai/marqo/pull/384).
  • Fixed a bug where multimodal document vectors are not combined in an expected way. numpy.sum was used rather than numpy.mean. (https://github.com/marqo-ai/marqo/pull/384).
  • Fixed a bug where an unexpected error is thrown when using_existing_tensor = True and documents are added with duplicate IDs (https://github.com/marqo-ai/marqo/pull/390).
  • Fixed a bug where the index settings validation did not catch the model field if it is in the incorrect part of the settings json (https://github.com/marqo-ai/marqo/pull/365).
  • Added missing descriptions and requirement files on our GPT-examples (https://github.com/marqo-ai/marqo/pull/349).
  • Updated the instructions to start Marqo-os (https://github.com/marqo-ai/marqo/pull/371).
  • Improved the Marqo start-up time by incorporating the downloading of the punkt tokenizer into the dockerfile (https://github.com/marqo-ai/marqo/pull/346).

Contributor shout-outs

  • Thank you to our 2.5k stargazers.
  • Thank you to @ed-muthiah for submitting a PR (https://github.com/marqo-ai/marqo/pull/349) that added missing descriptions and requirement files on our GPT-examples.

Release 0.0.15

New features

  • Multimodal tensor combination (https://github.com/marqo-ai/marqo/pull/332, https://github.com/marqo-ai/marqo/pull/355). Combine image and text data into a single vector! Multimodal combination objects can be added as Marqo document fields. For example, this can be used to encode text metadata into image vectors. See usage here.

Bug fixes

  • Fixed a bug that prevented CLIP's device check from behaving as expected (https://github.com/marqo-ai/marqo/pull/337)
  • CLIP utils is set to use the OpenCLIP default tokenizer so that long text inputs are truncated correctly (https://github.com/marqo-ai/marqo/pull/351).

Contributor shout-outs:

Release 0.0.14

New features

  • use_existing_tensors flag, for add_documents (https://github.com/marqo-ai/marqo/pull/335). Use existing Marqo tensors to autofill unchanged tensor fields, for existing documents. This lets you quickly add new metadata while minimising inference operations. See usage here.
  • image_download_headers parameter for search and add_documents (https://github.com/marqo-ai/marqo/pull/336). Index and search non-publicly available images. Add image download auth information to add_documents and search requests. See usage here.

Optimisations

  • The index cache is now updated on intervals of 2 seconds (https://github.com/marqo-ai/marqo/pull/333), rather than on every search. This reduces the pressure on Marqo-OS, allowing for greater search and indexing throughput.

Bug fixes

  • Helpful validation errors for invalid index settings (https://github.com/marqo-ai/marqo/pull/330). Helpful error messages allow for a smoother getting-started experience.
  • Automatic precision conversion to fp32 when using fp16 models on CPU (https://github.com/marqo-ai/marqo/pull/331).
  • Broadening of the types of image download errors gracefully handled. (https://github.com/marqo-ai/marqo/pull/321)

Release 0.0.13

New features

  • Support for custom CLIP models using the OpenAI and OpenCLIP architectures (https://github.com/marqo-ai/marqo/pull/286). Read about usage here.
  • Concurrency throttling (https://github.com/marqo-ai/marqo/pull/304). Configure the number of allowed concurrent indexing and search threads. Read about usage here.
  • Configurable logging levels (https://github.com/marqo-ai/marqo/pull/314). Adjust log output for your debugging/log storage needs. See how to configure log level here.
  • New array datatype (https://github.com/marqo-ai/marqo/pull/312). You can use these arrays as a collection of tags to filter on! See usage here.
  • Boost tensor fields during search (https://github.com/marqo-ai/marqo/pull/300). Weight fields as higher and lower relative to each other during search. Use this to get a mix of results that suits your use case. See usage here.
  • Weighted multimodal queries (https://github.com/marqo-ai/marqo/pull/307). You can now search with a dictionary of weighted queries. If searching an image index, these queries can be a weighted mix of image URLs and text. See usage here.
  • New GPT-Marqo integration example and article. Turn your boring user manual into a question-answering bot, with an optional persona, with GPT + Marqo!
  • Added new OpenCLIP models to Marqo (https://github.com/marqo-ai/marqo/pull/299)

Optimisations

  • Concurrent image downloads (https://github.com/marqo-ai/marqo/pull/281, https://github.com/marqo-ai/marqo/pull/311)
  • Blazingly fast fp16 ViT CLIP models (https://github.com/marqo-ai/marqo/pull/286). See usage here
  • Reduction of data transfer between Marqo and Marqo-os (https://github.com/marqo-ai/marqo/pull/300)
  • We see a 3.0x indexing speedup, and a 1.7x search speedup, using the new fp16/ViT-L/14 CLIP model, compared to the previous release using ViT-L/14.

Bug fixes

  • Fixed 500 error when creating an index while only specifying number_of_shards(https://github.com/marqo-ai/marqo/pull/293)
  • Fixed model cache management no parsing reranker model properties properly (https://github.com/marqo-ai/marqo/pull/308)

Contributor shout-outs

  • Thank you to our 2.3k stargazers
  • Thank you to @codebrain and others for raising issues.

Release 0.0.12

New features

  • Multilingual CLIP (https://github.com/marqo-ai/marqo/pull/267). Search images in the language you want! Marqo now incorporates open source multilingual CLIP models. A list of available multilingual CLIP models are available here.
  • Exact text matching (https://github.com/marqo-ai/marqo/pull/243, https://github.com/marqo-ai/marqo/pull/288). Search for specific words and phrases using double quotes (" ") in lexical search. See usage here.

Optimisations

  • Search speed-up (https://github.com/marqo-ai/marqo/pull/278). Latency reduction from Marqo-os indexes reconfigurations.

Contributor shout-outs

Thank you to our 2.2k stargazers and 80+ forkers!

Release 0.0.11

New features

  • Pagination (https://github.com/marqo-ai/marqo/pull/251). Navigate through pages of results. Provide an extensive end-user search experience without having to keep results in memory! See usage here
  • The /models endpoint (https://github.com/marqo-ai/marqo/pull/239). View what models are loaded, and on what device. This lets Marqo admins examine loaded models and prune unneeded ones. See usage here
  • The /device endpoint (https://github.com/marqo-ai/marqo/pull/239). See resource usage for the machine Marqo is running on. This helps Marqo admins manage resources on remote Marqo instances. See usage here
  • The index settings endpoint (/indexes/{index_name}/settings)(https://github.com/marqo-ai/marqo/pull/248). See the model and parameters used by each index. See usage here.
  • Latency log outputs (https://github.com/marqo-ai/marqo/pull/242). Marqo admins have better transparency about the latencies for each step of the Marqo indexing and search request pipeline
  • ONNX CLIP models are now available (https://github.com/marqo-ai/marqo/pull/245). Index and search images in Marqo with CLIP models in the faster, and open, ONNX format - created by Marqo's machine learning team. These ONNX CLIP models give Marqo up to a 35% speedup over standard CLIP models. These ONNX CLIP models are open sourced by Marqo. Read about usage here.
  • New simple image search guide (https://github.com/marqo-ai/marqo/pull/253, https://github.com/marqo-ai/marqo/pull/263).

Contributor shout-outs

  • ⭐️ We've just hit over 2.1k GitHub stars! ⭐️ So an extra special thanks to our stargazers and contributors who make Marqo possible.

Release 0.0.10

New features

  • Generic model support (https://github.com/marqo-ai/marqo/pull/179). Create an index with your favourite SBERT-type models from HuggingFace! Read about usage here
  • Visual search update 2. (https://github.com/marqo-ai/marqo/pull/214). Search-time image reranking and open-vocabulary localization, based on users' queries, is now available with the Owl-ViT model. Locate the part of the image corresponding to your query! Read about usage here
  • Visual search update 1. (https://github.com/marqo-ai/marqo/pull/214). Better image patching. In addition to faster-rcnn, you can now use yolox or attention based (DINO) region proposal as a patching method at indexing time. This allows localization as the sub patches of the image can be searched. Read about usage here.

Check out this article about how this update makes image search awesome.

Bug fixes

  • Fixed imports and outdated Python client usage in Wikipedia demo (https://github.com/marqo-ai/marqo/pull/216)

Contributor shout-outs

  • Thank you to @georgewritescode for debugging and updating the Wikipedia demo
  • Thank you to our 1.8k stargazers and 60+ forkers!

Release 0.0.9

Optimisations

  • Set k to limit to for Marqo-os search queries (https://github.com/marqo-ai/marqo/pull/219)
  • Reduced the amount of metadata returned from Marqo-os, on searches (https://github.com/marqo-ai/marqo/pull/218)

Non-breaking data model changes

  • Set default kNN m value to 16 (https://github.com/marqo-ai/marqo/pull/222)

Bug fixes

  • Better error messages when downloading an image fails (https://github.com/marqo-ai/marqo/pull/198)
  • Bug where filtering wouldn't work on fields with spaces (https://github.com/marqo-ai/marqo/pull/213), resolving https://github.com/marqo-ai/marqo/issues/115

Release 0.0.8

New features

  • Get indexes endpoint: GET /indexes (#181). Use this endpoint to inspect existing Marqo indexes. Read about usage here.
  • Non-tensor fields(#161). During the indexing phase, mark fields as non-tensor to prevent tensors being created for them. This helps speed up indexing and reduce storage for fields where keyword search is good enough. For example: email, name and categorical fields. These fields can still be used for filtering. Read about usage here.
  • Configurable preloaded models(#155). Specify which machine learning model to load as Marqo starts. This prevents a delay during initial search and index commands after Marqo starts. Read about usage here.
  • New example and article: use Marqo to provide context for up-to-date GPT3 news summary generation (#171, #174). Special thanks to @iain-mackie for this example.

Bug fixes and minor changes

  • Updated developer guide (#164)
  • Updated requirements which prevented Marqo being built as an arm64 image (#173)
  • Backend updated to use marqo-os:0.0.3 (#183)
  • Default request timeout has been increased from 2 to 75 seconds (#184)

Contributor shout-outs

  • For work on the GPT3 news summary generation example: @iain-mackie
  • For contributing the non-tensor fields feature: @jeadie
  • Thank you to our users who raise issues and give us valuable feeback
  • Thank you to our 1.4k+ star gazers and 50+ forkers!

Release 0.0.7

Bug fixes and minor changes

  • 429 (too many request errors) are propagated from Marqo-os to the user properly (#150)

Release 0.0.6

New features

  • Health check endpoint: GET /health. An endpoint that can be used to inspect the status of Marqo and Marqo's backend (Marqo-os) (#128). Read about usage here.
  • Marqo can be launched with environment variables that define limits around maximum number of fields per index, maximum document size and the maximum number of documents that can be retrieved (#135). Read about usage here.
  • README translations:
  • Chinese πŸ‡¨πŸ‡³ (by @wanliAlex, #133)
  • Polish πŸ‡΅πŸ‡± (by @MichalLuck, #136)
  • Ukrainian πŸ‡ΊπŸ‡¦ (by @dmyzlata, #138)
  • French πŸ‡«πŸ‡· (by @rym-oualha, #147)

Breaking API changes

  • The home / json response has been updated. If you have logic that reads the endpoint root, please update it (#128).
  • The Python client's add_documents() and update_documents() batch_size parameter has been replaced by server_batch_size and client_batch_size parameters (py-marqo#27), (py-marqo#28)

Non-breaking data model changes

  • Each text field just creates a top level Marqo-os text field, without any keywords (#135)
  • Very large fields get their tensor_facet keywords ignored, rather than Marqo-OS preventing the doc being indexed (#135)
  • Tensor facets can no longer have _id as a filtering field (#135)

Bug fixes and minor changes

  • FastAPI runs with better concurrency (#128)
  • Get documents by IDs and lexical search and no longer returns vectors if expose_facets isn't specified
  • Fixed batching bug in Python client (py-marqo#28)

Caveats

  • If a large request to add_documents or update_documents results in a document adding fields such that the index field limit is exceeded, the entire operation will fail (without resilience). Mitigate this sending add_documents and update_documents requests with smaller batches of documents.
  • For optimal indexing of large volumes of images, we recommend that the images are hosted on the same region and cloud provider as Marqo.

Contributor shout-outs

Release 0.0.5

Added Open CLIP models and added features to the get document endpoint.

New features

  • Added Open CLIP models (#116). Read about usage here
  • Added the ability to get multiple documents by ID (#122). Read about usage here
  • Added the ability to get document tensor facets through the get document endpoint (#122). Read about usage here

Release 0.0.4

Adding the attributesToRetrieve to the search endpoint and added the update documents endpoints

New features

  • Added the AttributesToRetrieve option to the search endpoint (55e5ac6)
  • Added the PUT documents endpoint (ce1306a)