Emergence of Vector Databases with AI wave.

With release of GPT models, Vector databases are gaining huge popularity and are coming into limelight. Vector databases will power next generation AI applications and will serve as long term memory to LLM (large language model) based applications and workflows.

Why Vector Databases?

In early days of internet, data was mostly structured data, which could easily be store and managed by relational databases. Relational databases were designed to store and search data in tables.

As the internet grew and evolved, unstructured data becomes so big in form of text, images, videos and became problematic to analyze, query and infer insight from the data. Unlike structured data, these are not easy to manage by relational databases. Imagine a scenario, where trying to search a similar shirt from collection of images of shirts, this would be impossible for relational databases purely from raw pixel values of shirt images.

This brings us to vector databases. Unstructured data has led to a steady rise in the use of machine learning models trained to understand such data. word2vec, a natural language processing (NLP) algorithm which uses a neural network to learn word associations, is a well-known early example of this. The word2vec model is capable of turning single words into a list of floating-point values, or vectors. Due to the way models is trained, vectors which are close to each other represent words which are similar to each other, hence the term…