Skip to main content

Hybrid Search Implementation

IntelliQ implements a sophisticated hybrid search system that combines the strengths of both keyword-based (lexical) search and semantic (vector) search to provide highly relevant results when users search for quizzes. Hybrid search combines two powerful search methodologies:
  1. Keyword Search (Full-Text Search): Matches specific words or phrases in content, excellent for finding exact matches.
  2. Semantic Search (Vector Search): Understands the meaning and context of a query, finding conceptually related content even when keywords don’t match.
By combining these approaches, hybrid search delivers more comprehensive and relevant results than either method alone.

Architecture Overview

The hybrid search implementation in IntelliQ follows these key steps:
  1. User Input: The user enters a search query in the UI
  2. Query Processing: The query is processed in two ways:
    • As text for keyword search
    • Converted to a vector embedding for semantic search
  3. Dual Search Execution: Both search methods run in parallel in PostgreSQL
  4. Result Fusion: Results are combined using Reciprocal Rank Fusion (RRF)
  5. Result Presentation: The final ranked results are returned to the user

Technical Implementation

Database Function

At the core of our hybrid search is a PostgreSQL function that performs both search types and combines the results:

Key Components Explained

1. Embedding Generation

We use OpenAI’s text-embedding-3-small model to generate embeddings for both quizzes and search queries:
For quizzes, we generate embeddings asynchronously using a queue system to avoid slowing down quiz creation:
The full-text search component uses PostgreSQL’s built-in text search capabilities:
  • to_tsvector: Converts text to a searchable format
  • websearch_to_tsquery: Parses the user’s query into a format suitable for searching
  • setweight: Assigns different weights to different fields (title, description, topics)
  • ts_rank_cd: Ranks results based on relevance
We prioritize matches in the title (weight A), then description (weight B), and finally topics (weight C). The semantic search component uses pgvector’s similarity search:
  • <#> operator: Calculates the inner product distance between embeddings
  • Smaller distances indicate higher similarity

4. Reciprocal Rank Fusion (RRF)

RRF combines the rankings from both search methods:
Where:
  • rank_ix: The position of each result in its respective list
  • rrf_k: A constant (default: 60) that smooths the impact of high rankings
  • full_text_weight and semantic_weight: Control the relative importance of each search method

API Implementation

Our API endpoint handles the search request, generates the embedding, and calls the database function:

Frontend Implementation

The frontend provides a seamless search experience:
  1. Improved Relevance: Finds both exact keyword matches and conceptually related content
  2. Better Recall: Captures results that might be missed by either method alone
  3. Enhanced User Experience: Users find what they’re looking for even if they don’t use exact terminology
  4. Flexibility: Weights can be adjusted to favor either keyword or semantic search

Performance Considerations

  1. Indexing: Both search methods use appropriate indexes:
    • GIN index for full-text search
    • HNSW index for vector search
  2. Asynchronous Embedding Generation: Embeddings are generated in the background to avoid slowing down quiz creation
  3. Pagination: Results are paginated to limit the amount of data transferred
  4. Caching: Frequently searched queries could be cached (future enhancement)

Future Enhancements

  1. Personalized Ranking: Adjust result ranking based on user preferences and history
  2. Multi-language Support: Extend search capabilities to multiple languages
  3. Faceted Search: Allow filtering of search results by various attributes
  4. Query Expansion: Automatically expand queries to include related terms
  5. Performance Optimization: Further optimize the search algorithm for larger datasets

Conclusion

The hybrid search implementation in IntelliQ provides a powerful and flexible way for users to find relevant quizzes. By combining the strengths of keyword and semantic search, we deliver a superior search experience that understands both the exact words and the meaning behind a user’s query. Written by Ricky Raveanu