15. Full text search using Elasticsearch for blazingly fast search
The Evolution of Search in E-commerce
Initial Challenges in Database Queries
- In 2005, a software engineer at an e-commerce company faced the challenge of querying a rapidly growing database with around 5,000 products.
- A basic SQL query was used to search for products, such as
SELECT * FROM products WHERE name LIKE '%laptop%', which initially returned results quickly.
- As the product count surged to millions, the same query began taking up to 30 seconds, leading to customer frustration and demands for optimization.
Requirements for Enhanced Search Functionality
- Beyond speed improvements, there were new requirements for smarter searches that prioritized relevance; for example, showing a MacBook Pro over unrelated items like laptop bags.
- The system needed to handle common typos in user queries effectively while still returning relevant results.
Introduction of Advanced Search Solutions
- These challenges led to the development of advanced search engines like Elasticsearch that could meet these evolving needs.
- Traditional relational databases functioned like librarians searching through every book one by one without understanding relevance or context.
Limitations of Traditional Databases
- The librarian analogy illustrates how traditional databases scan each entry sequentially, resulting in slow performance and lack of relevance-based sorting.
- Without an understanding of what constitutes "relevant" results, databases might return irrelevant matches indiscriminately.
The Rise of Inverted Indexing
Conceptual Shift in Searching
- To address inefficiencies from information overload during the early 2000s, researchers proposed flipping the search paradigm using inverted indexing.
- Instead of scanning documents for terms post-query, inverted indexing allows pre-indexing words so that searches can directly reference where terms appear across documents.
Implementation and Benefits
- An inverted index maps words to their occurrences within books (or documents), drastically improving search speed and accuracy by eliminating unnecessary scans.
- This method enhances both retrieval speed and relevance scoring—key features utilized by tools like Elasticsearch powered by Apache Lucene.
Relevance Scoring Mechanisms
Understanding Relevance Scoring
- Elasticsearch employs relevance scoring algorithms (like BM25), which consider factors such as term frequency and document length when ranking results.
- Field boosting is another feature where terms appearing in titles are weighted more heavily than those found in descriptions or content areas.
Practical Applications
- Using Elasticsearch not only speeds up searches but also ensures users receive meaningful results tailored to their queries rather than just any matching entries.
Typo Tolerance and User Experience
Enhancing User Interaction with Search Features
- Tools like Elasticsearch provide typo tolerance capabilities that allow systems to interpret user intent even when errors occur during input (e.g., "trending" vs. "treading").
Comparison Between Technologies
- When building search functionalities within applications, developers can choose between PostgreSQL's full-text search capabilities or leveraging existing Elasticsearch setups already integrated into their tech stacks.
Demonstrating Performance Differences
Benchmarking Traditional vs. Modern Searches
- A project comparing PostgreSQL against an Elasticsearch instance highlighted differences in query execution times under similar conditions using a dataset containing reviews and sentiments.
Conclusion on Technology Choices
- Ultimately , choosing between PostgreSQL’s built-in features versus dedicated solutions like Elasticsearch depends on specific use cases , especially regarding performance needs .
Overview of Data Streaming and Search Functionality
Introduction to Data Handling
- The process involves running a query that retrieves data based on the presence of specific terms, streaming results to the front end as they become available.
Elastic Search Implementation
- A search is executed through an Elastic Search index named "reviews," converting search terms to lowercase for broader matching and enhancing searches with default fields.
Performance Comparison
- Results from both Elastic Search and database queries are returned in real-time; however, the front end does not display any immediate changes during this process.
Demo: Query Execution Times
- In a demo, searching for "laptop" took approximately 1 second with Elastic Search compared to around 3 seconds for database queries.
Speed Demonstration with Different Queries
- Another search yielded results in 500 milliseconds via Elastic Search while the database took about 7.5 seconds, highlighting significant performance differences despite similar result counts.
Importance of Full Text Search Tools
Recommendations for Backend Engineers
- For use cases involving search functionalities, tools like Elastic Search or PostgreSQL's full-text search should be part of a backend engineer's toolkit without needing mastery over them.
Database Knowledge vs. Elastic Search Familiarity
- Mastery of databases is crucial as it constitutes a significant portion (99%) of backend codebases; understanding how to optimize and work with databases is essential.
Learning Curve for Optimization
- While basic usage of Elastic Search can often be achieved through documentation snippets, deeper optimization requires further study beyond initial implementations.
Turn any video into a summary like this
YouTube links, meetings, lectures. With transcripts, search, and chat.





