PSearch: AI-Powered Product Search Platform

Overview

PSearch is an advanced product search platform built on Google Cloud Platform. It leverages Spanner's native hybrid search capabilities, combining traditional text search with vector similarity search (using embeddings generated via Vertex AI) to understand the semantic meaning behind user queries. This results in more accurate, contextually relevant product search results stored and queried directly within Spanner.

Core Features:

• Spanner Hybrid Search: Blends vector similarity and text search directly within Google Cloud Spanner for relevance and efficiency
• AI Enhancements: Includes AI-driven filter suggestions, content enrichment (text with Gemini, images with Imagen), and marketing content generation
• Product Filtering: Supports filtering by various attributes (categories, brands, price, etc.)
• Rules Engine: Allows administrators to configure search behavior (with planned implementation in Spanner)
• Scalable Architecture: Built with microservices on Google Cloud (Cloud Run)
• Modern UI: Responsive React-based user interface with AI feature integration

Project Status

| Component | Status | Notes | |-----------|--------|-------| | Core Search | ✅ Complete | Hybrid search combining vector and text search in Spanner | | Frontend UI | ✅ Complete | React-based UI with product search and filtering | | Ingestion | ✅ Complete | Spanner Reverse ETL | | GenAI | ✅ Complete | Content enrichment, image enhancement, and marketing generation | | Rules Engine | 🔄 Refactoring | Currently using mock data, migrating from Firestore to Spanner |

Architecture

PSearch utilizes a microservices architecture deployed on Google Cloud:

graph TD
    Client[Frontend UI] --> SearchAPI[Serving API]
    Client --> GenAI[GenAI Service]
    
    SearchAPI --> Spanner[(Spanner DB)]
    SearchAPI --> VertexAI[Vertex AI - Embeddings]
    
    GenAI --> VertexAI
    
    Ingestion[Ingestion Service] --> Spanner
    Ingestion --> VertexAI
    Ingestion --> BigQuery[(BigQuery)]

Core Services

• Frontend UI (React src/application/ui/): Provides the user interface for searching, filtering, viewing products, and interacting with AI enhancements.

• Serving API (Go/Gin src/psearch/serving/): Handles search requests, performs hybrid search directly against Spanner using its native vector and text search capabilities, generates query embeddings via Vertex AI, retrieves data from Spanner, and interacts with other services.

• Rules Management: Currently implemented as a mock service in the UI with localStorage persistence. Future implementation will use Spanner for rules storage.

• Ingestion Pipeline (Cloud Run Job/Python src/psearch/ingestion/): Processes product data (e.g., from BigQuery), generates embeddings using Vertex AI, and stores both structured data and embeddings directly into Spanner.

• GenAI Service (Cloud Run/Python src/psearch/gen_ai/): Provides generative AI capabilities:

  • Content enrichment with Gemini
  • Image enhancement with Gemini
  • Marketing content generation
  • Conversational search features

Datastores

• Spanner:

  • Primary storage for structured product data and vector embeddings
  • Enables integrated hybrid search
  • Planned storage for rules configuration (consolidating all data storage)

• BigQuery:

  • Used for analytics
  • Potential source for the ingestion pipeline

Technology Stack

• Frontend:

  • React with hooks for state management
  • Material-UI for components
  • Axios for API requests

• Backend:

  • Go (Gin) for high-performance Serving API
  • Python for AI-related services and data processing

• Databases:

  • Google Cloud Spanner with vector search capabilities
  • Google Cloud BigQuery for analytics

• AI/ML:

  • Google Vertex AI Embeddings for semantic search
  • Gemini 2.0 for text generation and content enrichment
  • Gemini for image enhancement and generation

• Infrastructure:

  • Google Cloud Run for serverless microservices
  • Google Cloud Build for CI/CD
  • Terraform for infrastructure as code
  • Docker for containerization

Repository Structure

├── docs/                            # Project documentation
│   ├── contributing.md              # Contribution guidelines
│   └── USAGE.md                     # Detailed usage instructions
├── src/
│   ├── analytics/                   # Analytics components
│   ├── application/                 # Frontend application
│   │   ├── ui/                      # React-based user interface
│   │   │   ├── src/                 # UI source code
│   │   │   │   ├── components/      # React components
│   │   │   │   └── services/        # Frontend services
│   │   │   └── public/              # Static assets
│   ├── iac/                         # Infrastructure as Code (Terraform)
│   │   ├── modules/                 # Terraform modules
│   │   │   ├── gen_ai/              # GenAI service infrastructure
│   │   │   ├── ingestion/           # Ingestion pipeline infrastructure
│   │   │   ├── search_api/          # Search API infrastructure
│   │   │   └── spanner/             # Spanner database infrastructure
│   │   └── main.tf                  # Main Terraform configuration
│   │   └── variables.tf             # Variables Terraform configuration
│   ├── psearch/                     # Backend services
│   │   ├── gen_ai/                  # GenAI service
│   │   │   └── services/            # AI service implementations
│   │   ├── ingestion/               # Data ingestion pipeline
│   │   │   └── services/            # Ingestion service implementations
│   │   └── serving/                 # Go-based search API
│   │       ├── cmd/                 # Entry points
│   │       └── internal/            # Internal implementation
│   └── tooling/                     # Development and operational tools
├── .gitignore                       # Git ignore file
├── LICENSE                          # Apache 2.0 License file
└── README.md                        # This file

Hybrid Search Implementation

PSearch's core search functionality leverages Spanner's ability to store and query both structured data and vector embeddings in a single database. The hybrid search combines:

1. Vector Similarity Search: Using APPROX_COSINE_DISTANCE with embeddings generated by Vertex AI
2. Full-Text Search: Using Spanner's native text search capabilities
3. Fusion Algorithm: Combining results using Reciprocal Rank Fusion (RRF)

This approach provides several advantages:

• More relevant results by combining semantic meaning with keyword matching
• Simplified architecture by using a single datastore
• Lower latency by avoiding synchronization between separate databases
• Scalability through Spanner's distributed architecture

Setup and Deployment

Prerequisites

• Google Cloud SDK (gcloud) installed and authenticated
• Terraform installed
• Docker installed
• Go 1.21+ installed (for Serving API)
• Python 3.9+ installed (for other services)
• Node.js and npm/yarn installed (for UI development/deployment)
• A Google Cloud Project with billing enabled and required APIs enabled (Spanner, Cloud Run, Cloud Build, Vertex AI, etc.)

Clone Repository

git clone <repository-url>
cd psearch

Configure Infrastructure

1. Navigate to the IaC directory:

   cd src/iac
   

2. Create a terraform.tfvars file (or set environment variables):

   project_id     = "your-gcp-project-id"
   region         = "your-gcp-region"        # e.g., "us-central1"
   project_number = "your-gcp-project-number"
   # Other variables as needed (see variables.tf files in modules)
   # Ensure Spanner instance meets minimum PU requirements for vector search (>=1000)
   

   Note: You can find your project number using:

   gcloud projects describe your-gcp-project-id --format='value(projectNumber)'
   

3. The Spanner schema (src/iac/modules/spanner/schema.sql) defines the necessary tables, vector columns, and indexes for hybrid search.

Provision Infrastructure

1. Initialize Terraform:

   terraform init
   

2. Apply the Terraform configuration:

   terraform apply
   

   Review the plan and type yes to confirm. This process may take several minutes.

Deploy Application Services

The terraform apply command provisions infrastructure and triggers Cloud Build pipelines defined in the respective service directories:

• src/psearch/serving/cloudbuild.yaml
• src/psearch/ingestion/cloudbuild.yaml
• src/psearch/gen_ai/cloudbuild.yaml
• src/application/ui/cloudbuild.yaml

These build the Docker images and deploy them to Cloud Run. Monitor build progress in the Google Cloud Console under Cloud Build.

Local Development

For local development:

1. UI Development:

   cd src/application/ui
   npm install
   npm start
   

   This will start a development server on http://localhost:3000.

2. Go API Development:

   cd src/psearch/serving
   go mod download
   go run cmd/server/main.go
   

   This will start the Go API server on http://localhost:8080.

3. GenAI Services Development:

   cd src/psearch/gen_ai
   python -m pip install -r requirements.txt
   python main.py
   

   This will start the GenAI services on http://localhost:8081.

Google Cloud Costs

When deploying this project, consider the cost estimation for Google Cloud Products.

This estimation supports the following volume:

• 500k SKUs for hybrid search
• 70k queries/day

| Service | GCP Product | Cost | |-----------|--------|-------| | Catalog | Cloud Spanner | $987.90 | | Searc