Tai: Torob AI Shopping Assistant

Tai is a 10-day hackathon project built for Torob's AI Shopping Assistant contest, its a conversational agent that helps customers find products naturally through chat.

Overview

The system combines semantic search, LLM reasoning, and multi-agent orchestration to handle shopping scenarios. Built with FastAPI and Elasticsearch, it processes Persian language queries and returns structured product recommendations.

You can try Tai at tai.meower1.dev

---

Architecture

Request Flow:

# app/api/chat.py
@router.post("/chat")
async def chat_endpoint(chat_request: ChatRequest):
    messages = [message.model_dump() for message in chat_request.messages]
    return build_chat_response(chat_request.chat_id, messages)

All requests hit a single /chat endpoint that routes to specialized agents based on query intent.

---

Scenarios & Implementation

Scenario 1: Product Search

Finds products using semantic vector search on embeddings stored in Elasticsearch.

Example:

Input: لطفاً دراور چهار کشو (کد D14) را برای من تهیه کنید.
Output: Product key: bmubxu

How it works:

# app/services/product_search.py
class SearchAgent:
    def process_query(self, raw_query: str, top_k: int = 3):
        # 1. Generate query embedding
        embedding = self._get_embedding(cleaned_query)

        # 2. Vector similarity search
        results = self.es_client.search(
            index="products_embeddings",
            knn={
                "field": "embedding",
                "query_vector": embedding,
                "k": top_k,
                "num_candidates": 50
            }
        )

        return results

The agent generates embeddings using OpenAI's API and queries Elasticsearch's k-NN index for semantically similar products.

Scenario 2: Product Feature Inquiry

Answers questions about specific product attributes by retrieving product data from Elasticsearch and using LLM to extract the exact answer.

Example:

Input: عرض پارچه تریکو جودون 1/30 لاکرا گردباف نوریس به رنگ زرد طلایی چقدر است؟
Output: 1.18 meter

How it works:

# app/services/product_qna.py
class ProductFeatureAgent:
    def answer_feature_question(self, query: str):
        # 1. Find relevant product
        product = self._search_product(query)

        # 2. Extract feature context
        context = {
            "name": product["name_fa"],
            "features": product.get("extra_features", {}),
            "category": product.get("category_name_fa")
        }

        # 3. LLM extracts specific answer
        response = self.openai_client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{
                "role": "system",
                "content": "Extract the specific feature value from context."
            }, {
                "role": "user",
                "content": f"Query: {query}\nContext: {context}"
            }]
        )

        return self._normalize_answer(response)

The agent uses targeted prompts to extract numeric values, dimensions, or categorical features from product metadata.

Scenario 3: Seller Information

Queries the members index (shop-specific product variants) to find pricing and availability data.

Example:

Input: کمترین قیمت در این پایه برای گیاه طبیعی بلک گلد بنسای نارگل کد ۰۱۰۸ چقدر است؟
Output: 275,000

How it works:

# app/services/product_qna.py
def _find_min_price(self, base_random_key: str):
    # Query members index for shop variants
    results = self.es_client.search(
        index="members",
        query={
            "term": {"base_random_key": base_random_key}
        },
        aggs={
            "min_price": {"min": {"field": "price"}}
        }
    )

    return results["aggregations"]["min_price"]["value"]

Uses Elasticsearch aggregations to compute min/max/average prices across all shops selling the product.

Scenario 4: Guided Shopping

Multi-turn conversation that narrows down requirements through dialog before recommending a product.

Example:

User: من دنبال یه میز تحریر هستم که برای کارهای روزمره و نوشتن مناسب باشه.
Agent: چه رنگی دوست دارید؟
User: سفید
Agent: بودجه‌تان چقدر است؟
...
Output: Recommended product after gathering preferences

How it works:

# app/services/interactive_assistant.py
class InteractiveAssistant:
    def process_conversation(self, messages: List[Dict], session_state: Dict):
        # Track collected attributes across turns
        collected = session_state.get("attributes", {})

        # Identify what's still missing
        missing = self._identify_missing_attributes(query, collected)

        if missing:
            # Ask next clarifying question
            return self._ask_for_attribute(missing[0])
        else:
            # All info gathered, execute search
            return self._final_recommendation(collected)

Maintains conversation state to track which attributes have been collected and generates targeted follow-up questions.

Scenario 5: Product Comparison

Compares products and provides reasoned recommendations based on user priorities.

Example:

Input: کدام یک از این ماگ‌ها برای کودکان مناسب‌تر است؟
Output: Product A is better because it features cartoonish designs that appeal to children.

How it works:

# app/services/product_comparison.py
class ProductComparisonAgent:
    def compare_products(self, product_ids: List[str], query: str):
        # 1. Fetch product details in parallel
        products = self._fetch_products_parallel(product_ids)

        # 2. Extract comparable features
        comparison_data = {
            pid: {
                "name": p["name_fa"],
                "features": p.get("extra_features", {}),
                "price": self._get_avg_price(pid)
            }
            for pid, p in products.items()
        }

        # 3. LLM reasoning
        response = self.openai_client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{
                "role": "system",
                "content": "Compare products and recommend the best match."
            }, {
                "role": "user",
                "content": f"Query: {query}\nProducts: {comparison_data}"
            }]
        )

        return {
            "message": response.choices[0].message.content,
            "base_random_keys": [winner_id]
        }

The agent fetches all relevant product data, then uses LLM reasoning to evaluate trade-offs and make a justified recommendation.

Scenario 6: Image Recognition

Identifies the main object in an uploaded image using vision models.

Example:

Input: شیء و مفهوم اصلی در تصویر چیست? (with image)
Output: پتو

How it works:

# app/services/image_object_identifier.py
class ImageObjectIdentifier:
    def identify_object(self, base64_image: str):
        # Decode and validate image
        image_data = base64.b64decode(base64_image)

        # Vision model analysis
        response = self._client.chat.completions.create(
            model="gpt-4.1-mini",
            messages=[{
                "role": "user",
                "content": [
                    {"type": "text", "text": "What is the main object in Persian?"},
                    {"type": "image_url", "image_url": {
                        "url": f"data:image/jpeg;base64,{base64_image}"
                    }}
                ]
            }]
        )

        return response.choices[0].message.content

Uses gpt-4.1-mini's vision capabilities to analyze images and return Persian object names.

Scenario 7: Image-Based Product Search

Finds matching products from uploaded images using perceptual hashing and visual similarity.

Example:

Input: یک محصول مرتبط مناسب با تصویر به من بدهید. (with image)
Output: Product key: vdbkdf

How it works:

# app/services/image_Product_Finder.py
class UnifiedImageSearchPipeline:
    def search(self, base64_image: str, top_k: int = 3):
        # 1. Generate perceptual hash
        image_hash = self._compute_phash(base64_image)

        # 2. Search Elasticsearch by image hash
        candidates = self.es_client.search(
            index="based_products_image_hash",
            query={
                "match": {"image_hash": image_hash}
            },
            size=100
        )

        # 3. Visual similarity re-ranking
        descriptions = [self._describe_image(base64_image)]
        for candidate in candidates:
            descriptions.append(candidate["description"])

        embeddings = self._embed_batch(descriptions)
        similarities = cosine_similarity([embeddings[0]], embeddings[1:])

        # Return top matches
        return self._rank_by_similarity(candidates, similarities, top_k)

Combines hash-based retrieval with vision model embeddings for robust image-to-product matching.

---

Technical Stack

Backend:

• FastAPI for async API handling
• Elasticsearch for vector search and aggregations
• OpenAI gpt-4.1-mini for reasoning and vision tasks
• Pydantic for strict request validation

Data Pipeline:

• Product embeddings: 1536-dimensional vectors pre-computed and indexed
• Members index: 1.95M shop-product variants for pricing queries
• Image hashing: Perceptual hashes stored alongside product images

Response Format:

# All responses follow this contract
{
    "message": str | null,              # User-facing text
    "base_random_keys": List[str],      # Product IDs (max 10)
    "member_random_keys": List[str]     # Shop variant IDs (max 10)
}

---

Quick Start

Prerequisites:

• Docker and Docker Compose
• Elasticsearch credentials (cloud instance)
• OpenAI API key or GitHub Models access

Setup:

# Clone and configure
git clone [repository-url]
cd tai
cp .env.example .env
# Edit .env with your credentials

# Run with Docker (required)
docker compose up --build

# Test the API
curl -X POST http://127.0.0.1:8000/chat \
  -H "Content-Type: application/json" \
  -d '{
    "chat_id": "test-session",
    "messages": [{
      "type": "text",
      "content": "ping"
    }]
  }'

Environment Variables:

ELASTICSEARCH_URL=https://your-es-cluster.com
ELASTICSEARCH_USERNAME=your-username
ELASTICSEARCH_PASSWORD=your-password
GITHUB_TOKEN=your-openai-api-key
GITHUB_MODELS_BASE_URL=https://models.github.ai/inference

---

Project Structure

app/
├── main.py              # FastAPI app with middleware
├── schemas.py           # Pydantic request/response models
├── api/
│   └── chat.py         # Chat endpoint router
└── services/
    ├── chat_logic.py            # Main orchestration logic
    ├── product_search.py        # Vector search agent
    ├── product_qna.py           # Feature extraction agent
    ├── product_comparison.py    # Comparison reasoning agent
    ├── interactive_assistant.py # Multi-turn conversation handler
    ├── image_object_identifier.py  # Vision model integration
    └── image_Product_Finder.py    # Image-to-product matching

---

Development Notes

• Persian NLP: All user interactions in Farsi with proper RTL handling
• Functional services: Pure functions in service layer, side effects in API layer
• Logging: Auto-configured with Hijri timestamps in logs/tai_{timestamp}.log
• Error handling: Returns user-friendly Persian error messages
• Docker-first: Always use docker compose up --build for development

---

Built for the Torob AI Shopping Assistant Hackathon by MazAmin • Live Demo
Special thanks to Iterm0 for building the awesome frontend