Epic: Configuration + Adapter Discovery System

🎯 Overview

Enable per-project MCP adapter configuration and discovery with automatic Cursor integration, centralized storage, and a unified CLI experience.

Key Insight: Users install @lytics/dev-agent globally once, then configure adapters per-project. Each project's .dev-agent/config.json defines which adapters to load, and dev-agent mcp init automatically adds the project to Cursor's MCP config.

🏗️ Architecture

Core Components

Centralized Storage - All repository indexes stored globally in ~/.dev-agent/indexes/ (enables cross-repo search, clean project dirs)
Project Configuration - Per-project .dev-agent/config.json defines adapters and settings
Adapter Discovery - Automatic discovery and loading of built-in, npm, and local adapters
CLI Management - Unified commands for MCP setup and adapter management

User Flow

# One-time global install
npm install -g @lytics/dev-agent

# Per-project setup
cd my-project
dev-agent init              # Create config + index repository
dev-agent mcp init         # Add to Cursor MCP config
dev-agent adapter install @company/mcp-adapter-jira  # Install custom adapter

📋 Sub-Issues

This epic is broken down into three focused sub-issues:

Foundation: Storage + Config System #52: Foundation - Storage + Config System - Centralized storage, config schema, lazy loading
Adapters: Discovery + Loading System #53: Adapters - Discovery + Loading System - Adapter discovery algorithm, loading mechanism
UX: CLI Management Commands #54: UX - CLI Management Commands - User-facing commands for MCP + adapter management

✅ Success Criteria

Users can run dev-agent init && dev-agent mcp init to fully configure a project
Custom adapters can be installed via dev-agent adapter install
Team members can clone and get same adapter setup
Different projects can have different adapters from same global install
No manual editing of ~/.cursor/mcp.json required
Indexes stored centrally, enabling cross-repository search
Built-in adapters can be disabled per-project

🔗 Related Issues

Depends on: Explore + GitHub Adapters #30 (Explore + GitHub Adapters) - establishes adapter patterns
Blocks: Documentation + Examples #32 (Documentation) - need to document new setup flow
Blocks: Integration Testing + Release #33 (Integration Testing) - need to test adapter discovery
Related: feat(mcp): add structured metadata field to MCP tool responses #51 (Metadata field) - will enhance adapter responses

Estimate: 3-4 days (across 3 sub-issues)
Priority: High (blocks documentation and release)
Milestone: v1.0 - MCP Integration Complete

💾 Storage Strategy: Centralized Indexes

Overview

All repository indexes are stored globally in ~/.dev-agent/indexes/ rather than per-project. This enables:

✅ Clean project directories (no large files to gitignore)
✅ Cross-repository search and analysis
✅ Shared indexes across clones
✅ Easy storage management and cleanup
✅ Survives project moves/deletions

Directory Structure

~/.dev-agent/
  indexes/
    a1b2c3d4/              # Frontend (git remote hash)
      vectors.lance        # Code vectors
      github-state.json    # GitHub cache
      metadata.json        # Repository metadata
    e5f6g7h8/              # Backend
      vectors.lance
      github-state.json
      metadata.json
  
  cache/
    embeddings-model/      # Shared ML model (~100MB)
    github-api/            # GitHub API response cache
  
  config/
    global.json            # Global settings (optional)

Storage Location Algorithm

function getStoragePath(repositoryPath: string): string {
  // 1. Try git remote (stable across clones)
  const gitRemote = await getGitRemote(repositoryPath);
  
  if (gitRemote) {
    // Normalize: git@github.com:company/repo.git → company/repo
    const normalized = normalizeGitRemote(gitRemote);
    const hash = crypto.createHash('md5')
      .update(normalized)
      .digest('hex')
      .slice(0, 8);
    
    return path.join(os.homedir(), '.dev-agent/indexes', hash);
  }
  
  // 2. Fallback: absolute path hash (for non-git repos)
  const pathHash = crypto.createHash('md5')
    .update(path.resolve(repositoryPath))
    .digest('hex')
    .slice(0, 8);
  
  return path.join(os.homedir(), '.dev-agent/indexes', pathHash);
}

Metadata File

Each index includes metadata for identification:

// ~/.dev-agent/indexes/a1b2c3d4/metadata.json
{
  "version": "1.0",
  "repository": {
    "path": "/Users/you/workspace/frontend",
    "remote": "git@github.com:company/frontend.git",
    "branch": "main",
    "lastCommit": "abc123..."
  },
  "indexed": {
    "timestamp": "2025-11-25T12:00:00Z",
    "files": 243,
    "components": 1847,
    "size": 52428800
  },
  "config": {
    "languages": ["typescript", "javascript"],
    "excludePatterns": ["**/node_modules/**"]
  }
}

Memory Management

class MCPServer {
  private indexer?: RepositoryIndexer;
  private lastAccessed = Date.now();
  private readonly IDLE_TIMEOUT = 5 * 60 * 1000; // 5 minutes
  
  async ensureIndexer(): Promise<RepositoryIndexer> {
    if (!this.indexer) {
      // Lazy load on first use
      const storagePath = getStoragePath(this.repositoryPath);
      
      this.indexer = new RepositoryIndexer({
        repositoryPath: this.repositoryPath,
        vectorStorePath: path.join(storagePath, 'vectors.lance'),
      });
      
      await this.indexer.initialize();
      this.logger.info('Loaded indexes', { storagePath });
    }
    
    this.lastAccessed = Date.now();
    return this.indexer;
  }
  
  // Auto-unload after idle period
  startIdleMonitor() {
    setInterval(() => {
      const idleTime = Date.now() - this.lastAccessed;
      
      if (idleTime > this.IDLE_TIMEOUT && this.indexer) {
        this.indexer.close();
        this.indexer = undefined;
        this.logger.info('Unloaded indexes (idle timeout)', {
          idleMinutes: Math.floor(idleTime / 60000)
        });
      }
    }, 60000); // Check every minute
  }
}

CLI Storage Management

# Show storage usage and repository list
dev-agent storage info

Storage Location: ~/.dev-agent/indexes/
Total Size: 230 MB

Repositories:
  ID        Repository          Size    Last Indexed    Last Accessed
  a1b2c3d4  company/frontend    50 MB   2 hours ago     Active
  e5f6g7h8  company/backend     80 MB   5 mins ago      Active
  x9y8z7w6  old-project         100 MB  30 days ago     30 days ago

# Clean stale indexes
dev-agent storage clean

? Clean indexes not accessed in: (Select)
  ○ 7 days
  ○ 30 days
  ● 90 days

Found 1 stale index:
  x9y8z7w6  old-project  100 MB  Last accessed: 90 days ago

? Remove stale indexes? (Y/n) y
✓ Removed 1 index (100 MB freed)

# Force rebuild specific repository
dev-agent storage rebuild

? Which repository:
  ● Current (company/frontend)
  ○ All repositories
  ○ Select multiple

✓ Rebuilt company/frontend (50 MB, 243 files, 1847 components)

# Manually add/link repository
dev-agent storage link /path/to/other/clone

✓ Linked to existing index a1b2c3d4 (company/frontend)
  Same git remote detected, using shared storage

# Export indexes (backup or transfer)
dev-agent storage export frontend-indexes.tar.gz

✓ Exported company/frontend to frontend-indexes.tar.gz (50 MB)

# Import indexes (restore or transfer)
dev-agent storage import frontend-indexes.tar.gz

✓ Imported company/frontend to ~/.dev-agent/indexes/a1b2c3d4/

Cross-Repository Features

Centralized storage enables powerful cross-repo capabilities:

1. Search Across All Repositories

# Search all indexed repositories
dev-agent search --all "authentication middleware"

Found in 3 repositories:

company/frontend (2 results):
  • src/lib/auth.ts:45
    export function authMiddleware()
  
company/backend (5 results):
  • src/middleware/auth.ts:12
    export const authenticate = async ()
  • src/guards/jwt.guard.ts:23
    class JwtAuthGuard implements CanActivate

company/shared-lib (1 result):
  • packages/auth/src/index.ts:8
    export { AuthService }

2. Find Code Patterns Across Organization

# Find similar implementations across repos
dev-agent explore pattern --all "rate limiting"

Pattern found in 4 repositories:

company/api-gateway:
  • src/middleware/rate-limit.ts (Redis-based)
  
company/backend:
  • src/utils/throttle.ts (In-memory cache)

company/microservice-a:
  • lib/limiter.js (Token bucket)

# Suggests: Consolidate to shared package?

3. Dependency Analysis Across Organization

# See which repos use a package
dev-agent analyze dependencies "express"

express@4.18.2 used in:
  ✓ company/backend
  ✓ company/api-gateway
  ✓ company/admin-api

express@4.17.1 (outdated) used in:
  ⚠️ company/legacy-service
  ⚠️ company/old-api

Recommendation: Upgrade 2 repositories to 4.18.2

4. Monorepo Intelligence

# Understand monorepo structure
dev-agent analyze monorepo

Analyzing company/monorepo:
  Packages: 23
  Shared dependencies: 45
  Circular dependencies: 0 ✓
  
Package groups:
  • @company/ui-* (8 packages)
  • @company/api-* (5 packages)
  • @company/core-* (10 packages)

Implementation Tasks

Update RepositoryIndexer to accept storage path
Implement storage path resolver (git remote → hash)
Create metadata.json for each index
Implement lazy loading in MCP server
Add idle timeout and auto-unload
Create dev-agent storage command group
Add cross-repository search capability
Handle index migration (move existing to centralized)
Update documentation

Migration Path

For users with existing project-local indexes:

# Automatically detect and migrate
dev-agent storage migrate

Found local indexes:
  ✓ /Users/you/frontend/.dev-agent/vectors.lance (50 MB)
  ✓ /Users/you/backend/.dev-agent/vectors.lance (80 MB)

? Migrate to centralized storage? (Y/n) y

✓ Migrated frontend → ~/.dev-agent/indexes/a1b2c3d4/
✓ Migrated backend → ~/.dev-agent/indexes/e5f6g7h8/
✓ Updated configs to use centralized storage

? Remove old local indexes? (Y/n) y
✓ Removed local indexes (130 MB freed from projects)

Benefits Recap

For Users:

No large files in Git repos
Indexes survive project deletion
Fast when switching between clones
Easy cleanup of old projects

For Teams:

Cross-repository search
Organization-wide code patterns
Dependency analysis
Standardization insights

For System:

Centralized management
Deduplication potential
Easier backup/export
Clear storage monitoring

Status Update:

Parts of this epic have been implemented:

✅ Centralized storage (~/.dev-agent/indexes/)
✅ Per-project config (.dev-agent/config.json)
✅ dev init creates config
✅ dev mcp install --cursor adds to Cursor config
✅ Built-in adapters can be enabled/disabled

Not yet implemented:

🔲 Custom adapter discovery (npm packages)
🔲 dev adapter install command
🔲 Dynamic adapter loading at runtime

This is lower priority now that the core MCP integration works well. Moving to backlog.

docs: add comprehensive examples and close stale issues

Navigation Menu

Repository navigation

Epic: Configuration + Adapter Discovery System #31

🎯 Overview

🏗️ Architecture

Core Components

User Flow

📋 Sub-Issues

✅ Success Criteria

🔗 Related Issues

💾 Storage Strategy: Centralized Indexes

Overview

Directory Structure

Storage Location Algorithm

Metadata File

Memory Management

CLI Storage Management

Cross-Repository Features

1. Search Across All Repositories

2. Find Code Patterns Across Organization

3. Dependency Analysis Across Organization

4. Monorepo Intelligence

Implementation Tasks

Migration Path

Benefits Recap

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Epic: Configuration + Adapter Discovery System #31

Description

🎯 Overview

🏗️ Architecture

Core Components

User Flow

📋 Sub-Issues

✅ Success Criteria

🔗 Related Issues

Activity

prosdev commented on Nov 25, 2025

💾 Storage Strategy: Centralized Indexes

Overview

Directory Structure

Storage Location Algorithm

Metadata File

Memory Management

CLI Storage Management

Cross-Repository Features

1. Search Across All Repositories

2. Find Code Patterns Across Organization

3. Dependency Analysis Across Organization

4. Monorepo Intelligence

Implementation Tasks

Migration Path

Benefits Recap

prosdev commented on Nov 27, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions