From 45ac47f674f63a012d1e526750ca764dd650349b Mon Sep 17 00:00:00 2001 From: Spencer Grimes Date: Fri, 20 Feb 2026 22:35:52 -0600 Subject: [PATCH] "vault backup: 2026-02-20 22:35:52 from Flow" --- .../Personal Projects/Blight/AGENTS.md | 622 ------------------ 1 file changed, 622 deletions(-) delete mode 100644 20-Knowledge/Personal Projects/Blight/AGENTS.md diff --git a/20-Knowledge/Personal Projects/Blight/AGENTS.md b/20-Knowledge/Personal Projects/Blight/AGENTS.md deleted file mode 100644 index 08537ce..0000000 --- a/20-Knowledge/Personal Projects/Blight/AGENTS.md +++ /dev/null @@ -1,622 +0,0 @@ -## Project Overview - -Building a centralized AI-powered hub that connects personal devices, processes information, and makes intelligent decisions using Google Gemini API. The system manages markdown notes (Obsidian vault), sends reminders, and executes automations across 20-30 personal devices. - -## Core Requirements - -- **Scale**: Personal use, 20-30 devices maximum -- **Deployment**: Docker Compose on Linux server -- **AI**: Google Gemini API with caching and rate limiting -- **Notes**: Obsidian vault synced via Git (read-write access) -- **Network**: All devices connected via VPN (bidirectional communication) -- **Notifications**: Primary channel is Discord webhooks -- **Integrations**: Home Assistant (optional middleware) -- **Storage**: 7-day data retention for events/logs - -## Technology Stack - -### Hub (Docker Compose Services) -- **API**: FastAPI (Python) -- **Database**: PostgreSQL 16 -- **Vector DB**: ChromaDB (note embeddings) -- **Cache/Queue**: Redis 7 -- **Worker**: Background task processor - -### Device Agents -- **Language**: Python (cross-platform) -- **Deployment**: - - Linux: systemd service - - Windows: Task Scheduler / Windows Service - - Mobile: Termux (Android) / Shortcuts (iOS) - -### External Services -- **AI**: Google Gemini API -- **Notifications**: Discord webhooks -- **Home Automation**: Home Assistant REST API -- **Code Repository**: Gitea (local, for agent updates) -- **Version Control**: Git (Obsidian vault sync) - -## Architecture Decisions - -### Note Management -- Hub has **read-write** access to Obsidian vault -- Git workflow: pull → process → commit → push -- Reminder lines **deleted** after processing (clean removal) -- Git conflicts trigger Discord alerts for manual resolution - -### Device Communication -- Devices authenticate with API keys -- Agents report events and queue locally if hub offline -- Hub can send commands to devices (predefined command set) -- Agent state is authoritative over hub's cached state - -### AI Integration (Gemini) -- 24-hour cache for similar queries (Redis) -- Rate limit: 100 requests/hour -- Circuit breaker: 5 failures = 15min pause -- Fallback: Simple regex parsing if Gemini unavailable - -### Data Management -- Events/logs: 7-day retention, auto-cleanup -- Completed reminders: Delete after processing -- Agent timezones: Translate to hub timezone -- Incremental note indexing using `git diff` - -### Automation Rules -- YAML configuration files (version controlled) -- Home Assistant handles repeatable triggers -- Hub handles one-off events and AI decisions - -## Project Structure - -``` -personal-ai-hub/ -├── AGENTS.md # This file - update as you progress -├── README.md # User-facing documentation -├── docker-compose.yml # Service orchestration -├── .env.example # Environment variables template -├── .gitignore -│ -├── hub/ -│ ├── Dockerfile -│ ├── requirements.txt -│ ├── main.py # FastAPI application entry point -│ ├── worker.py # Background task processor -│ ├── models.py # Database models (SQLAlchemy) -│ ├── schemas.py # Pydantic schemas for API -│ ├── config.yaml.example # Automation rules template -│ ├── alembic/ # Database migrations -│ │ ├── alembic.ini -│ │ └── versions/ -│ ├── api/ -│ │ ├── __init__.py -│ │ ├── devices.py # Device registration endpoints -│ │ ├── events.py # Event submission/retrieval -│ │ ├── reminders.py # Reminder management -│ │ ├── webhooks.py # Webhook endpoints -│ │ ├── health.py # Health check endpoint -│ │ └── admin.py # Admin/management endpoints -│ ├── services/ -│ │ ├── __init__.py -│ │ ├── gemini.py # Gemini API client -│ │ ├── notes.py # Markdown parser & git operations -│ │ ├── reminders.py # Reminder scheduling logic -│ │ ├── webhooks.py # Discord/webhook sender -│ │ ├── home_assistant.py # HA integration -│ │ ├── automation.py # YAML rule engine -│ │ ├── vector_store.py # ChromaDB interface -│ │ └── cache.py # Redis caching layer -│ ├── utils/ -│ │ ├── __init__.py -│ │ ├── auth.py # API key authentication -│ │ ├── logging.py # Structured logging -│ │ └── timezone.py # Timezone conversion -│ └── tests/ -│ ├── test_api.py -│ ├── test_notes.py -│ ├── test_reminders.py -│ └── test_automation.py -│ -├── agent/ -│ ├── agent.py # Main agent script -│ ├── requirements.txt -│ ├── config.example.json # Agent configuration template -│ ├── version.txt # Current agent version -│ ├── collectors/ -│ │ ├── __init__.py -│ │ ├── system_metrics.py # CPU, memory, disk -│ │ ├── application.py # Running apps, active window -│ │ └── custom.py # User-defined collectors -│ ├── executors/ -│ │ ├── __init__.py -│ │ └── commands.py # Command execution handlers -│ ├── install/ -│ │ ├── install.sh # Linux installation script -│ │ ├── install.ps1 # Windows installation script -│ │ ├── systemd/ -│ │ │ └── hub-agent.service -│ │ └── windows/ -│ │ └── task-scheduler.xml -│ └── tests/ -│ └── test_agent.py -│ -├── docs/ -│ ├── setup.md # Initial setup guide -│ ├── api.md # API documentation -│ ├── automation-guide.md # Writing automation rules -│ ├── agent-installation.md # Device agent setup -│ └── troubleshooting.md # Common issues -│ -├── scripts/ -│ ├── backup.sh # Database backup script -│ ├── cleanup.sh # Manual data cleanup -│ └── init-vault.sh # Initialize test vault -│ -└── config/ - ├── automation-rules.yaml # Default automation rules - └── device-whitelist.yaml # Optional device restrictions -``` - ---- - -## Development Phases - -### Phase 1: Foundation ⬜ NOT STARTED - -**Goal**: Basic infrastructure running with device registration and health monitoring - -**Tasks**: -- [ ] Create `docker-compose.yml` with all services (postgres, redis, chromadb, hub-api, worker) -- [ ] Create `.env.example` with all required environment variables -- [ ] Set up FastAPI application structure in `hub/main.py` -- [ ] Define database models in `hub/models.py` (Device, Event, Reminder tables) -- [ ] Create Alembic migration for initial schema -- [ ] Implement `/health` endpoint showing system status -- [ ] Implement device registration endpoint (`POST /devices/register`) -- [ ] Implement API key authentication middleware -- [ ] Set up structured JSON logging -- [ ] Create basic README.md with quickstart instructions - -**Acceptance Criteria**: -- [ ] `docker-compose up` successfully starts all services -- [ ] Can register a device and receive API key -- [ ] `/health` endpoint returns status of all components -- [ ] Logs are structured and readable -- [ ] Database migrations apply cleanly - -**Files to Create**: -- `docker-compose.yml` -- `.env.example` -- `hub/main.py` -- `hub/models.py` -- `hub/api/devices.py` -- `hub/api/health.py` -- `hub/utils/auth.py` -- `hub/utils/logging.py` -- `hub/alembic/versions/001_initial_schema.py` -- `README.md` - -**Progress**: 0/10 tasks complete - ---- - -### Phase 2: Device Agent ⬜ NOT STARTED - -**Goal**: Cross-platform agent that reports to hub and executes commands - -**Tasks**: -- [ ] Create `agent/agent.py` main script -- [ ] Implement heartbeat mechanism (report every 5 minutes) -- [ ] Implement system metrics collection (CPU, memory, disk) -- [ ] Implement event queue for offline operation -- [ ] Create device command execution framework -- [ ] Add auto-update check on startup (version endpoint) -- [ ] Create Linux systemd service file -- [ ] Create Windows Task Scheduler XML -- [ ] Write `install.sh` for Linux -- [ ] Write `install.ps1` for PowerShell -- [ ] Document agent configuration format - -**Acceptance Criteria**: -- [ ] Agent successfully registers with hub on first run -- [ ] Agent sends heartbeat every 5 minutes -- [ ] Agent queues events when hub is unreachable -- [ ] Agent can execute basic commands from hub -- [ ] Agent installs as service on Linux -- [ ] Agent installs as scheduled task on Windows -- [ ] Agent checks for updates on startup - -**Files to Create**: -- `agent/agent.py` -- `agent/config.example.json` -- `agent/collectors/system_metrics.py` -- `agent/executors/commands.py` -- `agent/install/install.sh` -- `agent/install/install.ps1` -- `agent/install/systemd/hub-agent.service` -- `docs/agent-installation.md` -- `hub/api/devices.py` (add version endpoint) - -**Progress**: 0/11 tasks complete - ---- - -### Phase 3: Notes & Reminders ⬜ NOT STARTED - -**Goal**: Parse markdown notes, extract reminders, send Discord notifications - -**Tasks**: -- [ ] Implement git operations in `services/notes.py` (pull, commit, push) -- [ ] Create markdown parser for reminder syntax -- [ ] Implement reminder extraction with date parsing -- [ ] Set up ChromaDB for note embeddings -- [ ] Create note indexing worker (incremental via git diff) -- [ ] Implement reminder scheduler (checks every minute) -- [ ] Create Discord webhook sender -- [ ] Implement reminder deletion from markdown files -- [ ] Add git conflict detection and Discord alerts -- [ ] Create reminder management API endpoints -- [ ] Add error notification for malformed reminder syntax -- [ ] Write tests for reminder parsing - -**Reminder Syntax to Support**: -```markdown -@remind 2024-11-15 Review proposal -@remind in 3 days Check on project -@remind daily at 09:00 Stand-up meeting -``` - -**Acceptance Criteria**: -- [ ] Hub can read notes from mounted Obsidian vault -- [ ] Hub detects new/modified notes via git diff -- [ ] Reminders are correctly parsed from markdown -- [ ] Scheduled reminders trigger at correct time -- [ ] Discord webhook delivers notification -- [ ] Processed reminder line is deleted from note -- [ ] Git commits and pushes changes successfully -- [ ] Git conflicts are detected and alerted - -**Files to Create**: -- `hub/services/notes.py` -- `hub/services/reminders.py` -- `hub/services/webhooks.py` -- `hub/services/vector_store.py` -- `hub/api/reminders.py` -- `hub/worker.py` (reminder scheduler) -- `hub/models.py` (add Reminder table) -- `hub/tests/test_notes.py` -- `hub/tests/test_reminders.py` -- `docs/automation-guide.md` (reminder syntax section) - -**Progress**: 0/12 tasks complete - ---- - -### Phase 4: AI Integration ⬜ NOT STARTED - -**Goal**: Gemini API integration with caching, rate limiting, and semantic search - -**Tasks**: -- [ ] Implement Gemini API client in `services/gemini.py` -- [ ] Set up Redis caching layer (24hr TTL) -- [ ] Implement rate limiting (100 req/hour) -- [ ] Implement circuit breaker (5 failures = 15min pause) -- [ ] Create fallback regex-based reminder parser -- [ ] Implement note embedding generation -- [ ] Create semantic search over notes using ChromaDB -- [ ] Add cost tracking (log token usage) -- [ ] Enhance reminder parsing with natural language support -- [ ] Create Gemini health check for monitoring -- [ ] Add Gemini context builder (device states + notes) -- [ ] Write tests for Gemini integration - -**Acceptance Criteria**: -- [ ] Gemini API successfully processes queries -- [ ] Responses are cached and reused appropriately -- [ ] Rate limiting prevents quota exhaustion -- [ ] Circuit breaker triggers after repeated failures -- [ ] System falls back to regex parsing when Gemini down -- [ ] Natural language dates parsed correctly ("next Tuesday") -- [ ] Semantic search returns relevant notes -- [ ] Token usage is logged for cost monitoring - -**Files to Create**: -- `hub/services/gemini.py` -- `hub/services/cache.py` -- `hub/utils/circuit_breaker.py` -- `hub/tests/test_gemini.py` -- Update `hub/services/notes.py` (add embeddings) -- Update `hub/services/reminders.py` (add NLP parsing) - -**Progress**: 0/12 tasks complete - ---- - -### Phase 5: Automation Engine ⬜ NOT STARTED - -**Goal**: YAML-based rules that trigger actions based on device events - -**Tasks**: -- [ ] Create YAML rule schema definition -- [ ] Implement rule parser in `services/automation.py` -- [ ] Create rule evaluation engine -- [ ] Implement trigger matching (device, event, time) -- [ ] Implement condition evaluation -- [ ] Implement action execution (webhook, command, search) -- [ ] Add Home Assistant REST API client -- [ ] Create bidirectional HA webhook integration -- [ ] Add dry-run mode for testing rules -- [ ] Create rule management API endpoints -- [ ] Write automation guide documentation -- [ ] Write tests for automation engine - -**Example Rule Format**: -```yaml -rules: - - name: "Evening work reminder" - trigger: - device: "laptop" - event: "work_apps_closed" - time_after: "17:00" - action: - type: "search_notes" - query: "today's todos" - notify: discord - enabled: true -``` - -**Acceptance Criteria**: -- [ ] YAML rules load correctly from config file -- [ ] Rules trigger based on device events -- [ ] Time-based conditions work correctly -- [ ] Actions execute successfully (Discord, HA, device commands) -- [ ] Home Assistant can trigger hub via webhook -- [ ] Hub can trigger HA automations via REST -- [ ] Dry-run mode shows what would happen without executing -- [ ] Invalid rules are caught with helpful errors - -**Files to Create**: -- `hub/services/automation.py` -- `hub/services/home_assistant.py` -- `hub/api/automations.py` -- `config/automation-rules.yaml` -- `docs/automation-guide.md` -- `hub/tests/test_automation.py` - -**Progress**: 0/12 tasks complete - ---- - -### Phase 6: Agent Auto-Update ⬜ NOT STARTED - -**Goal**: Agents automatically update from Gitea repository - -**Tasks**: -- [ ] Create agent version endpoint in hub API -- [ ] Implement version checking in agent -- [ ] Create agent download endpoint (proxy to Gitea) -- [ ] Implement agent self-update logic (download, backup, replace, restart) -- [ ] Add rollback mechanism (keep last working version) -- [ ] Create agent release workflow documentation -- [ ] Test update on Linux -- [ ] Test update on Windows -- [ ] Add update notification to Discord -- [ ] Create version tracking in hub database - -**Acceptance Criteria**: -- [ ] Agent checks version on startup -- [ ] Agent downloads new version when available -- [ ] Agent backs up current version before updating -- [ ] Agent restarts after successful update -- [ ] Agent can rollback to previous version on failure -- [ ] Update process works on both Linux and Windows -- [ ] Hub tracks which agents are on which versions - -**Files to Create**: -- `hub/api/admin.py` (version endpoints) -- Update `agent/agent.py` (add update logic) -- `scripts/release-agent.sh` (helper for releases) -- `docs/agent-installation.md` (update section) - -**Progress**: 0/10 tasks complete - ---- - -### Phase 7: Monitoring & Polish ⬜ NOT STARTED - -**Goal**: Observability, documentation, and production readiness - -**Tasks**: -- [ ] Create simple web dashboard (device grid, event stream) -- [ ] Enhance `/health` endpoint with detailed metrics -- [ ] Implement data retention cleanup job (7 days) -- [ ] Add database backup script -- [ ] Create troubleshooting documentation -- [ ] Write comprehensive API documentation -- [ ] Add Prometheus metrics endpoint (optional) -- [ ] Set up critical alerts (hub down, device offline >1hr) -- [ ] Create testing guide -- [ ] Final end-to-end testing across all components -- [ ] Performance testing with 30 devices -- [ ] Security audit (API keys, git credentials, etc.) - -**Acceptance Criteria**: -- [ ] Dashboard shows real-time system status -- [ ] Old data automatically cleaned up after 7 days -- [ ] Database backups run automatically -- [ ] All documentation is complete and accurate -- [ ] Critical alerts deliver to Discord -- [ ] System handles 30 concurrent device connections -- [ ] No security vulnerabilities in authentication/authorization - -**Files to Create**: -- `hub/api/dashboard.py` (simple UI endpoints) -- `scripts/backup.sh` -- `scripts/cleanup.sh` -- `docs/troubleshooting.md` -- `docs/api.md` -- `docs/testing.md` -- Update `README.md` (comprehensive) - -**Progress**: 0/12 tasks complete - ---- - -## Current Status - -**Overall Progress**: 0% (0/7 phases complete) - -**Current Phase**: Phase 1 - Foundation - -**Blockers**: None - -**Notes**: -- Project planning complete, ready to begin implementation -- All architecture decisions finalized -- Development environment ready (Linux server, Docker, Gitea) - ---- - -## Instructions for Agents - -### How to Use This File - -1. **Start with Phase 1** and work sequentially through phases -2. **Check off tasks** as you complete them using `[x]` -3. **Update progress** counters (e.g., "3/10 tasks complete") -4. **Mark phases** as complete when all tasks done: ⬜ → 🟡 → ✅ - - ⬜ NOT STARTED - - 🟡 IN PROGRESS - - ✅ COMPLETE -5. **Update "Current Status"** section with your progress -6. **Add notes** in "Blockers" or "Notes" if you encounter issues -7. **Commit changes** to this file after each work session - -### Phase Status Icons - -Use these when updating phase headers: -- ⬜ NOT STARTED - No work begun on this phase -- 🟡 IN PROGRESS - At least one task started -- ✅ COMPLETE - All tasks finished and acceptance criteria met - -### Before Starting a Phase - -1. Read through all tasks and acceptance criteria -2. Review the files to create -3. Check dependencies on previous phases -4. Update phase status to 🟡 IN PROGRESS - -### When Completing a Task - -1. Mark the task checkbox: `- [x]` -2. Update the progress counter -3. Commit the code changes -4. Update this file - -### When Completing a Phase - -1. Verify all acceptance criteria are met -2. Mark phase as ✅ COMPLETE -3. Update overall progress percentage -4. Move to next phase -5. Add any lessons learned in Notes section - -### Git Commit Messages - -Use conventional commits format: -``` -feat(phase1): implement device registration endpoint -fix(phase3): correct reminder date parsing -docs: update AGENTS.md progress -test(phase2): add agent heartbeat tests -``` - -### Testing Requirements - -- Write unit tests for core logic -- Write integration tests for API endpoints -- Test happy path and error cases -- Run tests before marking phase complete - -### Documentation Requirements - -- Update relevant docs when adding features -- Include code examples in documentation -- Keep API docs in sync with implementation -- Document configuration options - ---- - -## Environment Setup - -### Required Environment Variables - -```bash -# Gemini API -GEMINI_API_KEY=your_key_here - -# Database -DATABASE_URL=postgresql://user:pass@db:5432/hub -REDIS_URL=redis://redis:6379 - -# Git Configuration (for notes) -GIT_USER_NAME="Personal AI Hub" -GIT_USER_EMAIL="hub@yourdomain.local" -GIT_REMOTE_URL=https://gitea.local/user/obsidian-vault.git - -# Home Assistant -HOME_ASSISTANT_URL=http://homeassistant.local:8123 -HOME_ASSISTANT_TOKEN=your_ha_token - -# Discord -DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... - -# Agent Updates -GITEA_URL=http://gitea.local -GITEA_TOKEN=your_gitea_token - -# Security -API_SECRET_KEY=generate_random_key_here - -# Paths -NOTES_PATH=/app/notes -NOTES_ALLOWED_FOLDERS=reminders,tasks,projects -``` - -### Development Tools Needed - -- Docker & Docker Compose -- Python 3.11+ -- Git -- Text editor / IDE -- curl or Postman (API testing) -- Access to: Gemini API, Discord webhook, Gitea instance - ---- - -## Helpful Resources - -### API Clients -- **FastAPI Docs**: https://fastapi.tiangolo.com/ -- **Gemini API**: https://ai.google.dev/docs -- **Discord Webhooks**: https://discord.com/developers/docs/resources/webhook - -### Libraries -- **SQLAlchemy**: https://docs.sqlalchemy.org/ -- **ChromaDB**: https://docs.trychroma.com/ -- **Redis-py**: https://redis-py.readthedocs.io/ -- **GitPython**: https://gitpython.readthedocs.io/ - -### Patterns -- **Structured Logging**: Use JSON format with correlation IDs -- **Error Handling**: Always log, alert on critical errors -- **API Design**: RESTful, versioned endpoints -- **Testing**: Pytest with fixtures for database/API - ---- - -## Contact & Support - -If you encounter issues or need clarification: -1. Check troubleshooting doc -2. Ask for help \ No newline at end of file