Project Overview
The Problem Statement
Manual expense logging is tedious, time-consuming, and error-prone. Freelancers, small business owners, and individuals struggle to organize paper receipts and invoices, frequently missing eligible tax deductions and lacking real-time visibility into their spending trajectories.
Engineering Motivation
Traditional accounting tools require laborious manually keyed line items or rigid rule-based regex parsing that breaks whenever a merchant alters receipt layouts. Generative vision models offer a breakthrough: true zero-shot receipt parsing.
Core Engineering Goals
- Achieve >95% accurate zero-shot field extraction across diverse receipt formats without prior model training.
- Provide instantaneous natural language query translation with strict SQL injection safeguards.
- Maintain strict financial data privacy using end-to-end encryption at rest.
Target Audience
Freelancers, independent contractors, small business operators, and personal finance power users seeking effortless receipt tracking and conversational financial analytics.
Key Capabilities & Features
Core technical features designed to fulfill project requirements with high reliability and performance.
Multimodal Receipt Extraction
Replaces manual entry by using multimodal LLMs to extract vendor, line items, tax, and totals directly from receipt images with zero pre-processing.
Natural Language Text-to-SQL
Translates conversational questions like 'How much did I spend on coffee in June?' into validated PostgreSQL AST queries for sub-10ms answers.
Visual Financial Dashboards
Renders real-time spending breakdowns, monthly category distribution, and budget anomaly alerts using responsive Recharts visualizations.
Interactive Architecture Explorer
Hover or click on any system layer to inspect data flow topologies, connection pipelines, and security specifications.
Frontend Client Layer
Single-page responsive web client serving UI components, handling optimistic state updates, and streaming live server notifications.
Technology Stack
Purposeful framework selection and technical rationale behind core architecture dependencies.
Frontend
- Next.js 15
Server Components & App Router for lightning-fast page loading and SEO.
- React 19
Declarative UI rendering with fluid concurrent mode state updates.
- Tailwind CSS
Utility-first design system ensuring mobile-first responsive layout.
Backend
- Node.js / Express
Asynchronous non-blocking network I/O for concurrent AI model requests.
- Drizzle ORM
Type-safe database client ensuring zero schema drift.
Database
- PostgreSQL 16
ACID compliance ensuring ledger-grade financial accuracy.
AI & ML
- Gemini 2.5 Flash
1M+ context window and fast sub-800ms multimodal inference.
Cloud & Infrastructure
- Vercel Platform
Edge network static serving and zero-downtime serverless deployments.
- Google Cloud Storage
Encrypted object bucket storage for raw receipt images.
Key Architectural Trade-Off Rationale
Next.js for Full-Stack Agility
Chose Next.js to colocate frontend components with serverless API routes, reducing latency and simplifying deployment.
PostgreSQL & Drizzle
Selected a relational database to ensure ACID compliance for financial data, with Drizzle providing type-safe database access.
Gemini 2.5 Multimodal Integration
Leveraged Google's vision models for raw receipt image parsing, bypassing brittle traditional OCR pipelines.
Development Journey
Systematic engineering progression from initial domain research to production deployment.
Receipt Dataset Analysis & OCR Benchmarking
Evaluated traditional Tesseract OCR vs Multimodal LLM zero-shot extraction across 200 diverse merchant receipts.
Schema Design & Security Specification
Designed relational PostgreSQL schema with Zod type definitions and AES-256 receipt image encryption specs.
UI Prototype & Charting Design
Created high-fidelity Figma components for receipt dropzones, conversational search bars, and analytics charts.
Full-Stack & LLM Pipeline Implementation
Built Next.js frontend, Drizzle database adapters, Gemini multimodal endpoint, and Text-to-SQL translation engine.
SQL Injection Guard & Unit Testing
Wrote unit tests for AST SQL validation, simulated edge case receipts, and verified WCAG contrast guidelines.
Vercel Production Launch & Monitoring
Deployed application to Vercel global edge CDN with automated Sentry exception tracking.
Challenges & Solutions
Deep-dive into non-trivial engineering obstacles, root causes, and verifiable technical outcomes.
Receipt layouts vary dramatically across merchants, crumpled paper, and faint thermal printing.
Traditional OCR regex rules broke on over 40% of non-standard receipts.
Implemented structured JSON Schema constraints with Gemini 2.5 Flash, enforcing fallback parsing rules when confidence drops.
Performance & Results
Empirical measurements, benchmark results, and production optimization techniques.
Applied Performance Optimization Techniques
- Prompt caching to reuse static system context across sequential receipt API calls.
- Gzip payload compression on REST response handlers reducing network transfer sizes by 65%.
- Composite indexing on (user_id, transaction_date, category) in PostgreSQL for instant chart aggregations.
Lessons Learned
Key engineering insights, architectural reflection, and takeaways for future systems design.
Integrating LLMs into financial workflows requires strict AST schema enforcement and validation logic rather than trusting raw text output.
End-to-end TypeScript type safety (from database ORM to frontend state) prevents silent runtime data shape mismatches.
Future Roadmap
Planned technical enhancements, new feature modules, and architectural extensions.
Direct Plaid Bank Integration
Synchronize live credit card and bank account feeds for automatic transaction reconciliation.
Multi-User Small Business Accounts
Add role-based access control (RBAC) and team approval workflows for expense reports.
On-Device Mobile Document Scanner
Develop a native web worker camera scanner for perspective crop and deskew before AI upload.
A high-throughput machine learning anomaly detection pipeline for identifying financial fraud in highly imbalanced datasets.
A high-performance algorithmic trading simulator providing real-time WebSocket market streaming and low-latency order execution.