Institutional Email Automation SaaS

Backend: NestJS + PostgreSQL

Chose NestJS for its enterprise-ready architecture, built-in dependency injection, and excellent TypeScript support:

• Modular Architecture: Clean separation between document processing, validation, email delivery, and auth modules
• Prisma ORM: Type-safe database queries with automatic migrations and excellent PostgreSQL support
• Dependency Injection: Easy to test and maintain with clear module boundaries
• Built-in Guards: JWT authentication and role-based access control implemented cleanly

OCR: Tesseract.js

Selected Tesseract.js for client-side OCR processing:

• Open Source: No licensing costs, active community, extensive language support
• Browser-Based: Runs in browser, reducing server load and API costs
• Accuracy: 94% accuracy after preprocessing pipeline implementation
• Trade-off: Slower than cloud OCR services (AWS Textract, Google Vision) but cost-effective for MVP

Email: Brevo API

Integrated Brevo for transactional email delivery:

• Reliability: 99.9% uptime SLA with automatic failover
• Deliverability: Built-in spam prevention and domain reputation management
• Tracking: Open rates, click tracking, and delivery confirmations
• Cost-Effective: Generous free tier (300 emails/day) suitable for initial deployment

Database: PostgreSQL

PostgreSQL chosen for robust relational data management:

• ACID Compliance: Critical for credential data integrity
• JSON Support: Flexible storage for OCR metadata and audit logs
• Full-Text Search: Built-in FTS for searching documents and logs
• Prisma Integration: Excellent ORM support with type safety

1. OCR Accuracy with Variable Quality Documents

Challenge: Documents arrived in wildly varying quality - faded photocopies, smartphone photos at angles, low-resolution scans. Raw Tesseract accuracy was 67% on real-world documents.

Solution:

• Built preprocessing pipeline with OpenCV.js for image enhancement (deskewing, contrast adjustment, binarization)
• Implemented adaptive thresholding based on image histogram analysis
• Added multiple OCR passes with different preprocessing settings, selecting best result
• Created document quality classifier to route low-quality docs directly to manual entry

Result: OCR accuracy improved from 67% to 94% on production documents.

2. Handling OCR Errors and Ambiguities

Challenge: OCR commonly confused similar characters (0/O, 1/I/l, 5/S) causing invalid student IDs and email addresses.

Solution:

• Implemented context-aware character correction using field constraints (student ID must be 8 digits)
• Built fuzzy matching against existing student database to catch similar-looking errors
• Added Levenshtein distance checking for name fields against enrollment records
• Created confidence penalty system for ambiguous character combinations

Result: Character confusion errors reduced by 78%, with remaining ambiguities flagged for review.

3. Balancing Automation with Accuracy

Challenge: Setting confidence thresholds - too high meant excessive manual review (defeating automation purpose), too low meant errors reaching students.

Solution:

• Implemented multi-factor confidence scoring combining OCR confidence, pattern match strength, database validation results
• Created configurable three-tier system: auto-approve (90%+), review queue (70-90%), manual entry (<70%)
• Built feedback loop tracking admin corrections to tune thresholds over time
• Added field-level confidence allowing partial automation (auto-approve ID but review name)

Result: Achieved 92% automation rate with 0.3% error rate reaching end users.

4. Email Delivery at Scale

Challenge: Sending thousands of credential emails without triggering spam filters or hitting API rate limits.

Solution:

• Implemented queue-based delivery system with rate limiting (300 emails/hour respecting Brevo limits)
• Added exponential backoff retry logic for transient failures (network issues, temporary DNS failures)
• Configured SPF, DKIM, DMARC records for institutional domain to improve deliverability
• Built monitoring dashboard tracking delivery rates, bounce rates, spam complaints

Result: 99.7% delivery success rate with <0.1% spam complaints.

5. Database Performance with Large Audit Logs

Challenge: Audit logging every action created massive table growth (50K+ records/month), slowing down queries and impacting UX.

Solution:

• Implemented database partitioning by month for audit log table
• Added compound indexes on common query patterns (userId + timestamp, documentId + timestamp)
• Created separate read replica for audit queries to avoid impacting transactional performance
• Implemented log archival system moving records older than 6 months to cold storage

Result: Query performance maintained under 100ms even with 500K+ audit records.

Authentication & Authorization

• JWT-Based Auth: Secure token-based authentication with refresh token rotation
• Role-Based Access Control: Admin, supervisor, operator roles with granular permissions
• Session Management: Automatic timeout after 30 minutes inactivity
• Password Security: bcrypt hashing with salt rounds = 12

Data Protection

• Encryption at Rest: PostgreSQL with TDE (Transparent Data Encryption)
• Encryption in Transit: TLS 1.3 for all API communications
• Credential Masking: Passwords partially masked in UI (show first 2, last 2 characters)
• Document Purging: Original documents deleted after 30 days (configurable)

Reliability & Monitoring

• Health Checks: Automated health endpoints for database, email service, OCR processing
• Error Tracking: Sentry integration for real-time error monitoring and alerting
• Backup Strategy: Daily automated PostgreSQL backups with 30-day retention
• Uptime SLA: 99.5% uptime over 6 months of production operation

Compliance & Audit

• Complete Audit Trail: Every action logged with user, timestamp, IP address, before/after states
• FERPA Compliance: Student data handling follows FERPA guidelines
• Data Retention: Configurable retention policies for different data types
• Export Capabilities: Compliance reports exportable to CSV/PDF

1. OCR is 80% Preprocessing, 20% Recognition

Initially focused on tuning Tesseract parameters but saw minimal improvement. Real gains came from investing in robust preprocessing pipeline - image enhancement, deskewing, adaptive thresholding. Quality preprocessing took accuracy from 67% to 94%. Lesson: In document processing, cleaning the input is more impactful than tuning the algorithm.

2. Embrace Human-in-the-Loop from Day One

Early version attempted 100% automation which resulted in unacceptable error rates. Pivoted to hybrid approach with confidence-based routing. Ironically, 92% automation with human oversight proved more valuable than 100% automation with errors. Lesson: Perfect automation is often impossible - design for supervised automation instead.

3. Audit Logs Are Not Optional for Enterprise

Initially treated audit logging as a nice-to-have feature. During pilot deployment, institutions demanded complete audit trails for compliance. Retrofitting comprehensive logging was painful. Lesson: For any system handling sensitive data, build audit logging from day one - it will be required eventually.

4. Email Deliverability is Hard

Naively assumed sending emails was straightforward. Encountered spam filter issues, rate limits, bounce handling complexity. Learned about SPF/DKIM/DMARC, warm-up periods, sender reputation. Lesson: Email delivery at scale requires as much engineering as the core product - don't underestimate infrastructure services.

5. TypeScript + Prisma is Enterprise-Ready Stack

NestJS + Prisma + PostgreSQL proved excellent for enterprise SaaS. Type safety caught bugs at compile time, Prisma migrations simplified schema evolution, NestJS modularity aided testing. Would choose this stack again for similar projects. Lesson: Modern TypeScript ecosystem has matured to the point of being production-ready for serious applications.