IndexMyUrls - URL Indexing Platform for Search Engines
Project Overview
IndexMyUrls is a SaaS web platform developed in Django that allows website owners to accelerate the indexing of their pages in search engines. The service uses official Google APIs and the IndexNow protocol to directly submit URLs to search engines, reducing the waiting time from several weeks to just a few hours.
Technical Architecture
Backend
- Main framework: Django with MVT (Model-View-Template) architecture
- Database: PostgreSQL in production, SQLite in development
- Cache: Redis for performance and session management
- Asynchronous tasks: Celery with Redis as the broker
- Authentication: Django Allauth with Google OAuth2 integration
- Payments: Paddle for subscription management
Frontend
- CSS Framework: Tailwind CSS with Flowbite
- Interactivity: HTMX for dynamic interactions
- JavaScript: Vanilla JS with Paddle.js integration
- Responsive design: Mobile-first with adaptive breakpoints
Main Features
Website Management
- Automatic import from Google Search Console
- Generation of unique IndexNow keys per site
- Automatic scanning of XML sitemaps
- Plan-based limitations (1 to 50 sites)
- Quota management per service and user
URL Submission
- Google Indexing API: Direct submission via OAuth2
- IndexNow: Multi-engine support (Bing, Yandex, Naver, Seznam)
- Batch submission: Batch processing for performance optimization
- Priority: Priority tagging system for important URLs
- Automatic verification: Index status check via Google Search Console
Automation
- Sitemap scanning: Automatic discovery of new URLs
- Automatic submission: Background processing via Celery
- Periodic verification: Daily index status checks
- Automatic cleanup: Deletion of outdated data
Data Models
Website
- Domain and IndexNow key management
- Configuration of automated features
- Quota tracking per service
- Integration with Google Search Console
URL
- Multiple statuses: pending, submitted, indexed, failed
- Submission tracking per service
- Priority system for submissions
- Indexing verification history
Subscription
- Plans: Free, Hobby, Professionals, Enterprise
- Google and IndexNow quota management
- Monthly/annual billing via Paddle
- Plan-based limits (sites, URLs, submissions)
API Integrations
Google APIs
- Indexing API: Direct URL submission
- Search Console API: Index status verification
- OAuth2: Secure authentication
- Token management: Automatic credential refresh
IndexNow
- Multi-engine protocol: Bing, Yandex, Naver, Seznam
- Unique keys: Automatic generation and validation
- Batch submission: Performance optimization
- Error handling: Automatic retries and logging
Asynchronous Task System
Quota Management
- Google: 200-2000 submissions/day depending on plan
- IndexNow: 4000-10000 submissions/day depending on plan
- Verifications: 2000 verifications/day per site
- Automatic reset: Daily quotas with rollover
Security and Performance
Security
- HTTPS required: Automatic redirection in production
- HSTS: Enhanced security headers
- CSRF protection: Defense against CSRF attacks
- Secure OAuth2: Token management with automatic refresh
- Data validation: URL and domain sanitization
Performance
- Redis cache: Dashboard data caching
- Static compression: WhiteNoise for asset serving
- Optimized database: Indexes on critical fields
- Asynchronous processing: Celery for heavy tasks
- Monitoring: Prometheus for metrics
User Interface
Main Dashboard
- Real-time statistics (total, pending, indexed URLs)
- Multi-site management with unified interface
- Batch submission with multiple selection
- Submission and verification history
- Quota and limit management
Advanced Features
- GSC Import: Automatic sync with Google Search Console
- IndexNow Generation: Automatic creation of keys and files
- Bulk operations: Mass actions on URLs
- Priority system: Tagging of important URLs
- Real-time updates: HTMX for dynamic updates
Deployment and Infrastructure
Environments
- Development: SQLite, local server
- Production: PostgreSQL, Redis, Gunicorn, Traefik
- Docker: Full containerization with docker-compose
- Monitoring: Prometheus, structured logging
Third-Party Services
- Paddle: Payment and subscription management
- Google APIs: Indexing API and Search Console API
- IndexNow: Multi-search engine protocol
- Email: SMTP for notifications
- Storage: S3-compatible for file handling
Business Model
Subscription Plans
- Free: 1 site, 100 URLs/day, limited features
- Hobby: 5 sites, 200 URLs/day, basic automation
- Professionals: 20 sites, 200 URLs/day, priority support
- Enterprise: 50 sites, 200 URLs/day, dedicated support
Billing
- Secure payments via Paddle
- Monthly or annual billing
- 2-month discount on annual subscriptions
- Real-time upgrade/downgrade management
This platform offers a complete solution for technical SEO optimization, enabling webmasters to significantly accelerate their content indexing while providing a modern, intuitive interface for managing their websites.
