dagster dbt strapi

PropertyAuctions.io - Real Estate Data Platform

Felis (https://felis.co.uk/) - Leading real estate property management and consultancy firm
partnered with MetaOps to build PropertyAuctions.io, a full SaaS platform enabling property investors and developers to identify profitable opportunities through advanced market intelligence, property price analysis, and geospatial insights. The platform features user authentication, horizontal scaling capabilities, and a subscription-based business model.

Felis needed a scalable, reliable data infrastructure to:

  • Aggregate and process data from 100+ diverse sources (government APIs, partner feeds, internal datasets)

  • Provide real-time property market intelligence to investors through a full SaaS platform

  • Handle complex data transformations and analytics at scale

  • Support user authentication, subscription management, and multi-tenancy

  • Ensure long-term platform sustainability with manageable operations and horizontal scaling

  • Enable rapid feature development and deployment with minimal operational overhead

The challenge was building a production-ready SaaS platform that could handle massive data volumes while maintaining high availability, supporting subscription-based business models, and providing the business with the tools to manage and scale the platform independently over time.

Kubernetes-Native Infrastructure:

  • Dagster Open Source deployed on Kubernetes with horizontal auto-scaling

  • PostgreSQL with PostGIS extension running on Kubernetes for robust data storage, analytics, and geospatial processing

  • S3 storage for scalable data lake architecture and pipeline artifacts

  • dbt integrated with Dagster for data transformations, deployed as Kubernetes jobs

  • Two-environment setup (dev/prod) with Kubernetes namespaces for comprehensive testing

  • Full containerization enabling seamless scaling and deployment across environments

SaaS Platform Components:

  • Web UI: Node.js application with responsive design and real-time data visualization

  • Mapbox integration: Interactive mapping and geospatial visualization for property location analysis

  • Strapi CMS: Highly scalable, modularized content management system for platform configuration and content delivery

  • User Authentication: Secure multi-tenant authentication system with role-based access control

  • Subscription Management: Integrated billing and subscription model supporting multiple pricing tiers

Data Pipeline Architecture:

  • Automated ingestion from 100+ data sources including government statistics, partner APIs, and internal datasets

  • End-to-end pipeline automation running completely unattended on Kubernetes

  • Comprehensive monitoring and failure detection with Grafana

  • Python-based processing jobs with robust error handling, all containerized for Kubernetes deployment

DevOps & Long-term Management:

  • GitHub CI/CD pipelines for automated deployments to Kubernetes clusters

  • Kubernetes orchestration providing horizontal scaling, self-healing, and resource optimization

  • Long-term sustainability: Comprehensive documentation, monitoring dashboards, and automated operational procedures enabling the business to manage and scale independently

  • Supporting stack: Hasura (GraphQL API), highly scalable modularized Strapi CMS, Grafana (monitoring) - all running on Kubernetes

Key Technical Achievements

  • Full SaaS platform delivery with user authentication, subscription management, and multi-tenant architecture

  • Kubernetes-native deployment with horizontal auto-scaling and self-healing capabilities

  • Production-scale deployment with zero critical failures since launch

  • Automated pipeline management with comprehensive dependency handling

  • Real-time data processing capabilities for timely market insights

  • Scalable architecture supporting platform growth, user acquisition, and subscription-based revenue model

  • Long-term operational sustainability with comprehensive management tools and documentation

Outcomes

  • 300+ daily visitors within first month of soft launch

  • Full SaaS platform with subscription-based revenue model operational

  • Zero system downtime since production deployment across all Kubernetes services

  • 100+ data sources successfully integrated and maintained

  • Multi-tenant user base with secure authentication and role-based access

  • Real-time market intelligence enabling informed investment decisions for subscribers

Technical Success

  • Kubernetes-native architecture providing seamless horizontal scaling and operational efficiency

  • Reliable data orchestration through Dagster's robust pipeline management on containerized infrastructure

  • SaaS platform capabilities including user management, billing integration, and multi-tenancy

  • Automated operations reducing manual intervention and enabling business team to manage platform independently

  • Comprehensive monitoring ensuring system health across all Kubernetes deployments