From Typesense Cloud to Self-Hosted: Our Journey to Cost Savings and Performance

The Migration Decision

Our journey began with a simple calculation: we were running two hosted clusters in Typesense Cloud (one for staging, one for production) and the costs were becoming significant. The move to self-hosting promised not just cost savings, but also:

• Greater control over our infrastructure
• Ability to tune performance for our specific workload
• Opportunity to contribute back to open source
• Reduced latency by running in the same cloud region as our application

Setting Up on GKE Autopilot

We chose GKE Autopilot for its serverless approach to Kubernetes management – no need to manage nodes, patch systems, or worry about infrastructure scaling. The Typesense Kubernetes operator made deployment straightforward:

apiVersion: typesense.com/v1alpha1
kind: TypesenseCluster
metadata:
  name: search-cluster
spec:
  replicas: 3
  image: typesense/typesense:0.24.1
  resources:
    requests:
      cpu: "1000m"
      memory: "2Gi"
    limits:
      cpu: "2000m"
      memory: "4Gi"
  storage:
    size: 100Gi
    storageClass: standard-rwo

The operator handled the heavy lifting – creating StatefulSets, PersistentVolumes, and Services. We could now manage our Typesense clusters through familiar Kubernetes tooling.

The First Challenge: Cluster Membership During Rolling Updates

Shortly after deployment, we encountered our first major issue during a routine upgrade. When the Kubernetes operator performed rolling updates, we noticed brief periods where search queries would fail or return inconsistent results. The problem was subtle but critical.

Root Cause: The Operator Bug

Through careful investigation and log analysis, we discovered that the operator wasn’t waiting for cluster membership changes to settle during rolling updates. Here’s what was happening:

1. Operator would scale down a pod for upgrade
2. Immediately start a replacement pod
3. The new pod would join the cluster before the old pod fully deregistered
4. During this overlap, the cluster would have inconsistent views of membership
5. Some queries would route to terminating nodes, causing failures

The issue was in the operator’s reconciliation logic – it wasn’t implementing proper quorum waiting before proceeding with the next step of the rolling update. In a distributed system like Typesense, you need to ensure:

• Node removal is acknowledged by all cluster members
• New node addition is complete and synchronized
• Data replication has caught up before proceeding

Fork #1: The Kubernetes Operator (Still Running Today)

We forked the Typesense Kubernetes operator and implemented changes to add proper membership settling:

// Added logic to wait for cluster membership to stabilize
func (r *TypesenseClusterReconciler) waitForMembershipSettled(cluster *typesensev1alpha1.TypesenseCluster) error {
    timeout := time.After(5 * time.Minute)
    ticker := time.NewTicker(5 * time.Second)
    defer ticker.Stop()

    for {
        select {
        case <-timeout:
            return fmt.Errorf("timeout waiting for cluster membership to settle")
        case <-ticker.C:
            if r.isClusterHealthy(cluster) && r.isMembershipStable(cluster) {
                return nil
            }
        }
    }
}

func (r *TypesenseClusterReconciler) isMembershipStable(cluster *typesensev1alpha1.TypesenseCluster) bool {
    // Check that all nodes report the same cluster members
    // Wait for replication lag to be minimal
    // Verify no nodes are in transitional states
    return true // simplified for example
}

We submitted a pull request to the upstream operator repository, explaining the issue and our solution. Unfortunately, the maintainer declined to accept our changes. The feedback was that they wanted to take a different architectural approach, but no timeline was provided for when that would happen.

This put us in a position many engineering teams face: we had a working solution to a real problem, but the upstream project wasn’t ready to accept it. We made the pragmatic decision to continue running our fork.

As of today, we’re still running our forked operator. It’s been working reliably in our environments for [RUNTIME_MONTHS] months across multiple GKE clusters. We periodically check the upstream operator to see if they’ve addressed the membership settling issue, but haven’t yet migrated back. The maintenance burden of the fork has been minimal – we primarily rebase on upstream releases to pick up security fixes and new features.

Lessons learned from maintaining an operator fork:

• Keep the diff small and well-documented
• Set up automated testing to catch regressions
• Rebase regularly to avoid drift
• Document why each change exists for future team members
• Be prepared to maintain it long-term

The Second Challenge: Graceful Shutdown in Typesense

With the operator fixed, we encountered a second issue during GKE’s automatic node upgrades. Pods were being terminated abruptly, causing:

• In-flight search queries to fail
• Incomplete indexing operations
• Brief but noticeable service disruptions

Root Cause: Missing SIGTERM Handling

When GKE Autopilot performs node maintenance or upgrades, it sends SIGTERM to pods, giving them 30 seconds to gracefully shut down. We discovered that Typesense wasn’t properly handling this signal:

1. Missing SIGTERM handler: The main process didn’t catch the signal
2. Incomplete shutdown logic: Even when caught, it didn’t wait for ongoing operations
3. Health check mismatch: The /health endpoint continued returning 200 OK during shutdown, so Kubernetes kept routing traffic to terminating pods

This meant that when a node was being upgraded, Typesense pods would accept new requests right up until they were killed, causing failures.

Fork #2: Typesense Core (Merged Upstream!)

We created a fork of the Typesense repository to implement proper graceful shutdown. Our approach was comprehensive:

1. Implement proper SIGTERM handling in the main process
2. Add shutdown state tracking
3. Modify the health check endpoint to return 503 during shutdown
4. Ensure in-flight operations complete before exit

AI-Assisted Development

Here’s where our story takes an interesting turn. We used Claude (Anthropic’s AI assistant) to help develop the initial implementation for the health check API changes. The AI proved valuable in several ways:

Understanding Kubernetes expectations:

We described our deployment scenario, and Claude helped us understand the expected contract between Kubernetes and containerized applications – particularly around readiness/liveness probes during shutdown.

Generating the initial implementation:

// AI helped us structure this handler with proper HTTP semantics
app.get("/health", [](const Request& req, Response& res) {
    if (graceful_shutdown) {
        res.status = 503;  // Service Unavailable
        res.set_header("Connection", "close");
        res.set_content("Shutting down gracefully", "text/plain");
    } else {
        res.set_content("OK", "text/plain");
    }
});

Edge cases we hadn’t considered:

The AI suggested several scenarios we needed to handle:

• What if SIGTERM arrives during a large bulk import?
• Should we have different timeouts for different operation types?
• How do we handle the case where shutdown takes longer than the 30-second grace period?
• Should the health check differentiate between “draining” and “fully shut down”?

These questions led us to implement a more sophisticated shutdown state machine:

enum ShutdownState {
    RUNNING,
    DRAINING,      // No new requests, completing existing
    SHUTTING_DOWN, // Actively closing connections
    STOPPED        // Fully terminated
};

void sigterm_handler(int signum) {
    LOG(INFO) << "Received SIGTERM, entering drain state...";
    shutdown_state = DRAINING;

    // Stop accepting new connections
    server.stop_listening();

    // Wait for in-flight operations (with timeout)
    wait_for_operations(25);  // Leave 5s buffer before SIGKILL

    shutdown_state = SHUTTING_DOWN;

    // Close remaining connections
    close_all_connections();

    shutdown_state = STOPPED;
}

The AI-assisted approach let us iterate quickly through different designs, ultimately arriving at a solution that was both robust and aligned with Typesense’s coding standards.

Three Different Open Source Experiences

We opened a pull request to Typesense with our graceful shutdown changes, and the experience was different again from the operator PR – but in an unexpected way.

The Typesense Team Response

The Typesense team was incredibly responsive and collaborative:

• Reviewed our changes within 24 hours of submission
• Engaged in detailed technical discussion about implementation trade-offs
• Validated that our diagnosis was correct – the SIGTERM handling was indeed problematic
• Tested across different deployment scenarios (Docker, Kubernetes, bare metal)

However, they didn’t merge our PR directly. Instead, after understanding our use case and the problem we identified, they opened their own PR implementing delayed SIGTERM handling with a different architectural approach that better fit their codebase and roadmap.

The result was still a win:

• The bug we identified got fixed
• The solution was architected to work across more deployment scenarios than we’d considered
• We got credit for identifying and reporting the issue
• Our fork’s lifetime was measured in weeks, not months
• The Typesense team implemented more sophisticated timeout handling and configuration options than we’d initially proposed

The Reality of Open Source Contribution

Our experience with these two projects taught us that there are actually three common outcomes for open source contributions:

Outcome 1: Direct rejection (Kubernetes Operator)

The maintainer had valid reasons (architectural concerns, different roadmap priorities) for declining our changes. This left us maintaining a fork, which is a pragmatic engineering decision when you have a real problem to solve.

Outcome 2: Inspired implementation (Typesense Core)

Our PR wasn’t merged, but it catalyzed the maintainers to implement their own solution. We correctly identified the problem, proposed one approach, and they implemented a better one. This is actually a very common pattern in mature open source projects.

Outcome 3: Direct merge (What we hoped for both times)

While neither of our PRs merged directly, this remains the ideal outcome – and many projects do accept external contributions readily. The key is understanding each project’s governance model and contribution bar.

Key lessons learned:

Success isn’t just about code quality:

Both PRs were well-written and solved real problems. The outcomes varied based on project governance, maintainer bandwidth, architectural fit, and timing.

Bug reports can be as valuable as code:

With Typesense, our value was identifying and clearly documenting the issue. Sometimes that’s more important than the specific implementation.

Forks aren’t failures:

Running our forked operator isn’t a failure – it’s pragmatic engineering. We solved our problem, documented our changes, and made them available. If other teams have the same issue, they can benefit from our work.

Responsiveness varies wildly, and that’s okay:

The Typesense team’s quick engagement was exceptional. The operator maintainer’s lack of bandwidth is understandable. Both are valid realities of volunteer-maintained open source.

Maintenance is an investment:

Maintaining a fork requires discipline. We’ve invested in automation, documentation, and regular rebasing to keep the maintenance burden low. For the operator, it’s been worth it. For Typesense, we could migrate away after their fix shipped.

Performance and Cost Improvements

Cost Savings

The financial impact was immediate and significant:

• Typesense Cloud: $CLOUD_COST/month for two clusters (staging + production)
• Self-Hosted GKE: $SELF_HOSTED_COST/month total including compute, storage, and network costs
• Savings: COST_SAVINGS_PERCENT% reduction in monthly search infrastructure costs

These savings came from:

• Paying only for the resources we actually use
• Eliminating the Typesense Cloud management premium
• Leveraging GKE Autopilot’s efficient resource utilization

Latency Improvements

Moving to the same cloud region as our application yielded measurable performance gains:

• Query latency: Reduced from QUERY_LATENCY_BEFORE ms to QUERY_LATENCY_AFTER ms (QUERY_IMPROVEMENT_PERCENT% improvement)
• Indexing latency: Reduced from INDEXING_LATENCY_BEFORE ms to INDEXING_LATENCY_AFTER ms for batch operations
• Network hops: Eliminated cross-cloud network traversal

Storage Performance

One concern was storage performance. Typesense Cloud provided NVMe storage, and we wondered if GKE’s standard SSDs would suffice. Our testing revealed:

• GKE SSD: GKE_SSD_IOPS IOPS, GKE_SSD_THROUGHPUT MB/s throughput
• Typesense Cloud NVMe: CLOUD_NVME_IOPS IOPS, CLOUD_NVME_THROUGHPUT MB/s throughput
• Real-world impact: Minimal difference in search query performance (<REAL_WORLD_VARIANCE% variance)

The SSD performance was more than adequate for our search workload, proving that expensive NVMe wasn’t necessary for our use case.

Optimizing Ingestion Performance

With the basic setup working and graceful shutdown handling in place, we focused on optimizing our ingestion pipeline. Our workload involves frequent updates and deletions, which can trigger expensive compaction operations in Typesense.

Compaction Strategy

We discovered that during bulk delete operations, Typesense was performing real-time compaction, causing high I/O and CPU usage. Our solution was to modify the ingestion behavior:

# Typesense configuration optimizations
env:
- name: TYPESENSE_DIRTY_COMPACT_INTERVAL_SECONDS
  value: "COMPACTION_INTERVAL"  # Compact every X minutes instead of during operations
- name: TYPESENSE_MAX_MEMORY_USAGE
  value: "MEMORY_LIMIT_MB"  # Limit memory usage to X GB
- name: TYPESENSE_ENABLE_CACHING
  value: "true"

The key insight was to batch compaction operations rather than performing them during each delete. We implemented a queue-based approach:

1. Accept delete operations immediately
2. Queue compaction tasks
3. Process compaction in batches during low-traffic periods
4. Use background jobs to maintain index health

This reduced delete operation latency by DELETE_IMPROVEMENT_PERCENT% and smoothed out CPU utilization patterns.

Monitoring and Alerting

We set up comprehensive monitoring using Prometheus and Grafana:

• Query latency: Track p50, p95, and p99 response times
• Index size: Monitor storage growth trends
• Memory usage: Alert on approaching memory limits
• CPU utilization: Track during compaction operations
• Error rates: Monitor both HTTP and application-level errors
• Cluster membership: Alert on unstable cluster state (from our operator fix)

Embracing Automatic GKE Upgrades

One of the biggest benefits of GKE Autopilot is automatic node upgrades and patching. Initially, we were nervous about this – what if an upgrade broke our Typesense clusters? But our work on both the operator membership bug and Typesense graceful shutdown gave us confidence.

Testing Upgrade Resilience

We put our setup through rigorous testing:

1. Simulated node failures: Terminated pods and nodes to test failover
2. Version upgrades: Tested both patch and minor version upgrades
3. Regional failures: Simulated availability zone outages
4. Resource pressure: Tested behavior under high load during upgrades
5. Chaos engineering: Random pod/node terminations during peak traffic

Our fixes handled all scenarios beautifully. During upgrades:

• Operator waits for membership to settle before proceeding with next pod
• Pods receive SIGTERM and stop accepting new requests
• Ongoing operations complete within the 30-second grace period
• Health checks return 503 during shutdown, preventing new traffic
• New pods start and automatically join the cluster with proper synchronization
• No data loss or query failures

Real-World Upgrade Performance

We’ve now been through several automatic GKE upgrades without any incidents:

• Upgrade time: Typically UPGRADE_TIME_MINUTES minutes for the entire cluster
• Downtime: Zero (queries continue serving during upgrades)
• Manual intervention: None required
• Rollbacks: Never needed
• Failed queries during upgrades: 0.00%

This level of resilience has transformed our operations team’s confidence in our search infrastructure. The phrase “GKE is upgrading tonight” went from causing anxiety to being a complete non-event.

Key Takeaways

Our journey from Typesense Cloud to self-hosted infrastructure taught us several valuable lessons:

Technical Insights

1. Distributed systems need membership coordination – The operator’s rolling update strategy must wait for cluster consensus before proceeding
2. Graceful shutdown is critical for containerized applications – proper SIGTERM handling prevents data loss during upgrades
3. Health checks must reflect application state – returning 503 during shutdown prevents traffic to terminating pods
4. Storage choice matters less than proper tuning – standard SSDs performed adequately when properly configured
5. Batch operations are more efficient – avoiding real-time compaction during deletes significantly improved performance

Open Source Realities

1. Three outcomes are common – direct merge, inspired implementation, or rejection – all can be valid paths forward
2. Identifying problems is valuable – even if your code doesn’t merge, a well-documented bug report can catalyze fixes
3. Forks are a pragmatic solution – maintaining a small, focused fork can be the right engineering choice
4. Responsiveness varies wildly – the Typesense team’s engagement was exceptional, while the operator maintainer had different priorities
5. Document everything when forking – future you (and your teammates) will thank you
6. Keep forks minimal and tested – smaller diffs are easier to rebase and maintain

Operational Benefits

1. Cost savings can be substantial – we reduced infrastructure costs by COST_SAVINGS_PERCENT% while improving performance
2. Control over infrastructure enables optimization – we could tune compaction, caching, and resource allocation
3. Bug reports drive improvements – identifying issues clearly can be as valuable as submitting code
4. Automated upgrades become safe with proper graceful shutdown and membership handling
5. Confidence is earned through testing – chaos engineering and rigorous testing eliminated our fear of infrastructure changes

Strategic Advantages

1. Vendor independence – no longer tied to a specific cloud provider for search
2. Scalability on our terms – can scale horizontally based on our specific needs
3. Observability improvements – full visibility into infrastructure performance and health
4. Team confidence – operations team is no longer afraid of infrastructure changes
5. Engineering empowerment – fixing bugs ourselves instead of waiting for vendors

Looking Forward

Our self-hosted Typesense deployment has been running successfully for RUNTIME_MONTHS months now. We’re exploring additional optimizations:

• Multi-region deployment for global latency optimization
• Custom builds with performance tweaks specific to our workload
• Advanced caching strategies for frequently accessed data
• Migration back to upstream operator – we’ll revisit this periodically to see if membership handling is addressed

We successfully migrated to upstream Typesense after they shipped their delayed SIGTERM handling implementation – exactly the outcome we’d hoped for. We’re also keeping an eye on the upstream operator. If the maintainer implements membership settling in a way that works for our use cases, we’d happily migrate back. Until then, our fork continues to serve us well.

Conclusion: The Power of Engagement

The migration that started as a cost-saving initiative has evolved into a strategic advantage. By taking ownership of our infrastructure – including maintaining a fork when necessary, reporting bugs clearly, and engaging with upstream projects – we’ve gained:

• Control over our search infrastructure
• Performance improvements through proximity and tuning
• Cost savings of COST_SAVINGS_PERCENT%
• Confidence in automated infrastructure changes
• Knowledge of how our critical systems work at a deep level

Most importantly, we’ve demonstrated that you don’t need to be intimidated by open source infrastructure. When you find bugs, you can fix them or report them clearly. When maintainers can’t accept your changes, you can run a fork. When teams are responsive, you can collaborate and drive improvements even if your exact code doesn’t ship.

It’s a testament to what’s possible when you combine cloud-native technologies with open-source software and a willingness to engage deeply with the projects you depend on – whether that means maintaining your own forks or working with maintainers to drive upstream improvements.

—

This migration was a team effort involving engineers from our platform, infrastructure, and search teams. Special thanks to the Typesense team for their excellent support, engagement on our bug reports, and their implementation of delayed SIGTERM handling. The Kubernetes operator we use is a fork we maintain ourselves, which has been instrumental in our success.