Rafael Pereira

VBET

VBET: analytics on top of a SQL Server we could not change

The database belonged to another team. The dashboard had to stop depending on a schema we did not control.

Senior Backend Engineer Oct/2023 – Feb/2025

~7 min → <1 s commissions, from the original load to the warm cache

Pipeline de comissões
Pipeline de comissões SQL Server externo ETL PostgreSQL pré-cálculo Cache Dashboard <1 s quente REALTIME tendência CLOSED pagamento ~7 min → ~15 s p99 → <1 s com cache quente

Cada estágio corresponde a uma decisão incremental documentada no case. Números de outras histórias não entram neste desenho.

Context

At VBET, between October 2023 and February 2025, the analytics product served iGaming influencers and affiliates. The dashboard gathered dozens of metrics; commission was the most critical reading. Influencers accepted a small lag in same-day data as long as the screen responded. Payout depended on consolidated prior-day data, not on the live value.

Constraints

The SQL Server was external, shared, and not reliably modifiable. Temporary indexes could be removed by the database owner. The original API mixed SQL queries built from parameters, with injection risk, and aggregated too much in memory.

Problem

The system had been sized for smaller influencers. With larger bases, the worst dashboard peak reached about seven minutes. Security and maintainability came before performance: raw queries, little test coverage, and a synchronous path that recalculated too much on every request.

Decision

The evolution was incremental, in the order the constraints appeared:

  1. remove unsafe SQL, parameterize access, document, and test;
  2. optimize queries and temporary indexes, as mitigation rather than invariant;
  3. parallelize independent queries with Go, goroutines, and channels;
  4. when the bottleneck moved back to SQL Server, build an owned ETL and PostgreSQL, with pre-computation, checkpoints, and reconciliation;
  5. separate REALTIME reads (trend, eventual consistency) from CLOSED reads (financial accuracy and payout);
  6. cache-aside with TTL aligned to the accepted lag of about five minutes;
  7. controlled degradation if the cache failed, instead of taking the screen down.

Discarded alternative

Insisting on indexes in the external database as architecture, or recalculating years of history on every access. Also discarded: paying affiliates from REALTIME data.

Result

The reconciled line in the experience dossier, for the commission/dashboard path, is:

  • initial worst peak: about 7 minutes;
  • after queries and indexes: about 3 minutes;
  • after parallelization: about 1 minute;
  • after ETL/PostgreSQL: about 15 seconds at commission p99 without cache;
  • warm cache: under 1 second.

Each number belongs to its stage. It does not describe the gain of a later decomposition into microservices.

Limitations

The investigation, the ETL + owned database decision, the REALTIME/CLOSED split, and the degradation policy are the attributable core here. Decomposing the monolith into Kubernetes is a separate story and does not mix the “500%” nor sub-60 ms latency into this case. Older resume versions citing 30 seconds on cold load, 11 seconds, or SLA percentages without a scenario are left out. If the product starts requiring realtime accuracy for payout, the CLOSED split stops being enough.