Going Live as an Indexer on a Mainnet

After a few months of experimenting with The Graph on the TestNet, we have finally brought the live Indexer on a Mainnet. This required a lot of trial and error, these guys are not joking when they say the infrastructure behind TheGraph needs to be beefy!

Our infrastructure

We are running Mainnet and Testnet Archive Nodes using Erigon. Those are bare-metal servers with SSD storage running on Hetzner. SSD’s are a must, running on disk will not work effectively and lead to you dropping behind on the chainhead regularly.

The Graph components run on a Kubernetes Cluster with Testnet and Mainnet indexers deployed into different namespaces. We use Istio Service Mesh with mutual TLS for extra added security/monitoring capabilities.

Prometheus and Grafana deployed into a different namespace on the same Kubernetes Cluster provide the primary monitoring tool.

Subgraph Indexing

For now, we are manually choosing subgraphs to index and managing allocations via the graph command line tool (we have a mobile app that we’ll be using for this, more on that later….).

We have indexed several subgraphs off-chain and are allocating some stakes to them. Now we are waiting for queries to come through so we can collect our query fees and retire.

We recently closed our test cycle allocations earning some nice GRT in the process.

Lessons Learned

• The Graph ecosystem might seem complex, but persistence pays off.
• Learn on Testnet prior to going into Mainnet.
• Everything is well documented on the Graph site. Read documentation and code for every component on GitHub.
• The Graph discord has all the answers, search it’s history first.
• You do need SSD drives on Archive Node servers, no way around this.
• Erigon is great and saves so much space.

Next Steps

Indexer Management

Managing your indexer via the command line is not easy. Going through hundreds of lines with Deployment IDs, allocations and POI is error-prone.

We explored several projects working on the visualisation of the Indexer operations. Some of them look promising, but none is covering everything we need.

We have created our own mobile app to manage the Indexer and run it internally in a Development mode. The first Public alpha release is expected on this side of Christmas.

Selecting Subgraphs and managing allocations

We have started work on automating the allocation process and will gradually move into semi-automated and hopefully fully automated mode. This is something we would like to bring into the mobile app as well. Watch this space.

Scaling Indexing operations

As we progress with the above tasks we are looking to scale out Indexing into multiple chains and increase the number of subgraphs we can handle.

Even though Indexer is on Mainnet we are still in a “learning mode” (isn’t it true for all web3? 😉 ) so we will move in baby steps.

Thanks

We would like to thank the Graph discord community for answering our questions, theGraph for running “Indexer Office Hours” and various other community events, and for helping us catch up (re-sync?) with the latest changes.

Happy Indexing!