reading time

Article

reading time

Sep 27, 2022

A Look At the LinkPool Metrics Engine

Supplementary to this post, the video of this SmartCon workshop is available on LinkPool’s YouTube channel: Link Pool Metrics Engine SmartCon Workshop

separator
reading time

5 minutes

reading time

Jon Gregis

LinkPool Metrics Engine illustration

LinkPool Metrics Engine illustration

Hidden In Plain Sight

The blockchain is a public ledger in its simplest form. It is very easy to monitor certain wallets or contract addresses to understand when and where they are sending transactions to and from. But, when you need to collect advanced analytics over periods of time from a wide range of wallets or contracts all of these hex encoded transactions camouflage themselves and become hidden in plain sight. This is what the LinkPool Metrics Engine was designed to solve. It is a way to parse raw blockchain transactions into human readable data and enable analysis of current and historical use to understand possible future trends.

What is the LinkPool Metrics Engine?

The LinkPool Metrics Engine is a mechanism to parse incoming blocks from a blockchain and extract transaction data to understand what is happening. What we are really looking for are events emitted in contracts. Let’s take a look at what a sample transaction log looks like on a blockchain explorer.

 

Transaction log from a blockchain explorer

As we can see, the address that emitted the event is listed first. After that is the event itself which, in this case, is PunkBought. This is what we are looking for when parsing transactions. However, it’s not quite in human readable form yet. We actually want the Keccak-256 hash of that event called the Event Signature found in the event signature from the Topics of the event, specifically topic index 0 (topic[0]). 

DistributeRewards(address,uint256,uint256)

“3a748d40c2a4d1652cf9e6faa37c5e347459597272d02843e21e96eb95642372”

Above we have an event called DistributeRewards which has 3 parameters, an address, uint256, and another uint256. If we hash this using Keccak-256 we should get the string listed above. This is what we will be polling in our engine to collect when parsing each block.

Now that we have a basic idea of what the engine is doing, let’s take a look at the steps we need to take when parsing each block.

Block Parsing Lifecycle

Diagram of the LinkPool Metrics Engine block parsing lifecycle.

The block parsing lifecycle is broken down into four parts.

Collect Event Signature

This step is where we figure out which events we want to listen to on-chain. Once we know the event and have it hashed to create the event signature, we will then build a poller to start checking specifically for that signature.

 

Look for the Topic[0]

The Topic[0] in the event log is the event signature. When parsing each block and running through every transaction we want to confirm the event signature (topic[0]) matches the signature we have plugged into our poller. If it is a match we can move onto the next step.

 

Parse Event from Application Binary Interface (ABI)

Now that we have a match we can start dissecting the information in that transaction. But before we do that we need the contract ABI. Because the contract is in binary form and not human readable, we use the contracts ABI to read and write from the contract. Once we have the ABI, and because we are using Golang, we can use a helpful tool called abigen to create custom packages with useful functions and types that are generated from the contract ABI. We use these functions to parse the hex code from the event logs into human readable data and continue the process.

 

Process Event

The final step is to process the event into a data structure of our choosing and store it in our database. We transform the data from the function generated by the ABI and start thinking how we want the model to look in our database. Once we transform the data and write to the database, the data parsing operation  is complete.

Technology Behind the LinkPool Metrics Engine

The metrics engine was created using three key tools.

The core LinkPool backend is written in Go and, for consistency, we maintained this when building the metrics engine for its concurrency and ease of use when processing billions of transactions.

We also use postgres for core database needs and added the use of timescaleDB as a key, third piece of tooling. We will briefly go over why we chose timescaleDB at a high level below.

 

TimescaleDB

TimescaleDB is a time-series database built on top of postgres. It uses normal postgres syntax when writing queries and can be 10-100x faster than traditional Postgres at querying large data samples from the use of compression.

The LinkPool Metrics Engine saves items as time-series data to aggregate large data samples and create / store data into hyper-tables instead of normal postgres tables. We also create views that continuously aggregate data on intervals that we pre-set. For example, we have multiple views each with a 5m, 1h, 12h, 1d, 7d interval. This allows us to have a more exact or broad set of data when choosing what to query. With this in place we can query millions of rows almost instantly.

Current Use Cases

Marketplace

Market.link is the best way to visualize the power of the metrics engine.

Visualization of the LinkPool Metrics Engine data in Market.Link app

We can see all the different Chainlink products used on-chain across every Chainlink supported blockchain. 

We can also dive into specific feeds and gather data to understand how each feed is performing. Ex. ETH/USD

Visualization Market.Link feeds powered by the LinkPool Metrics Engine

And, if data on individual node providers is what you’re looking for, then check out the nodes page and search your favorite node provider to view responses, revenue, and more.

Node service provider LinkPool as seen on Market.Link

NFT Metrics 

The second largest use case for the LinkPool Metrics Engine is our NFT Metrics Time Adjusted Market Index (TAMI) implementation. We will briefly go over what a TAMI is and why it is so powerful before diving into how we built the metrics engine to collect every NFT transaction.

 

Time Adjusted Market Index (TAMI)

A TAMI is a universal mechanism for calculating the estimated value of a collection of assets. That is, a TAMI allows us to calculate the market cap of an asset more accurately than metrics like floor price. And, better price discovery allows for the creation of lending and derivatives instruments such as futures/options markets with higher degrees of precision.

We began by porting over an existing TAMI library, written in TypeScript, over to Go to be consumed by our metrics engine. The Go implementation of the LinkPool Metrics Engine is completely open sourced and can be found in the links below. The original TAMI library is available as well and includes a thorough walk through of what a TAMI is, the theory behind it, and pros/cons by use case.

The LinkPool TAMI engine currently supports the Opensea, LooksRare, and CryptoPunks markets with a X2Y2 implementation in progress.

 

NFT Metrics Lifecycle

Like metrics gathered from Chainlink specific events, NFT metrics are very similar with one extra step. Because we are now dealing with NFT’s, we need to identify if we are collecting ERC-721 or ERC-1155 tokens (or ERC-20 in the case of cryptopunks contracts). 

Our initial event signature comes from the marketplace ‘escrow’ contract used to transfer from buyer to seller. Then, from the logs, we then need to capture which transfer event was emitted because ERC-721/1155 have different transfer event signatures. Our lifecycle below explains where this new step is executed.

Diagram of the LinkPool Metrics Engine NFT metrics lifecycle

CryptoPunks Marketplace

Now that we have everything set up it should be very easy to start collecting data right? …then in comes the cryptopunks contract…

Since the cryptopunks contract was one of the original NFT collections it did not follow ERC-721/1155 standards and is, rather, an ERC-20. The contract is actually a marketplace in itself with the ability to buy, sell, and bid for individual punks. Some would argue that it is how all NFT’s should be set up but we won’t get into that debate here.

Typically, we find the event we want to track and grab the event signature. In this case, the PunkBought event. As we began collecting data from the event, however, we noticed very odd transaction data where many PunkBought events had zero ETH transacted. This certainly raised some eyebrows so we put on our detective hats and analyzed the data more closely.

Turns out, the contract allows a seller to accept a bid for a punk they were selling, firing the AcceptBidForPunk function and, eventually, emitting the PunkBought event as any normal transaction. All current punk listings are gathered in a mapping in the CryptoPunks contract and when the AcceptBidForPunk function is called the mapping for the current punk, seller address, and price of the accepted bid are zeroed out prior to emission of the PunkBought event. This emits the PunkBought event with zero values when in actuality it was bought at a certain ETH amount. This occurs on line 231 of the CryptoPunksMarket contract.

With this new information we were able to update the logic in the engine and when we find a zero ETH buy on a punk we return the state of the transaction and identify the mapping value and include that into our model instead of the zero value to get a more accurate TAMI.

The problematic line in CryptoPunksMarket.sol

How to Use the Metrics Engine

Now that we know a little more on how the metrics engine works, let’s see how you can start using the data collected on-chain to create some really cool Dapps.

We have created a SmartCon2022 repo where you can grab the postman collection for many of the query’s we will be going over in here and is a great way to get your feet wet.

 

Endpoint, Tables, Views

Here, we will go over all you need to start querying some data. We will start off with the public endpoint listed below.

const URL = “https://api.market.link/v1/metrics/”

const METHOD = “POST”

If you choose to call directly from the terminal you can use the curl command. Make sure the method is POST and the –header flag is set to ‘content-type: text/plain’. Our payload will just be a string of the SQL query we want to make.

Ex.

curl –location –request POST ‘https://api.market.link/v1/metrics’ –header ‘Content-Type: text/plain’ –data-raw ‘SELECT sum(link_reward_sum) as value FROM node_aggregates_by_1d WHERE interval >= now() – interval ‘\’‘7 days’\’‘;’

Below is a list of the available tables, views, and intervals that you will be able to query.

Tables

  • feed_events
  • feed_users
  • keeper_events
  • keeper_registries
  • networks
  • oracle_request_events
  • vrf_v1_coordinators
  • vrf_v2_events
  • vrf_v1_coordinators
  • vrf_v2_events

Views

  • feed_aggregates
  • feed_answers
  • feed_node_aggregates
  • feed_users
  • node_aggregates

Intervals

  • 5m
  • 1h
  • 12h
  • 1d
  • 7d

Ex. feed_aggregates_by_5m

 

Postman Collection

Lets now dive into the postman collection to see how the queries are formed and what some of the possibilities are.

The collection is broken into four main directories. The Global Queries section includes broad queries on certain aggregated and non aggregated tables. The Individual Feed Queries section is an aggregated look at specific feeds over a time range. Individual Node Queries cover statistics of individual node data. And the TAMI directory lists our NFT Metrics TAMI endpoint on querying the time adjusted market index on any NFT collection on Ethereum Mainnet. Let’s take a look at some example responses on each section.

 

Global Queries

You will notice that with many of the queries listed we limit the amount of rows returned to X amount. Technically, you have the ability to query every row in the DB on each table but because there are possibly 100’s of millions of rows the endpoint will timeout and you will receive an error. We suggest starting with smaller limits for a better user experience.

 SELECT * FROM feed_events ORDER BY ts DESC LIMIT 5;

LinkPool Metrics Engine global feed query output

We can now see the other columns we have the ability to query and update our query again for more specific data. The first feed above tells us that this feed is on HECO Mainnet, it is the BTC/USD feed, the node it came from, and the current answer.

 

Individual Feed Queries

Here we start using the power of aggregation to query large data sets. The query below responds with the price of LINK/USD every 5 min over a 30 day time period.

SELECT time_bucket(‘5m’, interval) as time,

       network_id,

       network_group,

       feed_address,

       feed_name,

       locf(last(answer, interval), treat_null_as_missing => true) as “value”

FROM feed_answers_by_5m

WHERE feed_name = ‘LINK / USD’

  AND interval >= now() – interval ’30 days’

  AND network_name = ‘Mainnet’

GROUP BY time, network_id, network_group, feed_address, feed_name

ORDER BY time;

LinkPool Metrics Engine individual feed query output

Individual Node Queries

We can also do the same with nodes. The query below starts to get a little more complex but will let us know the daily profit of each network on a certain node provider.

SELECT

   time,

   network_id,

   network_group,

   node_name,

   coalesce(sum(link_rewards) – sum(gas_costs), 0) as “value” 

FROM

   (

      SELECT

         events.interval as time,

         network_id,

         network_group,

         node_name,

         sum(events.total_eth) * last(gas_pair.answer, answer) as gas_costs,

         sum(events.total_link) * last(link_usd.link_answer, answer) as link_rewards 

      FROM

         (

(

            SELECT

               interval, node_id, network_id, network_group, node_name, ether_spent_sum as total_eth, link_reward_sum as total_link 

            FROM

               node_aggregates_by_1d 

            WHERE

               node_name = ‘LinkPool’ 

               AND “interval” >= now() – interval ’30 days’ 

               AND network_name = ‘Mainnet’ 

            ORDER BY

               interval) events 

               LEFT JOIN

                  (

                     SELECT

                        id,

                        gas_token 

                     FROM

                        networks

                  )

                  as gas_token 

                  ON events.network_id = gas_token.id 

               LEFT JOIN

                  (

                     SELECT

                        interval,

                        feed_name,

                        last(answer, interval) as answer 

                     FROM

                        feed_answers_by_1d 

                     WHERE

                        “interval” >= now() – interval ’30 days’ 

                        AND feed_name IN 

                        (

                           SELECT DISTINCT

                              (gas_token || ‘ / USD’) as gas_token 

                           FROM

                              networks

                        )

                     GROUP BY

                        interval,

                        feed_name 

                     ORDER BY

                        interval

                  )

                  gas_pair 

                  ON events.interval = gas_pair.interval 

                  AND gas_token.gas_token || ‘ / USD’ = gas_pair.feed_name 

               LEFT JOIN

                  (

                     SELECT

                        interval,

                        last(answer, interval) as link_answer 

                     FROM

                        feed_answers_by_1d 

                     WHERE

                        feed_name = ‘LINK / USD’ 

                        AND network_name = ‘Mainnet’ 

                        AND “interval” >= now() – interval ’30 days’ 

                     GROUP BY

                        “interval” 

                     ORDER BY

                        “interval”

                  )

                  link_usd 

                  ON events.interval = link_usd.interval 

         )

      GROUP BY

         events.interval,

         network_id,

         node_id,

         network_id,

         network_group,

         node_name

   )

   as gas_sums 

GROUP BY

   time,

   network_id,

   network_group,

   Node_name;

LinkPool Metrics Engine individual node query output

TAMI

Finally, we have our newest endpoint which retrieves our TAMI answer from a specific NFT collection. Also note it is possible to query only transactions from a specific marketplace and by not including this query param, you will query all markets by default.

Ex

markets=opensea

The endpoint below is quite simple and will simply need the contract address of the collection you want to query in the path.

https://nft-metrics-tami.p.rapidapi.com/{address}

Once we include a contract address our response should include an object with three fields–answer in ETH, GWEI, WEI–and a success message.

LinkPool Metrics Engine TAMI query output

What's Next?

So what is next for the LinkPool Metrics Engine? We are happy to announce that we will be open sourcing it so that others can audit and build on top of it. Being that we work in a decentralized field it only makes sense that others can harness the power of the engine for other projects outside of Chainlink and LinkPool’s TAMI.

So the question really falls on all of you. What can you see yourselves building in the future with all of this?

Notes & Links

All of the relative notes and links will be listed below so you can start diving into all of this yourselves. Thanks again to all of our supporters and if you have any questions feel free to reach out to us. Cheers!

Golang TAMI Implementation

Original TAMI Repo (TypeScript)

Market Metrics API Docs

NFT Metrics TAMI Collection API

This blog post was originally presented as a live, deep dive technical workshop at Chainlink SmartCon 2022 and reformatted as a blog post. You can view the original video presentation on LinkPool’s YouTube channel: Link Pool Metrics Engine SmartCon Workshop. You can follow Jon on Twitter at @An0n_Jon.

Related Articles