Color Mode
  • Use system setting
  • Light mode
  • Dark mode
  • Subscribe to our Newsletter Newsletter
    Follow us on Twitter Twitter
    RSS Feed RSS

    Building the world's fastest token and decentralised exchange search

    Which exchange currently offers the best liquidity on ETH-USDC? What trading pairs had more than $1M volume in the last 24 hours? What are the trending small-cap tokens? Trading Strategy can give you answer to these and other technical trading questions on decentralised markets. In this blog post, our engineers discuss how we deliver these amazing features using Typesense search engine and SvelteKit technologies.

    What are Trading Strategy protocol and algorithmic trading?

    Trading Strategy is a new service for algorithmic and technical trading of cryptocurrencies on decentralised exchanges (DEXes).

    Explore technical trading data for decentralised exchanges

    Algorithmic trading is a derivative of technical analysis; taking trading positions based on pure mathematics and data. Algorithmic trading is part of quantitative finance. Algorithmic trading provides a systematic approach to trading compared to methods based on trader intuition or instinct.

    Traders and investors need fast access to the latest trading data in order to analyze trends, make trading decisions, or simply look up details on a token or pair. If the Internet has taught us anything in the last 25 years, it's that fast and accurate search results are king. It's true for websites (Google), videos (YouTube), products (Amazon). Now, it's also true for decentralised markets, as Trading Strategy is excited to announce the launch of its token search. For more information about Trading Strategy, please read our announcement blog post.

    Searching for trading pairs and tokens and with built-in risk management

    Trading Strategy offers quick search and advanced search, allowing search through hundreds of thousands of tokens that trade on decentralised exchanges. For the comparison, NASDAQ lists around 3000 stocks, so the dataset is massive and 99% of the content is irrelevant.

    Quick search allows you quickly to find out tokens and their trading pairs. The search has basic risk management built-in. To make the results more relevant we sort out trading pairs by their liquidity and automatically flag (grey out) results we believe are high risk or not interesting.

    The advanced search allows you to drill down through thousands of trading pairs. You can choose criteria like liquidity, price change, different blockchains and exchanges to find attractive pairs to trade.

    An example how to search mid-sized liquidity ($500k - $5M) USDC trading pairs across all decentralised exchanges on Binance Smart Chain. Try out the advanced search here.

    All of the search functionality is open source and available as an API service for professional users and partners to integrate with.  

    Choosing a search index backend

    Martin Fowler defines software architecture as those decisions which are both important and hard to change. In designing a search solution, the decision that is most clearly architectural is that of the search index backend.

    In terms of importance, where you store and index your searchable records affects nearly everything: hosting and maintenance, data latency, import/indexing process, performance, scalability, security, result relevance, cost – you get the point! As to difficulty to change, good architecture seeks to minimize itself – all else being equal, having fewer things hard to change (i.e., less architecture), is preferred. Given these criteria, the ideal search backend should optimize the concerns listed above while being as fungible as reasonably possible.

    A further consideration for our team: we have a strong bias towards open-source solutions. We value the freedom to run software where and how we choose and the ability to modify it when needed. The Trading Strategy platform is built entirely on open-source products and frameworks and we are working to open-source all of our core modules – we prefer not to introduce a proprietary dependency for search if at all possible.

    With these principles in mind, let's review the options our team considered.

    Trading Strategy uses PostgreSQL (with TimescaleDB extension) as our backend database. This made PostgreSQL's Full-Text Search (FTS) a good candidate to consider. Using PostgreSQL avoids the introduction of a new architectural component and minimizes the need to shovel data between systems. Beyond this, some team members had past experience with this approach, so it initially seemed like a no-brainer. The one major downside we identified was configuration and management overhead – PostgreSQL FTS is quite capable, but it requires a fair amount of expertise to optimize performance and search result relevance.

    As we explored this option, we realized that our initial data assumptions were not correct. Some of the trading entity content required for searches does not actually reside in the database – there are mappings and transformations in our SQLAlchemy model-tier. We would either need to duplicate the business logic in PostgreSQL (a bad idea) or re-import the fully marshalled trading entity data back into the database. Other data sets that we plan to add to the search index (blog posts and documentation) also reside outside of the database.

    We concluded that the apparent benefits of this approach were not as great as we'd hoped, and no longer outweighed the downsides (configuration and management overhead).

    Elasticsearch

    As we were evaluating search solutions, we were also in the middle of implementing log aggregation using Elastic Stack. This made Elasticsearch the natural next option to consider. It has the same benefit mentioned above – it avoids introducing an additional component to our environment. This benefit proved to be minimal, however – we quickly realized that we would need to run separate Elasticsearch instances to keep performance, scale and security concerns for search decoupled from logs.

    While Elasticsearch remained a strong contender, we were also curious to explore newer solutions that are more targeted to the use case of site search.

    Algolia

    In the last 5 years, Algolia has emerged as a clear front-runner in the search space. They provide a robust, feature-rich search service along with an outstanding developer experience. Using Algolia, you can quickly launch a high-quality search solution that will scale with your needs.

    The other thing that will scale are your costs. Algolia is only available as a proprietary SaaS offering. Based on the number of records we need to index, our monthly cost would be in the hundreds from day one. Our database of trading entities and user community are both growing rapidly – with their usage-based pricing model, so would our costs.

    It's worth noting that while their core search engine product is proprietary and SaaS-based, the developer tools needed to implement search – including API clients, UI frameworks and site crawlers – are all open-source. In addition, they offer a discounted pricing plan for open-source projects (with some limitations).

    For Trading Strategy, the SaaS-only hosting model and high monthly fees were serious drawbacks.

    Typesense

    Typesense bills itself as "An Open Source Algolia Alternative & An Easier-to-Use ElasticSearch Alternative." It provides the key features of a modern search offering like Algolia, shrink-wrapped in a developer-friendly open-source package. According to their docs and demo videos, it's super easy to run (single binary, minimal config), provides excellent performance and scalability, and exposes everything though a developer-friendly REST API and client libs. In addition, they offer a cloud-hosted option – including high-availability and Geo-distributed clustering – with a much lower price tag than Algolia. To be honest… it sounded too good to be true!

    Since it's open-source and claims to be easy to run, we decided to put it to the test. In less than two hours, we had Typesense installed on an old 2013 MacBook Pro (via docker), had imported a test data set of 1.5M records, and were running search queries using the Typesense JavaScript client. Performance was excellent (sub-100ms on average) and memory consumption was quite reasonable (well under 2GB).

    For the most part, we found their guides and API docs to be very good. Not as polished as Algolia's (to be expected), but clear, accurate and easy to follow. We had some questions about how to optimize our search insert/update process and got quick, clear responses in their slack workspace.

    We spent several hours experimenting with various search API parameters to ensure we could get the kind of results we needed for our UI (facet counts, grouped results, advanced filter and sort options, etc.). Within a day, we were convinced that Typesense was the right solution.

    Search and indexing architecture overview

    The selection of the search backend enabled us to finalize our high-level architecture.

    The four main components are:

    1. Search backend (Typesense instance)
    2. Search admin module – used for configuring API keys and search collections
    3. Trading Entities importer – this process lives within our trading data oracle backend; it exports all trading entities and batch imports them into the search backend
    4. Search frontend – search requests are sent directly from the browser to the search backend (see next section)

    Typesense has a straightforward security model: the admin API key is kept private and is only used for admin tasks, such as managing other API keys; other modules use a separate API key with permissions scoped to their function. The data in our search index is all publicly available blockchain data and accessible on our site without a login, so the search API key is considered public.

    Building the search user experience

    Our initial search design includes two search UIs: a "quick search" in our top-nav where users can quickly find top-matching trading entities; and an "advanced search" page which includes more complete search results and advanced search options.

    As we've written about previously, the Trading Strategy website is built using Svelte and SvelteKit. We considered a couple of approaches for creating the search experience within our Svelte-based application.

    InstantSearch.js with Typesense adapter

    Typesense doesn't provide their own off-the-shelf frontend components. Instead, they've chosen to integrate with the open-source InstantSearch and Autocomplete UI libraries maintained by Algolia. The Typesense team has created a Typesense InstantSearch Adapter wrapper around their JavaScript client library that is plug-compatible with Algolia's JavaScript API client.

    Algolia's UI libraries are available for a variety of web component frameworks, including React, Vue and Angular. There's no Svelte version yet, so we would have to fall back to their vanilla JS library. We created a proof-of-concept to see if this would "play nice" within our Svelte app. It worked!. Well… it technically worked, but we had some concerns about using this approach in production.

    First, the impact to our build size was significant. The "quick search" UI alone added 150kb to our top-nav component (used on every page) and the "advanced search" UI was considerably larger. Second, the Algolia UI libraries use embedded mustache templates to control layout. Nothing against mustache, but the idea of maintaining two different templating languages in the same frontend was a non-starter.

    Custom Svelte store and components

    After searching for alternatives and not finding anything that was the right fit, we bit the bullet and decided to roll the search UI from scratch. This turned out to be surprisingly straightforward.

    Our first step was to encapsulate Typesense search requests with a custom Svelte store. Using a store gives us an internal search API that can be used across different components with Svelte's reactive store syntax. The code below represents a simplified version of our actual tradingEntities store:

    import { SearchClient } from "typesense";
    import { writable } from "svelte/store";
    
    const client = new SearchClient({
      nodes: [{
        host: 'your.typesense.url',
        protocol: 'https' ,
        port: 443
      }],
      apiKey: 'TYPESENSE_SEARCH_API_KEY'
    });
    
    const collection = client.collections('trading-entities').documents();
    
    const { subscribe, set } = writable({});
    
    async function search(options) {
      try {
        const response = await collection.search(options);
        set(response);
      } catch (error) {
        console.error(error);
      }
    }
    
    export default { subscribe, search };
    
    lib/trading-entities.js

    Now, we can import this store into a component like this über-simplified "quick search" component:

    <script>
      import tradingEntities from "./trading-entities";
      let q = "";
      $: tradingEntities.search({ q });
    </script>
    
    <div>
      <input bind:value={q} type="search" placeholder="search" />
      <ul>
        {#each $tradingEntities.hits as { document }, index (document.id)}
          <li>{document.description}</li>
        {/each}
      </ul>
    </div>
    
    lib/Search.svelte

    You can pass additional Typesense search options into tradingEntities.search({ q }), and you'll likely want a custom component for your results layout. From there… you have the full power Typesense search combined with the elegance of Svelte to build a search UI that meets your needs. For further inspiration, feel free to check out our quick search component and advanced search page – both are works-in-progress, but we think they provide a pretty solid v1 search experience.

    Conclusion

    We began this post by highlighting the importance of search and the factors we considered in identifying a search solution. Having completed and launched our initial solution, here are a few key takeaways:

    • Typesense: We're very impressed with Typesense so far – it has lived up to its brand promise of being a powerful, purpose-built search backend that is easy to manage and use.
    • Svelte: We continue to be happy with our choice of Svelte as our frontend framework. We were delighted by how easy it was to wire up the Typesense API and build a search UI tailored to our needs.
    • Search: Search is a nuanced and challenging problem space. Exposing the right search/sort options and prioritizing the most valuable results requires significant analysis, design and creativity. While we have a good foundation, we recognize there are a lot of opportunities to optimize and improve our search experience. We look forward to gathering and incorporating feedback from users.
    • Open source: We continue to be blown away by amazing open-source products and communities. As we build on our initial search offering, we hope to have an opportunity to partner with others in the Svelte and Typesense communities to produce a reusable Svelte Typesense library.

    Let us know if you found this post helpful, or if you have any suggestions on how we can improve upon our search solution!

    Trading Strategy is an algorithmic trading protocol for decentralised markets, enabling automated trading on decentralised exchanges (DEXs). Learn more about algorithmic trading here.

    Join our community of traders and developers on Discord.

    We are hiring

    We are currently hiring for frontend (Svelte), backend (Python/PostgreSQL) and quant research (Jupyter Notebook/Pandas) positions. If you are interested in working with cryptocurrencies and algorithmic trading please email us at: [email protected].