AG Grid Case Study with Proof Trading

Proof Trading is a startup with the goal to revolutionize equity trading, we sat down to learn about their business and how they use AG Grid in their real time trading platform.

Proof Trading's first target is the broker, the people who buy and sell stocks on the exchange on behalf of an investor. By introducing new algorithmic trading approaches supported by a streamlined GUI and fast backend infrastructure all hosted in the cloud they plan to shake up the industry.

Proof Trading are open about how they are building the platform and release detailed blog posts covering their aims, approaches and lessons learned building a technically advanced product. By embracing transparency and making their algorithms public, there is a risk that other companies may well copy and implement their research. But they believe that by becoming the recognised market thought leader, clients will naturally want to work with the originator of the research and stay ahead of the game. Proof Trading are working hard to be the original, with everyone else following behind as an imitation and copy.

In addition they are helping make the US equity trading landscape accessible to everyone through their educational documentation explaining how the market works in their Market Structure Primer.

Proof Trading described their architecture in a highly technical blog post. They needed a cutting edge UX that matches the technical excellence of their algorithms. In short they wanted to build the perfect blotter (the view of financial trades made over time). After extensive comparisons and evaluations of multiple data grids and tables they decided to use the Enterprise edition of AG Grid - it was customizable and affordable.

Proof trading knew what they required for their blotters, some of the necessary functionality comes in the community edition by standard:

Fast updates, multiple times a second.
No lag when navigating with continued row updates.
Column sorting, filtering, and grouping.
Speedy and customizable search through the entire grid.
Custom renderers for cell data.
CSS control for Rows and Cells based on conditional logic.

Proof Trading picked the Enterprise edition for a truly cutting edge experience and gain access to:

Virtual rows/columns for a highly performant grid to handle millions of records.
Customizing Status Bar to show summary and counts.
Selection of Rows and Cells for copy to clipboard.
Customizing right-click context menu.
Save/load current configuration of the grid.

Interview with Proof Trading

We had the chance to discuss the Proof Trading business and their use of AG Grid with Prerak Sanghvi (CTO) and Han Dong (Designer / Developer).

How would you explain Proof Trading to someone who only knows that people buy and sell shares on the stock market?

Proof Trading is like Robinhood but for institutional investors. If you have a 401(k) account with mutual funds, your 401(k) provider or those mutual funds may be a customer of Proof Trading.

There is a lot of unwarranted secrecy in the industry, which limits accountability. We want to bring a new level of transparency and educate participants on how an equities broker dealer actually operates.

You are very transparent around your trading algorithm design.

We liken our transparency in algorithmic design to the world of encryption algorithms. All of the best known encryption algorithms in use are peer reviewed and the implementations are open source, yet that doesn’t mean they are easily breakable. In the same sense, we’re not worried about being gamed. If more people adopt our methods, we would be happy as we believe it would be the right step for investors and the industry as a whole.

What are you using AG Grid to implement on your GUI?

So far just the various implementations of the blotter, mixing and matching features: handling static data, handling streaming data, allowing filtering, sorting, grouping, copy/paste, custom cell rendering.

Currently messages are pushed to the client as 1 second bucket payloads, but we can turn up or down the granularity.

Editable Data Entry in the grids?

No. Any modification to data in the blotter is serious business, we tend to put many guardrails around such actions, so in contrast, direct data entry is probably too unimpeded for our liking.

We use custom right click actions to pop up a form with sufficient context where a user can specify any update actions.

Are you using the inline embedded graphs that AG Grid provides?

Not currently, the concern is that screen real estate is very valuable for our use case. Although we have implemented a simple horizontal single-bar chart to show percentage of an order filled. In future, we are not averse to implementing sparklines.

How do you approach Build or Buy decisions? And Open Source vs Commercial?

We prefer the use of open source components where possible. This is not only cost effective but allows us sufficient control over what we can do now and in the future. While having commercial support can provide some peace of mind, we think having the source code and ability to change it is even more so.

We are open to considering building our components more than most others. We considered building our own grid, but as we explored this, we found that AG Grid had made essentially the same design decisions that we would make for a full-featured high-performance grid. We realized most (if not all) of the features we wanted to implement were already checked off by AG Grid. Coupled with a very reasonable price point, it was a no-brainer.

We did not evaluate Ag-grid Enterprise (the evaluation was based on the community edition). But in general, I think Ag-grid may still be a hidden gem amongst grid libraries at this point (likely as a result of catering to a niche field), not every answer is floating around on the internet. So commercial support is definitely important.

How many grids did you evaluate? And what criteria?

4 grids total: Ag-grid, our own grid, and two others.

We’re a small team so time is often in shortage, so I don’t believe we thoroughly documented the grid comparison process in writing.

We mocked up the 4 aforementioned grids and did performance comparisons with a million static rows or 10,000 streaming rows.

It became evident fairly quickly that some grids could not handle larger datasets (500k+ rows) once we mocked them up side by side.

We were using the community version during development. We eventually reached a point where we needed one of the enterprise features (I think it was either Context Menu or Status Bar) but at that point we were fairly confident about Ag-grid as a product.

I think a big part of the developer experience is determined by the quality of the documentation. In this area, the Ag-grid website does a good job by providing plenty of examples and building a good navigation structure. Once we became familiar with Ag-grid, the search box became the routine entry point.

We only used contacted support a couple times, over both technical topics and general design. The support team was responsive and very knowledgeable.

I can imagine larger dev teams may have experts across multiple frameworks. But for us, React-compatibility was a requirement.

We only require Google Chrome at this point. Our GUI is currently internal facing, so browser compatibility is not a priority at this point.

Are you trying to automate any testing?

So far our testing has been manual. Although we would like to implement actual DOM testing for the GUI at some point, which would be a project of its own due to the complexity involved.

The widget that wraps our grid is deployed via drag-and-drop so this complicates the testing process. The time it would require to implement something like puppeteer did not seem like a good investment for us at this point (though that may certainly change down the road with added capacity).

We find that we’re able to adequately test end-to-end pipelines by skipping the UI and working directly with client-side APIs. Adding the UI to the mix simply expands testing to cover 3rd party implementation, which definitely has its value in adding to completeness, but falls short of being a priority for us.

How much data are you putting through the GUI?

Currently around 50k records a day just in the narrow context of the GUI, but this number would need to reach upwards of 500k or more. Also, keep in mind that each of these records may update 5-100 times. The actual number of messages in the trading system is orders of magnitude more.

Not all messages inside the trading system reach the GUI. There is a UI state app in the system that condenses messages into information usable by end-users. And even after that, we apply optimizations (eg. conflation) to reduce the amount that reaches the browser.

We optimise for the ‘general’ situation, you need a specialized high-performant grid. How did you customize it?

We made use of the custom tool panel component to provide controls for different actions in our grid, things like toggling the filter and status bar, a widget log for debugging. Because pixel space is precious, and grid usage can be both active and passive depending on the content, we want to provide a certain degree of flexibility and control to the user, letting them decide whether to prioritize for ease of access or maximizing information in their workspace.

We also customized the context menu component to define a set of right-click actions (eg. column positioning) for all views and view-specific actions based on user permission (eg. cancel order).

AG Grid handles sorting elegantly with deltaSort, it runs at O(n) and is stable and faster than Javascript’s built-in sort. For our use case, with reasonable expectation to exceed 500k rows daily, handling streaming updates in a sorted model will inevitably undercut performance.

While AG Grid can sort data at O(n) while maintaining complex relationships like master-details, we decided a faster but less-capable data model was better for our use case.

For most of our views, we still use AG Grid out-of-the-box, but for certain specialized grids, we built a variant to get O(log n) sorting using custom trees.

The Viewport Row Model API provided everything we needed to build our own data model.

Currently, our custom data model lives client-side in the worker thread. At the same time, we wanted to reserve the option to move it server-side if ever needed. The Viewport Row Model interface was able to provide this flexibility.

For our use case, we prioritized speed over memory. While both presented challenges, speed was the more pressing of the two.

CPU bottleneck has to be solved at the software level. Because we want to leave open the option of potentially having server-side sorting and filtering, it was important that we solve our problems single-threaded. Parallel sorting may be possible but the added complexity does not seem like a good trade off when simpler single-thread solutions exist. When it comes to resources in the cloud, the intended way to scale up compute is horizontally, through increasing vCPU units, which is not something we can take advantage of. And so it was important for us that we optimize for compute at the application level.

Memory is an easier problem to solve. Scaling up memory is fairly straightforward and rather inert in terms of architectural implications. Additionally at the software level, pagination will always be something we have in the back of our pockets to address memory issues.

We use Server Sent Events as the primary method for data delivery. We chose this because our GUI was mostly one-way and heavily push-oriented. SSE is also HTTP which simplifies infrastructure and deployment.

We conflate messages in the backend so that only the most recent state is sent. This reduces client-side compute and network load.

When we were investigating sorting performance, we looked into the AG Grid source code. The way it handled data transformation was modeled after the Apache Commons Pipeline, which I was very excited to learn about. We tried to implement something similar In our own data model, and found it to be a great way to structure our code.

We use AG Grid to render both static and streaming data, so we had to wrap AG Grid in different ways to make it fit those data pipelines.

We send a variety of message types (40+) to the UI, each carrying upwards of 50 fields. To help end users manage this amount of information it’s crucial that we create an extensive workspace system to persist their configurations for each view. To do this, we use AG Grid’s event listeners to capture the grid’s UI state, things like sorting, positioning, visibility. Events like double-click are also intercepted to launch custom forms or potentially kick off other user flows.

We’re currenting testing our own data model in UAT using real-world data and scenarios

How did you find the ease of customization?

AG Grid does not seem overly opinionated in how things should be done, beyond the standard set of features, it hands over the tools and lets the developer figure things out. We like this part very much. It allows us to be flexible and pick-n-choose when and where to roll up our sleeves to get exactly what we want. We would much rather this approach than something more opinionated and locked-down, resulting in more situations where it’s make or break.

AG Grid covers a wide array of features, which so far, we find adequately exposed through its API. There are callbacks and event listeners that allow for easy grafting into the UI lifecycle. Early on we did find differences in how some of the column models were implemented, but it seems that AG Grid had recently put in work to consolidate this area, which definitely translated to quality of life improvements downstream.

We had to override a couple Sass parameters to get the grid as compact as possible, but have yet to utilize the Sass API for custom theming, which I’m personally very much looking forward to.

The Row Class Rules styling system is a great little feature: either tell AG Grid how to style, or tell it how to qualify data and tie them to external styling.

We implemented conditional formatting per view and across views using this system, and it was very quick and painless. It’s another good example of AG Grid finding the middle ground between doing too little and doing too much.

We are working on a pretty hefty refactoring to make the grid even more agnostic with a new rendering engine with a tighter integration to React.

I think that’s great news, though I hope AG Grid can do so without sacrificing the unique flexibility/extensibility that makes it such a wonderful product.

There are certainly libraries out there that are “React-to-the-bone”, with cool things like JSX components. But I find all of that to be moot unless the library is fast and extensible, especially for niche areas like fintech.

AG Grid may not be the most sleek-looking product compared to some of the other “millennial” libraries out there. But it does complicated things (like master-detail) well and is highly customizable.

Because we extended AG Grid quite a bit, being able to reference the GridApi and ColumnApi was very useful. Having to wait for these references and pass them around feels odd with how React flows. But at the same time, I’m not sure if there’s always a way around it because sometimes we can’t walk around doing things imperatively. Sure it would be nice to live in a perfect world with one-way traffic, but I can’t help but be suspicious of the kind of gymnastics it would require to make things that pure.

The way AG Grid passes its APIs into the callbacks makes it easier to design around things like this.

Thank you Han and Prerak for taking the time to answer our questions.

You can learn more about Proof Trading at ProofTrading.com and an explanation of the system architecture on their blog.

Learn more about the React Data Grid used by Proof Trading, or follow our Getting Started Guide